He is not sure whether he said Rabbana wa laka’l-hamd after the imam said the takbeer for prostration

Firstly: 

Tasmee’ (which means saying
Sami’a Allaahu liman hamidah) when rising from bowing, and tahmeed (which
means saying Rabbana laka’l-hamd) when standing up straight are Sunnah
mustahabbah according to the majority of scholars, but the Hanbalis are of
the view that they are obligatory, which is the correct view. 

Shaykh Ibn ‘Uthaymeen (may
Allaah have mercy on him) said in al-Sharh al-Mumti’ (3/433): 

The evidence for that
(i.e., that it is obligatory) is as follows: 

(i)

The Messenger (peace
and blessings of Allaah be upon him) always did that, and he did not omit to
say Sami’a Allaahu liman hamidah in any circumstances. 

(ii)

It is the sign for moving
from bowing to standing. 

(iii)

The Prophet (peace and
blessings of Allaah be upon him) said: “When he says Sami’a Allaahu liman
hamidah, then say Rabbana wa laka’l-hamd.” End quote. 

See the answer to question
no. 102700. 

Secondly: 

If a person is not sure
whether he omitted one of the obligatory duties of prayer during the prayer,
then he is like one who omitted it. If he is still at that point in the
prayer, then he should do it, and he does not have to do anything else. If
he has passed that point in the prayer then it is not possible for him to do
it, but he has to do sujood al-sahw (the prostration of forgetfulness). 

Based on this, if your
doubt and uncertainty occurred whilst you were standing up straight after
bowing, then you should have said these words then, because you were still
at that point in the prayer. 

If the uncertainty came to
you after you prostrated, then the time for saying tahmeed had passed, and
in that case you were like one who omitted it out of forgetfulness. If you
had caught up with the first rak’ah with the imam, then you should have said
the tasleem with the imam and the prostration of forgetfulness would have
been waived for you in that case. 

If you had come late and
missed one or more rak’ahs with the imam, then you should do the prostration
of forgetfulness before the tasleem, after completing your prayer. If you
did not do the prostration for forgetfulness in this case because you did
not know that it is obligatory, then your prayer is still valid and you do
not have to do anything. 

To sum up: your prayer is
valid and you do not have to repeat it. As for the prostration of
forgetfulness, it is according to the details mentioned above. 

And Allaah knows best.