He is not sure whether he said Rabbana wa laka’l-hamd after the imam said the takbeer for prostration

Firstly: 

Tasmee’ (which means saying Sami’a Allaahu liman hamidah) when rising from bowing, and tahmeed (which means saying Rabbana laka’l-hamd) when standing up straight are Sunnah mustahabbah according to the majority of scholars, but the Hanbalis are of the view that they are obligatory, which is the correct view. 

Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on him) said in al-Sharh al-Mumti’ (3/433): 

The evidence for that (i.e., that it is obligatory) is as follows: 

(i)

The Messenger (peace and blessings of Allaah be upon him) always did that, and he did not omit to say Sami’a Allaahu liman hamidah in any circumstances. 

(ii)

It is the sign for moving from bowing to standing. 

(iii)

The Prophet (peace and blessings of Allaah be upon him) said: “When he says Sami’a Allaahu liman hamidah, then say Rabbana wa laka’l-hamd.” End quote. 

See the answer to question no. 102700. 

Secondly: 

If a person is not sure whether he omitted one of the obligatory duties of prayer during the prayer, then he is like one who omitted it. If he is still at that point in the prayer, then he should do it, and he does not have to do anything else. If he has passed that point in the prayer then it is not possible for him to do it, but he has to do sujood al-sahw (the prostration of forgetfulness). 

Based on this, if your doubt and uncertainty occurred whilst you were standing up straight after bowing, then you should have said these words then, because you were still at that point in the prayer. 

If the uncertainty came to you after you prostrated, then the time for saying tahmeed had passed, and in that case you were like one who omitted it out of forgetfulness. If you had caught up with the first rak’ah with the imam, then you should have said the tasleem with the imam and the prostration of forgetfulness would have been waived for you in that case. 

If you had come late and missed one or more rak’ahs with the imam, then you should do the prostration of forgetfulness before the tasleem, after completing your prayer. If you did not do the prostration for forgetfulness in this case because you did not know that it is obligatory, then your prayer is still valid and you do not have to do anything. 

To sum up: your prayer is valid and you do not have to repeat it. As for the prostration of forgetfulness, it is according to the details mentioned above. 

And Allaah knows best.