He was not sure about the number of prostrations, and he omitted the prostration of forgetfulness out of ignorance; what is the ruling on his prayer?

Firstly:

If a person is unsure about the number of prostrations, in the sense that he is not sure whether he prostrated once or twice, then he should proceed on the basis of what is certain, which is the smaller number. So he should assume that he has done one prostration, and do a second prostration; then he should do the prostration of forgetfulness before saying the salaam, by way of preference. This view is the one favoured by Shaykh Ibn Baz (may Allah have mercy on him).

Shaykh Ibn Baz (may Allah have mercy on him) said:

With regard to uncertainty in the prayer, he should do another prostration and proceed on the basis of what is certain. If he is not sure whether he prostrated once or twice, he should do another prostration, whether that is in the first rak‘ah or in the second, third or fourth, and he should do the prostration of forgetfulness before saying the salaam, but if he does it after the salaam it does not matter, although it is preferable to do it before the salaam.

End quote from Majmoo‘ Fatawa Ibn Baz (30/11)

Some of the scholars are of the view that uncertainty about not having done essential parts of the prayer is like uncertainty concerning the number of rak‘ahs, so one should proceed on the basis of what is certain, which is the smaller number, if one of the two possibilities is not more likely in his view, and in this case he should do the prostration of forgetfulness before saying the salaam.

But if one of the two possibilities is more likely in his view, then he should proceed on that basis, and in that case the prostration of forgetfulness should come after the salaam.

Al-Mardaawi (may Allah have mercy on him) said:

“If he is not certain about having omitted an essential part of the prayer, then it is as if he did omit it” – this is our view and it is the view of most of our companions, and was stated definitively by some of them. And it was said that he should try to work it out and proceed on the basis of what he thinks most likely to be the case. 

End quote from al-Insaaf (2/150)

Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) said:

“If he is not certain about having omitted an essential part of the prayer, then it is as if he did omit it” – that is, if he is not sure whether he did that essential part or omitted it, then the ruling in this case is the same as the ruling on one who did omit it.

For example, if he got up to do the second rak‘ah, then was not sure whether he had prostrated twice or only once,

Uncertainty about whether one omitted an essential part of the prayer is like omitting it, because the basic principle is that it was not done. So if he was uncertain as to whether he had done it, but he thinks it most likely that he did do it, then according to the more correct view, which is that one should act upon what one thinks most likely to be the case, then he is deemed to have done it and he does not have to do go back and repeat it, because we stated that if he is uncertain about the number of rak‘ahs, he should proceed on the basis of what he thinks was likely to be the case, but he has to do the prostration of forgetfulness after the salaam. End quote from ash-Sharh al-Mumti‘ (3/384)

Secondly:

The scholars (may Allah have mercy on them) stated that whoever omits to do the prostration of forgetfulness because he forgot to do it should make it up so long as it is not been a long time since he finished the prayer. However, if a long time has passed since he finished the prayer, then the prostration of forgetfulness is waived in his case, and his prayer is valid.

Al-Bahooti (may Allah have mercy on him) said:

If he forgets the prostration (of forgetfulness), and it was preferable to do it before saying the salaam, then he should make it up. If he has started another prayer, then he should make it up when he has said the salaam at the end of that prayer, if only a short time has elapsed and he has not invalidated his wudoo’ or left the mosque. But if a long time has elapsed, according to custom, or he invalidated his wudoo’ or left the mosque, then he should not make up the prostration of forgetfulness, because the time for it has passed, but his prayer is still valid – as in the case of all other obligatory (waajib) duties if a person forgets to do them by mistake.

End quote from Sharh Muntaha al-Iraadaat (1/235)

With regard to this matter, the one who is ignorant is like the one who forgot.

It says in Fataawa al-Lajnah ad-Daa’imah (vol 2 – 6/10):

If he omitted the prostration of forgetfulness deliberately, his prayer is invalid and he has to repeat it. If he omitted it because he forgot or was ignorant of the ruling, then he does not have to repeat it and his prayer is valid. End quote.

Shaykh Ibn Baz (may Allah have mercy on him) was asked:

If a person does too many rak‘ahs in the prayer, or omits anything from it, and does not do the prostration of forgetfulness, is the prayer rendered invalid?

He replied:

That depends. If he decided to omit this prostration whilst he was praying, if he did that deliberately and was aware of the shar‘i ruling, then his prayer is rendered invalid. 

But if he was ignorant of the ruling or forgot, then it is not rendered invalid, and his prayer is still valid.

End quote from Fatawa Noor ‘ala ad-Darb by Ibn Baz

For more information, please see the answer to question no. 95410

And Allah knows best.