He was not sure about the number of prostrations, and he omitted the prostration of forgetfulness out of ignorance; what is the ruling on his prayer?

Praise
be to Allah

Firstly:

If a person is unsure about the number of prostrations, in
the sense that he is not sure whether he prostrated once or twice, then he
should proceed on the basis of what is certain, which is the smaller number.
So he should assume that he has done one prostration, and do a second
prostration; then he should do the prostration of forgetfulness before
saying the salaam, by way of preference. This view is the one favoured by
Shaykh Ibn Baaz (may Allah have mercy on him).

Shaykh Ibn Baaz (may Allah have mercy on him) said:

With regard to uncertainty in the prayer, he should do
another prostration and proceed on the basis of what is certain. If he is
not sure whether he prostrated once or twice, he should do another
prostration, whether that is in the first rak‘ah or in the second, third or
fourth, and he should do the prostration of forgetfulness before saying the
salaam, but if he does it after the salaam it does not matter, although it
is preferable to do it before the salaam.

End quote from Majmoo‘ Fataawa Ibn Baaz (30/11)

Some of the scholars are of the view that uncertainty about
not having done essential parts of the prayer is like uncertainty concerning
the number of rak‘ahs, so one should proceed on the basis of what is
certain, which is the smaller number, if one of the two possibilities is not
more likely in his view, and in this case he should do the prostration of
forgetfulness before saying the salaam.

But if one of the two possibilities is more likely in his
view, then he should proceed on that basis, and in that case the prostration
of forgetfulness should come after the salaam.

Al-Mardaawi (may Allah have mercy on him) said:

“If he is not certain about having omitted an essential part
of the prayer, then it is as if he did omit it” – this is our view and it is
the view of most of our companions, and was stated definitively by some of
them. And it was said that he should try to work it out and proceed on the
basis of what he thinks most likely to be the case.

End quote from al-Insaaf (2/150)

Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) said:

“If he is not certain about having omitted an essential part
of the prayer, then it is as if he did omit it” – that is, if he is not sure
whether he did that essential part or omitted it, then the ruling in this
case is the same as the ruling on one who did omit it.

For example, if he got up to do the second rak‘ah, then was
not sure whether he had prostrated twice or only once,

Uncertainty about whether one omitted an essential part of
the prayer is like omitting it, because the basic principle is that it was
not done. So if he was uncertain as to whether he had done it, but he thinks
it most likely that he did do it, then according to the more correct view,
which is that one should act upon what one thinks most likely to be the
case, then he is deemed to have done it and he does not have to do go back
and repeat it, because we stated that if he is uncertain about the number of
rak‘ahs, he should proceed on the basis of what he thinks was likely to be
the case, but he has to do the prostration of forgetfulness after the
salaam. End quote from ash-Sharh al-Mumti‘ (3/384)

Secondly:

The scholars (may Allah have mercy on them) stated that
whoever omits to do the prostration of forgetfulness because he forgot to do
it should make it up so long as it is not been a long time since he finished
the prayer. However, if a long time has passed since he finished the prayer,
then the prostration of forgetfulness is waived in his case, and his prayer
is valid.

Al-Bahooti (may Allah have mercy on him) said:

If he forgets the prostration (of forgetfulness), and it was
preferable to do it before saying the salaam, then he should make it up. If
he has started another prayer, then he should make it up when he has said
the salaam at the end of that prayer, if only a short time has elapsed and
he has not invalidated his wudoo’ or left the mosque. But if a long time has
elapsed, according to custom, or he invalidated his wudoo’ or left the
mosque, then he should not make up the prostration of forgetfulness, because
the time for it has passed, but his prayer is still valid – as in the case
of all other obligatory (waajib) duties if a person forgets to do them by
mistake.

End quote from Sharh Muntaha al-Iraadaat (1/235)

With regard to this matter, the one who is ignorant is like
the one who forgot.

It says in Fataawa al-Lajnah ad-Daa’imah (vol 2 –
6/10):

If he omitted the prostration of forgetfulness deliberately,
his prayer is invalid and he has to repeat it. If he omitted it because he
forgot or was ignorant of the ruling, then he does not have to repeat it and
his prayer is valid. End quote.

Shaykh Ibn Baaz (may Allah have mercy on him) was asked:

If a person does too many rak‘ahs in the prayer, or omits
anything from it, and does not do the prostration of forgetfulness, is the
prayer rendered invalid?

He replied:

That depends. If he decided to omit this prostration whilst
he was praying, if he did that deliberately and was aware of the shar‘i
ruling, then his prayer is rendered invalid. 

But if he was ignorant of the ruling or forgot, then it is
not rendered invalid, and his prayer is still valid.

End quote from Fataawa Noor ‘ala ad-Darb by Ibn Baaz

See:

http://www.binbaz.org.sa/mat/15296

For more information, please see the answers to questions no.
95410 and 134518

And Allah knows best.