He is not sure whether he said the opening takbeer or not

The opening takbeer is
one of the pillars or essential parts of the prayer, which is not waived
because of forgetting or not knowing about it, and it cannot be substituted
with anything else. If a person remembers during his prayer that he forgot
to say the opening takbeer, or he is not sure whether he said it, he has to
start the prayer over again. If his doubt about that comes after the prayer,
then it does not matter, because doubts after completing an act of worship
do not affect it. 

Based on this, if the doubt of this worshipper who is not
sure whether he said the opening takbeer or not came during the prayer, he
has to stop praying and start the prayer over again. If the doubt came after
the prayer, then he does not have to do anything and his prayer is valid. 

The evidence that the opening takbeer is an essential part of
the prayer is the report narrated by al-Bukhaari (757) and Muslim (397) from
the hadeeth of the one who prayed badly. The Prophet (peace and
blessings of Allaah be upon him) said to him: “Go back and pray, for you
have not prayed.” Then he said: “When you stand up to pray, say takbeer then
recite whatever you know of the Qur’aan. Then bow until you are at ease in
bowing, then stand up until you are standing up straight. Then prostrate
until you are at ease in prostration, then sit up until you are sitting up
straight, and do that throughout your prayer.” 

And Abu Dawood (61), al-Tirmidhi (3) and Ibn Maajah (275)
narrated that ‘Ali (may Allaah be pleased with him) said: The Messenger of
Allaah (peace and blessings of Allaah be upon him) said: “The key to
prayer is purification, its opening is to say Allaahu akbar and its
closing is to say al-salaamu’ alaykum.” Classed as saheeh by
al-Albaani in Saheeh Abi Dawood. 

Al-Nawawi (may Allaah have mercy on him) said: 

The opening takbeer is
one of the pillars (essential parts) of prayer without which the prayer is
not valid. This is our view and the view of Maalik, Ahmad and the majority
of earlier and later scholars. Ibn al-Mundhir and our companions narrated
that al-Zuhri said: The prayer begins simply with the intention, without any
takbeer. But Ibn al-Mundhir said: No one said that except al-Zuhri. 

Then he said: We have
stated that the prayer is not valid without the opening takbeer. If the imam
or the person praying behind him omits it by mistake or deliberately, his
prayer has not begun, and the takbeer for bowing does not make up for it,
neither does any other takbeer. This is our view and is the view of Abu
Haneefah, Maalik, Ahmad, Dawood and the majority. 

End quote from al-Majmoo’, 3/250 

Whoever is not sure whether he has said the opening takbeer
or not should assume that he has not said it. Shaykh Ibn Baaz said: 

If a person forgets to say the opening takbeer or is not sure
about that, then he has to say takbeer straight away, and follow the imam in
whatever comes after he said this takbeer. If he said the takbeer after the
imam has done the first rak’ah, then he should regard himself as having
missed the first rak’ah, which he should make up after the imam says the
salaam. If he said the takbeer in the third rak’ah, then he should regard
himself as having missed two rak’ahs, which he should make up after the imam
says the salaam. This applies if he is not affected by waswasah (whispers
from the Shaytaan). If he is affected by waswasah, then he should regard
himself as having said the opening takbeer at the beginning of the prayer,
and not make up anything, in order to annoy the Shaytaan and combat his
whispers. Praise be to Allaah. End quote. 

Fataawa al-Shaykh Ibn Baaz,
11/275 

If the doubt comes after
finishing the prayer, then he should not pay any attention to it. 

Ibn Rajab (may Allaah have mercy on him) said: If a person is
uncertain after finishing a prayer or any other act of worship as to whether
he omitted a essential part of it, he should not pay any attention to that
doubt. 

End quote from al-Qawaa’id, p. 340 

See also the answer to question no.
211 

And Allaah knows best.