He started to pray, then he wasn’t sure whether he formed the intention to pray or not; does he have to repeat it?

The intention (niyyah)
is one of the conditions of prayer being valid. If a person starts to pray
without having formed the intention, then his prayer is not valid.

Ibn Qudaamah (may Allah have mercy on him) said: We do not know
of any difference of opinion among the scholars concerning the necessity of
forming the intention to pray; prayer cannot be done without it. The basic
principle concerning that is the words of Allah, may He be exalted (interpretation
of the meaning): “And they were not commanded except to worship Allah,
[being] sincere to Him in religion” [al-Bayyinah
98:5]. Sincerity is an action of the heart, and this refers to the
intention, seeking Allah alone and no other. And the Prophet (blessings and
peace of Allah be upon him) said: “Actions are but by intentions, and each one
will have but that which he intended.” And the seat of the intention is the
heart.

End quote from al-Mughni,
1/287

But it is very unlikely that this thing (praying
without intention) could happen. The intention could accompany the opening takbeer (when one says “Allahu akbar” to start the prayer), or it may be shortly before
it.

When the Muslim hears the adhaan, then he gets up, does wudoo’
and goes to the mosque, and sits down waiting for the iqaamah
(call immediately preceding the prayer), then when the iqaamah
is given he gets up and stands in the row and so on, then he has definitely
formed the intention to offer the prayer for which the call was given and the iqaamah was recited, so how could he have any doubts about
his intention after that?

Based on that, what appears to be the
case is that the doubt you are feeling comes under the heading of whispers from
the Shaytaan (waswaas) who
wants to distract you from your prayer and prevent you from pondering and
focusing properly. So you should not pay any attention to it.

For more information on dealing with waswaas, please see the answer to question no. 100268 and 62839.

It should be noted that if there is a great
deal of doubt, you should not pay any attention to it, because it is whispers
from the Shaytaan.

Shaykh Ibn ‘Uthaymeen (may Allah have
mercy on him) said: If a person experiences so many doubts that he can hardly
do anything without doubting it – if he does wudoo’
he doubts it, and if he prays he doubts it, and if he fasts he doubts it – then
this also does not mean anything, because this is a kind of sickness. What we
discussed above applies to the healthy person who is free of sickness; if a
person is experiencing a great deal of doubt like this, then it is to be
assumed that his mind is not stable, so no attention should be paid to these
doubts.

End quote from ash-Sharh al-Mumti‘, 3/379

And Allah knows best.