He was not sure whether he had said the opening takbir, so he repeated it; was the prayer of those who were praying behind him rendered invalid?

Firstly:

The opening takbir (takbirat al-ihram) is one of the essential parts of the prayer

The opening takbir (takbirat al-ihram) is one of the essential parts of the prayer; it is not waived if one forgets it or is unaware of the ruling, and nothing else can take its place. If a person remembers during his prayer that he forgot to say the opening takbir, or he is not sure whether he did it, he must start his prayer all over again, as we explained previously in the answer to question no. 69853 .

As for being uncertain as to whether you pronounced it correctly or not, if that stems from waswasah (intrusive thoughts or whispers from the Shaytan), then you should ignore it and not pay any attention to it.

Secondly:

Repeating the opening takbir

The one who repeats the opening takbir has invalidated his first prayer, and the ones who are praying behind him have now preceded him in saying the opening takbir with regard to his second prayer, in which they followed him. But this preceding of the imam may be overlooked, because there is a valid reason for it, as some of the scholars pointed out

Al-Bujayrami said in his commentary on al-Khatib (2/14):

The words “if the worshipper delayed his opening takbir until after the imam [had said it]” refer to the entire takbir. If his takbir partially overlaps with that of the imam, then his following of the imam in prayer is not valid, and he is not regarded as having begun the prayer except in two scenarios in which it is permissible for the one who is praying behind the imam to say the opening takbir before the imam:

[The first scenario is] if he said the opening takbir on his own, then joined a congregation after that.

The second scenario is if the imam and the people behind him said the opening takbir, then the imam was not sure of his own intention, so he repeated the takbir with the intention of starting the prayer, saying it in such a way that only he himself could hear it, and he continued to lead the prayer. End quote.

Al-Jamal (may Allah have mercy on him) said in his commentary on Sharh al-Manhaj (1/567): az-Zarkashi said: al-Hannati was asked about a man who said the opening takbir for a prayer in which he was leading the people, then he repeated the takbir quietly to himself, and he did not make the people aware of that after they said the opening takbir themselves.

He said: The prayer of the congregation is valid according to the more correct of the two scholarly views. End quote.

The view of the Hanafis is more lenient than that. They think that he is still in a state of prayer, and that his second takbir does not interrupt his prayer and does not invalidate his first takbir.

In al-Asl by Muhammad ibn al-Hasan ash-Shaybani, it says: What about a man who started to pray and recited Qur’an, then he became uncertain and was not sure whether he had said the opening takbir or not, so he repeated the opening takbir and the recitation, then he realised that he had said the takbir the first time?

He said: He should continue his prayer, and he must do the two prostrations of forgetfulness.

I said: Doesn’t this takbir interrupt his prayer? He said no; do you not see that his intention is only to do the same prayer, and he does not intend any other prayer. "(al-Asl 1/226).

The hadith of Sahl ibn Sa‘d as-Sa‘idi may be quoted as evidence for it being permissible for the one who is praying behind an imam to say the opening takbir before his imam in such exceptional circumstances. This hadith says that the Messenger of Allah (blessings and peace of Allah be upon him) went to Banu ‘Amr ibn ‘Awf to resolve a dispute among them. The time for prayer came, and the mu’adhdhin came to Abu Bakr and said: Will you lead the people in prayer, and I will say the iqaamah? He said: Yes. So Abu Bakr led the people in prayer, then the Messenger of Allah (blessings and peace of Allah be upon him) came whilst the people were still praying. He came and stood in the row, and the people started clapping. Abu Bakr used not to pay attention to anything whilst he was praying, but when the people’s clapping increased, he turned around and saw the Messenger of Allah (blessings and peace of Allah be upon him). The Messenger of Allah (blessings and peace of Allah be upon him) gestured to him to stay where he was. Abu Bakr raised his hands and praised Allah, may He be glorified and exalted, for the command of the Messenger of Allah (blessings and peace of Allah be upon him). Then Abu Bakr moved backwards until he was level with the row, and the Prophet (blessings and peace of Allah be upon him) came forward and led the prayer … Narrated by al-Bukhari (684) and Muslim (42`).

Conclusion:

The prayer of the congregation is valid, and nothing is required of you or them.

But what you must do is ignore waswas (intrusive thoughts or whispers from the Shaytan) and not pay attention to it, because it could invalidate a person’s prayer and, moreover, cause him a great deal of trouble.

And Allah knows best.