If he catches up with the imam when he is prostrating or sitting

Firstly: 

If a worshipper enters the mosque when the imam is prostrating or sitting or whatever, he should join him, because the Prophet (peace and blessings of Allaah be upon him) said: “If you come to prayer and we are prostrating, then prostrate but do not count that as anything (i.e., as a rak’ah), and whoever catches the rak'ah has caught up with the prayer.” Narrated by Abu Dawood, 893; classed as hasan by al-Albaani in Saheeh Abi Dawood. 

See also question no. [46811]. 

But this rak’ah does not count for him, because he did not catch up with bowing with the imam. 

Secondly: 

The scholars (may Allaah have mercy on them) have stated that the worshipper who catches up with his imam when he is prostrating should say the opening takbeer and then prostrate with his imam, without saying takbeer, because he has not caught up at a point where takbeer is required. 

Ibn Qudaamah said in al-Mughni (2/183): If he catches up with the imam in a pillar (obligatory part of the prayer) other than bowing, he should not say any takbeer apart from the opening takbeer, and he should go down (in prostration) without saying takbeer, because this (rak’ah) is not counted, and he has missed the point where takbeer is required. If he catches up with him when he is prostrating or in the first tashahhud, he should say takbeer when he stands up with the imam for the third rak’ah, because he is following him, so he should follow him in saying takbeer, like one who catches up with him at the beginning. 

See al-Majmoo’ Sharh al-Muhadhdhab, 4/218 

This is unlike the takbeer for bowing, which is counted for him, and it is unlike what he should do after that when he moves with the imam, getting up from prostration etc. He should say takbeer, following the imam in his movements, even if it is not counted for him, as stated by Ibn Qudaamah (may Allaah have mercy on him) 

The second opinion is that he should go down in prostration with the imam, saying takbeer. So he should say takbeer the first time to enter the prayer (opening takbeer), and the second time to prostrate or sit, because he has to follow the imam, who is sitting or prostrating, and when one moves from standing to prostrating, there should be a takbeer. 

See al-Majmoo’, 4/218; Haashiyat Ibn Qaasim, 2/277; al-Insaaf, 2/225; al-Sunan al-Kubra, by al-Bayhaqi, 2/91 

Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on him) said in al-Sharh al-Mumti’, 4/128: 

The well known view of the fuqaha’ (may Allaah have mercy on them) is that he should go down (in prostration) without saying takbeer. 

But despite that we say that if a person says takbeer it does not matter, and if he does not say it, it does not matter, and we say that it is the person’s choice, because there is no clear evidence to differentiate between bowing and anything else. So it is possible to say that sitting does not come after standing, but what made me sit is the fact that I am following the imam, so now I am moving to an essential part of the prayer that I am enjoined to move to. But I am doing that to follow the imam, not because this is how the prayer should be in principle. Undoubtedly he supports the view that he should say takbeer, so what we think concerning this matter is that in order to be on the safe side he should say takbeer. End quote. 

Thirdly: 

If he enters the prayer and goes down in prostration, then the imam raises his head before he puts his forehead on the ground, then it seems that he should come back up with him and not prostrate, because the time for following him (in prostration) ceased when the imam raised his head from the ground before the person put his forehead on the ground, unlike if he had been with him from the beginning of the prayer. 

If he catches up with him in the first prostration and he goes down in prostration then the imam raises his head, and sits between the two prostrations, the person should sit with him, then when he does the second prostration he should prostrate with him. 

And Allaah knows best. 

See Ahkaam Hudoor al-Masaajid, p. 143-144, by Shaykh ‘Abd-Allaah ibn Saalih al-Fawzaan.