Should he do the prostration of forgetfulness after doing the part of the prayer that he has to make up?

Yes, you
have to do the prostration of forgetfulness after completing your prayer,
because if one who is praying behind an imam makes a mistake, he does not
have to do the prostration of forgetfulness unless he joined the prayer
late. 

Al-Nawawi
(may Allaah have mercy on him) said in al-Majmoo’ (4/63): 

If he makes
a mistake when praying behind the imam, the imam carries his mistake, and
neither of them should do the prostration of forgetfulness. There is no
difference of scholarly opinion on this point. Shaykh Abu Haamid said: This
is the view of all the scholars except Mak-hool, who said: The one who is
praying behind an imam should do the prostration of forgetfulness for a
mistake that he made. End quote. 

The evidence
for that is the words of the Prophet (peace and blessings of Allaah be
upon him): “The imam has been appointed to be followed, so do not differ
from him.” Narrated by al-Bukhaari (722) and Muslim (414). 

If the
person praying behind an imam does the prostration of forgetfulness, this
means that he is differing from the imam. 

But if he
joined the prayer late, then he may do the prostration of forgetfulness when
he has completed his prayer, because he is completing it on his own, so then
his prostration of forgetfulness does not mean that he is differing from the
imam. 

Secondly: 

Imam
al-Nawawi (may Allaah have mercy on him) said in al-Majmoo’ (4/64): 

If he joins
the prayer late, then he makes a mistake after the imam says the salaam, the
imam will not carry it for him, because he is no longer following him. End
quote. 

It says in
al-Mawsoo’ah al-Fiqhiyyah (8/206): 

If the
person who joined the prayer late says the salaam after the imam by mistake,
he should continue making up what he has missed of his prayer and do the
prostration of forgetfulness. End quote. 

Shaykh
al-Sa’di (may Allaah have mercy on him) said: 

If he joined
the prayer late and he makes a mistake in the part of the prayer that he is
making up, then he has to do the prostration of forgetfulness. End quote. 

Fataawa
al-Sa’di (157). 

Shaykh Ibn
‘Uthaymeen (may Allaah have mercy on him) said in Majmoo’ al-Fataawa
(14/ no. 698): 

If the
person praying behind the imam makes a mistake in his prayer, and he joined
the prayer late, he has to do the prostration of forgetfulness if his
mistake is one that requires the prostration of forgetfulness. End quote. 

That
includes the person who joined the prayer late saying the salaam after the
imam by mistake. In that case he has to stand up and offer the part of the
prayer that he missed, then do the prostration of forgetfulness. 

And Allaah
knows best.