He said salaam before the end of the prayer by mistake then he started another prayer

If a person forgets and
says the salaam before completing his prayer, then he remembers shortly
afterwards, then he should go back and complete his prayer and do the
prostration of forgetfulness, because of the report narrated by al-Bukhaari
(1227) and Muslim (573) from Abu Hurayrah (may Allaah be pleased with him)
who said: The Messenger of Allaah (peace and blessings of Allaah be
upon him) prayed Zuhr or ‘Asr and he said the salaam (i.e. after two
rak’ahs). Dhu’l-Yadayn said to him: Has the prayer been shortened, O
Messenger of Allaah? The Prophet (peace and blessings of Allaah be upon
him) said to his companions: “Is it true what he is saying?” They said: Yes.
He prayed two more rak’ahs then he prostrated twice. 

If he has started to offer
a naafil prayer, then he remembers that he did not complete the obligatory
prayer, he should stop the naafil prayer and complete the obligatory prayer,
then do the prostration of forgetfulness. 

Ibn Qudaamah said in
al-Mughni (1/373): The one who says the salaam before completing his
prayer by mistake, then realizes before a long time has passed or his wudoo’
is invalidated, has to do what remains of his prayer, then recite the
tashahhud and say the salaam, then do the two prostrations of forgetfulness.
If a long time has passed or he has invalidated his wudoo’, then he should
repeat the prayer from the beginning. 

If he does not remember
until he has started another prayer, then it depends. If he has only done a
little of the second prayer and a long time has not passed, then he should
go back to the first prayer and complete it. If a long time has passed then
the first prayer is invalidated. This is the view of al-Shaafa’i. End
quote. 

In Kashshaaf al-Qinaa’
(1/400) it says: If the one who said the salaam before completing the prayer
does not remember until he has started another prayer, he should stop it if
a short time has passed, and go back to the first prayer and complete it, so
that he can attain continuity between its essential parts, then he should do
the prostration of forgetfulness. End quote. 

There is no specific
definition of what counts as a long time or a short time; he should refer to
what is customary. 

As for making the first
(deficient) obligatory prayer a voluntary prayer, that is not correct,
because actions are by intentions, and you prayed it with the intention of
offering an obligatory prayer, and you exited it likewise. Because you
repeated the obligatory prayer after that, then you have done what was
required of you. But it would have been better for you to stop the naafil
prayer and go back to the obligatory prayer and complete it, then do the two
prostrations of forgetfulness after saying the salaam at the end of the
prayer, then pray the naafil prayer after that. 

And Allaah knows best.