He gave zakaah to a debtor and did not tell him that it was zakaah, and after a while he came and returned the money to him

One of the categories of people who are entitled to receive zakaah are debtors; this refers to someone who borrowed money for personal reasons, and does not have the means to pay off his debt, or borrowed money in order to reconcile between people.

It says in Daleel at-Taalib (p. 87): The debtor is the one who borrows money to reconcile between people, or borrows money for his own interests, then finds himself in financial difficulty. End quote.

If your brother was in debt and was entitled to receive zakaah, then he took the money lawfully and the zakaah was given appropriately, so it is not  permissible for you to take anything of it, and you must tell him that what he received was zakaah and cannot be returned. You do not have to tell him that it was zakaah from you; there is nothing wrong with it if he understands that it was zakaah from someone, and you can use some kind of double entendre when telling him that.

You did well by not telling him that the money was zakaah.

An-Nawawi (may Allah have mercy on him) said: If the owner or someone else gives zakaah to someone who is entitled to it, and does not tell him that this is zakaah, and does not say anything about it at all, that is acceptable and the zakaah is valid. This is the correct and well-known view. End quote from al-Majmoo‘ (6/233).

Ibn Qudaamah (may Allah be pleased with him) said: Ahmad ibn al-Husayn said: I said to Ahmad: Can one man give zakaah to another and say: This is zakaah, or should he keep quiet? He said: Why upset him by saying that? He should give it to him and keep quiet. Why does he need to annoy him (by telling him that)? End quote from al-Mughni (2/482).

If he refuses and will not accept the money that you try to give back to him, or he gives back to you the rest of the money that he took, and will not agree to keep it for himself, then give what he returns to you to someone else among the poor and needy, so long as it is known that he falls into one of the categories who are entitled to zakaah. Thus you will have done what you are required to do of giving zakaah on your wealth.

And Allah knows best.