Can customs duty be taken from non-Muslims?

Yes, there is nothing wrong with taking customs duty from non-Muslims for the goods that they bring to sell to the Muslims. 

Al-Bayhaqi (18543) narrated from Anas (may Allaah be pleased with him) that ‘Umar ibn al-Khattaab (may Allaah be pleased with him) ordered that one half of one-tenth be taken from ahl al-dhimmah (non-Muslims living under Muslim rule and protection) and that one-tenth be taken from those who were not living under Muslim protection. 

It says in al-Mawsoo’ah al-Fiqhiyyah (30/102, 103): 

One-tenth is to be taken from non-Muslim merchants when they enter the Muslim land, and that is in general terms. … The fuqaha’ cited as proof that it is prescribed to take one-tenth from non-Muslims the Sunnah, scholarly consensus and common sense. As for the Sunnah, it is the hadeeth: “The ‘ushoor (one-tenth) is to be taken from the Jews and Christians, and there is no ‘ushoor for the Muslims.” [The hadeeth is da’eef, and was classed as such by al-Albaani in Da’eef Abi Dawood (3046).] 

This hadeeth indicates that no wealth is to be taken from the Muslims apart from the zakaah, but one-tenth of trade goods is to be taken from the Jews and Christians as the jizyah is taken from them. As for scholarly consensus: ‘Umar ibn al-Khattaab (may Allaah be pleased with him) sent the ‘ushshaar (collector) to collect the ‘ushoor (one-tenth), and that command was issued in the presence of the Sahaabah (may Allaah be pleased with them), and no one disagreed with him concerning that, so their silence represented consensus. 

As for common sense: the merchant who brings goods from one country to another needs security and protection against thieves and bandits. The Islamic state guarantees to secure its roads and trade routes, and the ‘ushoor that is taken from the merchant is in return for that protection and in return for using the public facilities of the Islamic state. End quote.

With regard to taking customs duty from Muslims, this has been discussed in the answer to questions no. 39461, 42563 and 93088. It is haraam and is a major sin.