If he swallows bits of his skin, does that invalidate his fast?

It is not permissible for a fasting person to allow any food, drink or medicine to enter his stomach. Eating means introducing any substance into the stomach by way of the mouth, even if it is harmful or not beneficial, such as pebbles, nails, skin and so on. This is the view of the four imams and no difference of opinion is known among them. 

See: Haashiyat Ibn Qaasim ‘ala al-Rawd al-Murbi’ (3/389) 

Al-Shiraazi al-Shaafa’i (may Allah have mercy on him) said: 

It makes no difference whether he eats something edible or inedible. If he swallows soil or a pebble or a coin, his fast is invalidated, because fasting means refraining from everything that reaches the stomach, but this person did not refrain. End quote. 

Al-Nawawi (may Allah have mercy on him) commented on this by saying:  

Al-Shaafa’i and his companions (may Allah have mercy on them) said: If a fasting person swallows things that are not usually eaten, such as coins, soil, pebbles, grass, metal, thread and so on, he has broken his fast and there is no dispute among us on this point. This is the view of Abu Haneefah, Maalik, Ahmad, Dawood and the majority of earlier and later scholars. End quote. 

Al-Majmoo’ (6/340) 

Based on this, swallowing this bit of skin is  regarded as breaking the fast, but if a person swallows it unintentionally, then his fast is valid and there is no sin on him. 

The scholars of the Standing Committee said: 

If there are ulcers on his gums or they bleed when he uses the miswaak, it is not permissible to swallow the blood and he has to spit it out. But if it enters his stomach without him meaning it to or swallowing it deliberately, there is no sin on him. The same applies to vomit, if it goes back into his stomach without him meaning it to, then his fast is still valid. End quote. 

Fatawa al-Lajnah al-Daa’imah, 10/254. 

And Allah knows best.