If he swallows bits of his skin, does that invalidate his fast?

It is not
permissible for a fasting person to allow any food, drink or medicine to
enter his stomach. Eating means introducing any substance into the stomach
by way of the mouth, even if it is harmful or not beneficial, such as
pebbles, nails, skin and so on. This is the view of the four imams and no
difference of opinion is known among them. 

See:
Haashiyat Ibn Qaasim ‘ala al-Rawd al-Murbi’ (3/389) 

Al-Shiraazi
al-Shaafa’i (may Allaah have mercy on him) said: 

It makes no
difference whether he eats something edible or inedible. If he swallows soil
or a pebble or a coin, his fast is invalidated, because fasting means
refraining from everything that reaches the stomach, but this person did not
refrain. End quote. 

Al-Nawawi
(may Allaah have mercy on him) commented on this by saying:  

Al-Shaafa’i
and his companions (may Allaah have mercy on them) said: If a fasting person
swallows things that are not usually eaten, such as coins, soil, pebbles,
grass, metal, thread and so on, he has broken his fast and there is no
dispute among us on this point. This is the view of Abu Haneefah, Maalik,
Ahmad, Dawood and the majority of earlier and later scholars. End quote. 

Al-Majmoo’ (6/340) 

Based on
this, swallowing this bit of skin is  regarded as breaking the fast, but if
a person swallows it unintentionally, then his fast is valid and there is no
sin on him. 

The scholars
of the Standing Committee said: 

If there are
ulcers on his gums or they bleed when he uses the miswaak, it is not
permissible to swallow the blood and he has to spit it out. But if it enters
his stomach without him meaning it to or swallowing it deliberately, there
is no sin on him. The same applies to vomit, if it goes back into his
stomach without him meaning it to, then his fast is still valid. End quote. 

Fataawa al-Lajnah al-Daa’imah, 10/254. 

And Allaah knows best.