Yes, there is a difference in taste. I can't tell you what causes it, possibly the skin preventing the water from properly penetrating the potatoes. But when I eat a peeled boiled potato, I can always tell if it has been boiled skin on or off.
My personal preference is very much for skin offboiled-skin-off. Skin onBoiled-skin-on potatoes have their own specific taste even after peeling, which is slightly bitter and has a slight physical reaction on the tongue - not exactly hot, not exactly astringent, but somewhat reminiscent of both.
This is a taste I get when eating the potatoes as they are. Mashed, or swimming in some dip, the difference is too slight to notice.