If you're going to be using a ricer, just halve the potatoes -- you can then put them cut side-down into the ricer, and then press -- the skin will be left behind. (I'd still give them a quick scrub before boiling them, though). Of course, this is easier, but not necessarily faster as it'll take longer for the potatoes to cook through.
If you bake them, just clean, bake, then cut in half, and put through the ricer. (Again, not really that much faster).
If you're cooking red potatoes, even if you're not using a ricer, there's no need to peel them beforehand, as you can boil them whole, then rub the skin off with a dry towel. (of course, for red potatoes, they're always going to be lumpy, and have texture, so in that case, I tend to leave the skins on).
To reduce the overall time, if I'm going to be going for a smooth mash:
- Only use large potatoes. Save the smaller ones for applications where you don't need to peel potatoes.
- Cut the potatoes into slabs. There's no need to get them to cubes. I tend to go to about 2cm (~3/4") thick. Any extra cooking time is made up for by reduced cutting / transfering bits to the pot / dealing with the chunk that fell and rolled under the cabinet / etc.
- Start the water warming while you're peeling. You don't want to place the potatoes into boiling water, but you can take some of the chill off it.
- Toss a few cloves of garlic (whole, but paper removed, end trimmed off) in with the potatoes, so you don't have to season afterwards.
- Use stock rather than milk. I keep boxed vegetable and chicken stocks at room temp, not chilled, so I don't have to heat it up first to deal with the issues of it cooling down the starches too fast.
- Decide if the ricer's really worth it -- I get good results just tossing everything back into the same pot after straining, hit 'em with a potato masher, add some liquid, then go through a few more times. It saves a lot of cleanup, but it might not be exactly the texture you're used to,