Yes, there are differences. Unfortunately, many potatoes have been bred for crop yield and storage over flavor ... and so the flavor differences that you might see in South America don't tend to be so prevalent in the US and Europe.
"Factory farming" in many ways started because of potatoes. McDonalds wanted to get rid of regional differences in their french fries, and so contracted with farmers in Idaho to grow a single variety of potato for them so they could get the uniformity that they wanted.
The main difference for those sold in the US is the type of starch in the potato -- most potatoes can be sorted into 'floury' vs. 'waxy' types:
Floury potatoes (also called 'baking potatoes', 'mealy potatoes', or 'starchy potatoes') will fall apart after cooking. This means that they disintegrate if you try to use them for soups and stews, but they make a lump-free mashed potato or a "fluffy" baked potato. Russets are in this category.
Waxy potatoes (also called 'boiling potatoes') will soften but stay a little bit firm and mostly stay in one piece after cooking if you don't agitate them too much. This makes them better for stews or potato salad where you want distinct chunks of potatoes, but worse for really fluffy mashed potatoes. "Red" potatoes tend to fall into this category.
There are also in-between types, such as "Yukon Gold" (some people put "white" potatoes in this category) -- they'll mash up okay (maybe with a few lumps), and if you use them in stews, they'll have firm chunks but the outsides will start to break up if you stir too vigorously. This is actually a benefit in some recipes.
As the starches change over time, "new" potatoes tend to behave more like waxy/boiling potatoes even when they come from a baking-type potato.
Some potatoes are sweeter than other varieties, although this can also change during storage. (Cold storage will get many varieties of potatoes to change their starches into sugars ... but it may happen unevenly). Sweeter potatoes will brown more, which can cause problems if they brown too quickly or unevenly when subjected to high heat (frying, roasting), which means the potato is either undercooked while looking pretty, or too dark when it's fully cooked through.
If you're working with a recipe that calls for a specific variety of the potato, sugar levels can often be the reason. There's at least one recipe out there that relies on cold-induced sweetening to try to convert American potatoes to more closely match an Austrian variety.
When starting out, you might not worry so much about the facet, other than considerations about long-term cold storage.
How wet a potato is affects how it cooks and how well it stores. It can also affect how much liquid it'll absorb when making mashed potatoes or similar, so a drier potato will allow you to add more flavorful liquid.
But I can never remember which varieties are dryer than others, so I rarely use this as a consideration. It's often more important to know that when a recipe calls for letting the potatoes to steam after cooking but before mixing in other ingredients, it might be about moisture release, not just cooling.
"New" potatoes and "fingerling" potatoes have a large surface area for their volume. This makes them a complete waste if you're going to be pealing them. They're better for skin-on, whole or mostly whole (halved / quartered) preparations.
If I'm going to be doing some sort of peeled and cut up preparation, then I want larger potatoes so I don't spend so much time peeling them (and waste so much volume compared to what's left).
Even if I'm leaving the skin on, I tend to favor slightly larger potatoes as I don't have to spend as much time scrubbing them. I'd rather quarter larger potatoes for roasting instead of dealing with the tiny potatoes.
If I'm baking potatoes, then I'll try to select ones that are uniformly sized, and a reasonable portion for what I'm preparing. (larger if it's the main thing, like chili over a baked potato ... but smaller if it's intended as a side). If I'm selecting loose potatoes, I might try to get some variety of size (so that people have a choice), but I try to stick with ones that are roughly the same circumference but different lengths so they cook up in roughly the same time.
For applications where you're not peeling the potatoes, especially when they're an important part of the dish such as potato skins or twice-baked potatoes, you may want to consider the texture of the skins. "New" and "Red" potatoes tend to have thinner skins, while Russet potatoes tend to have a thicker, rougher skin.
As the colors come from chemicals produced by the plant, there can be distinctive flavors associated with them, but they also provide variety on a plate. (although beware of mixing your own, as they'll cook up differently. Stick to small "new" potatoes.
Also note that skin color is independent of the flesh color. Most "Red" potatoes in the US such as "Red Bliss" are stark white on the inside ... but there are some varieties that have red flesh.
Flavor is a bit of a weird thing, as everyone processes flavors a little bit differently. I like the various yellow potatoes, as I find them to be more "buttery", but I've never done a blind taste test so I have no idea if it's the yellow color that's tricking me.
If you're serving the potatoes with a fairly plain preparation, then the potato flavors are going to come through better... but if you're covering it with vinegary pulled pork or sour cream and chives, then it's probably not worth paying extra for an exceptionally flavored potato.
You can get pre-washed, microwave-in-the-bag potatoes if you're willing to pay more. For smaller potatoes, they may not be vastly overpriced compared to loose potatoes of a similar size (and smaller potatoes are a PITA, so they can sometimes be worth it)
Bagged potatoes are a cost-savings, but they're also a bit of a gamble -- are the potatoes all the right size/shape for what I'm cooking? Is there going to be an off/weird/sprouting potato hiding in the bottom of the bag? Am I buying the right amount for what I need? (ie. I going to eat 5lbs of potatoes before I have to worry about sprouting?) Are the potatoes fairly smooth, or will they have lots of wrinkles or shovel damage that I have to scrub at?
In general, potatoes in grocery stores tend to be cleaner than in the past, but not all processors have the same equipment to wash & dry their potatoes before sale, so some are just ... dirtier. Usually, there won't be as much variation within a store, but you might see it from store-to-store. (or store-to-farmstand)