Sometimes when I steam potatoes, they have a lovely dry, fluffy, crystaline texture. Other times, they're the complete opposite, looking dull with a texture like that of banana.
Could it be you are not using the same type of potatoes all the time? Different varieties of potatoes do have very different textures. The ones with the brown, rough, dusty skins ("russet" I think they are called?) usually cook up with the fluffy, dry, almost "sparkly" flesh. The smoother skinned ones tend to have denser, waxier texture inside. The pink, smooth ones ("rose" or "early rose") seem to fit the description of "looking dull" inside when cooked.
You mentioned in the comments that you store your potatoes at close to freezing in your refrigerator. When potatoes get cold, microscopic ice crystals can form inside that will help create steam during cooking, producing fluffy texture like what you describe.
I'll bet that the inconsistency in your potatoes is due to slight differences in temperature in your refrigerator (i.e., some areas of your refrigerator might be 5°C but others 0°C; refrigerators are rarely 100% consistent). It could also be the case that your potatoes were stored at near- or below-freezing temperatures before you bought them.
The cold will also promote the starch in potatoes to convert into sugar, which can cause the potatoes to taste sweet and/or cause premature browning. This effect and the textural changes may be desirable for certain recipes, e.g., french fries. However, for most recipes you will want to use potatoes that have never been close to freezing temperatures.