Yes, they're realistic concerns, but you can also take precautions against rancidity and such.
Keep the oil out of the light. Either use an opaque oil dispensor (mine's stainless steel), or keep it in a cabinet (which my mom does).
Keep the oil away from heat. Don't keep it near the stove or on the window sill.
Don't put anything other than pure oil in the container. (no herbs, garlic, or other infused oils)
My mom would go through a 3L can pouring it into a smaller olive oil bottle she was reusing. I don't recall her ever cleaning it. I do occassionally clean my container, mostly because I keep it out, and after a bit of frying, you get that film on the can, so when it's low I'll drain the remaining oil into my oil sprayer and wash it ... but maybe only 2 or 3 times a year.
The containers I generally mostly empty before refilling ... except for the sprayer. There's a minimim amount required in it so that it'll work, so I just refill it. And I can't remember ever cleaning out the ones I have, other than the sprayer portion when it starts acting up. (I'm on my third one in maybe 14 years?)
I've never had any problems with mold, but I use my containers regularly, and I've only lived for long periods in a moderate climate (Kentucky, Maryland, Netherlands)
update : I should also add ... if you wash the container, it is critically important that it be fully dried before you use it again. Botulism needs moisture to thrive, so you want to make sure that no water is in there. (I suspect that the mold on cheese actually requires the moisture from the milk, not just the fat.) Remember that the French packed meats in fat to preserve them (duck confit & pâté)