The bottom line: yes, you are crazy.
Botulism arises when you have a substance which carries clostridium botulinum and keep it for a while (timescales of days rather than hours itor minutes) in an anaerobic environment (like a bottle of oil).
The bacteria cannot burrow into the bottle from the outside, nor can it swim against gravity from your vegetables into the oil. The problem arises only if you end up with some of the vegetable matter inside the bottle.
If you are still concerned about this, you can decant some oil into a measuring cup and pour from there.