Not generally; your skin is quite good at protecting you from invasion by microbes and spores. The bigger risk is (as you identify) inhalation, mostly the actual spores rather than any toxic fumes or vapors. There might also be some risk of infection if you had an open wound that came into contact with moldy produce for any length of time.
In any case, I wouldn't regard this as a greatly elevated risk over just having that stuff around; mold is good at aggressively spreading its spores through the air, so you did the right thing by throwing it out immediately. Assuming you are otherwise healthy and don't have a compromised immune system for any reason, you're very likely fine (if a bit disgusted). Wash your hands well, and clean the area where the potatoes were stored, preferably with a bleach solution to kill any lingering spores. If you develop any feverish symptoms, go see a doctor; if you identify whoever left potatoes around long enough to rot, chastise them. There's probably much worse somewhere in a dorm environment but rotting food is a definite public health risk.