The Source (you know which it is, right?*) says that toxicity has been proven - for a single species of fern. The exact quote is
A common species especially enjoyed in Japan and Korea, the bracken fern Pteridium aequilinium, has been found to contain a potent DNA-damaging chemical. It should be avoided. Stalks of the ostrich fern, varikous species of Matteuccia, are thought to be safer to eat.
and on another page
Bracken-fern toxins caise several blood disorders and cancer in animals that graze on this common fern (Pteridium)... Matteucia... are thought to be safer... but there's little solid information about the safety of eating ferns. It's prudent to eat fiddleheads in moderation...
So if you believe him, make sure you know they come from the correct kind of fern. I guess this isn't a problem with supermarket bought vegetables.
Nowhere (or in no location mentioned in the index under fiddleheads) does he mention an unusually high contamination of microorganisms. But if you are afraid of them carrying dangerous agents on the outside, I think that they, when grown in the wild, will harbor the usual suspects carried by animal waste (E. coli, parasites like liver flukes or tenia). But I doubt that they will be higher than from other vegetables picked from the countryside (nettles, sorrel, dandelions).
*McGee on food and cooking