Seems almost every question about bacteria seems to bring about both the bacteriophobes and the cavalier. In this case, the cavalier are probably closer to being right, but I'll try to present the facts so that you can decide for yourself.
The most common microbe groups responsible for household food poisoning are:
Salmonella, found in poultry, produces the CdtB toxin. CDT can be denatured (technically, "heat-labile") but is one of the most resistant toxins, taking 30 minutes at 70° C to destroy. This type of bacteria isn't normally associated with eggs, but since chicken eggs and other poultry products may be handled in the same area, it's not unthinkable that the shells could become contaminated - and depending on how you handle the eggs, so could the frying pan.
E.Coli is normally associated with beef but is also known to be found on egg shells. It's improbable, but not impossible for it to be found in yolks as well. This is a bit of a weird one because egg yolks also have immunization properties against e.coli, but not in the yolk itself or the frying pan. The strains of e.coli associated with food poisoning encode Shiga-Like Toxin (SLT) and also Enterotoxins, the latter of which are often heat stable. I'm not sure about SLT, but the STs can withstand boiling for 30 minutes.
Campylobacter (causing Campylobacteriosis) produces certain enterotoxins and also CDT. As above, these tend to be heat-resistant.
C.botulinum is responsible for for botulism and is ubiquitous. The spores are incredibly difficult to kill but they are also only harmful to infants and the immunodepressed. More importantly, the bacteria are killed by heating to 85° C for 5 minutes and are also anaerobic - meaning they can only grow when they are not exposed to air. In other words, these may be present, but won't grow and produce toxins on a reasonably clean pan.
C.perfringens is one of the most common sources of food poisoning, and I believe is actually what drives a lot of the specific food safety/food inspection rules. The spores are very heat resistant (can survive boiling for a full hour), and even though it is anaerobic, it is also aerotolerant. It can grow on a pan - just more slowly than it would grow in an enclosed container or in raw/fresh food.
S.Aureus causes staph infections but also produces heat-resistant enterotoxins. It doesn't generally come from food, but more than 20% of humans are carriers, and also many pets (cats and dogs), so it is very likely in your household somewhere and can easily get on your frying pan and multiply on leftover food bits.
B.cereus is less common but very resilient (they're known for "fried rice syndrome" because they can survive steaming) and afterward produce spores and heat-resistant enterotoxins. Important to note that many strains of this are harmless - but not all.
V.cholerae causes (as you may have guessed) cholera symptoms which come from CTX and TCP toxins. Both of these are heat-labile although I wasn't able to find a time/temperature data point.
Chances are, you have at least one these friends in your kitchen. Chances are, you have several. The question is, did enough of them manage to get onto your frying pan after your last meal to produce dangerous levels of toxins?
Probably not. It's really very unlikely. But it is possible, and you're taking a chance if you don't wash your dishes.
In fact, it's probably more dangerous with eggs, because typically you'd cook eggs only up to a temperature of around 70° C, at most (63° is the temperature at which they set). So day after day you're bringing all those lovely bacteria right into the danger zone for several minutes and not staying above it for very long at all!
Doing this one time... really not a big deal. Bacteria don't get to multiply much in the 5 minutes (or less) it takes to fry an egg, and most eggs aren't even contaminated to begin with, although there have been small outbreaks.
But doing this several times, day after day - you're increasing your risk each day because the pan may never get hot enough, long enough, to kill all of the new bacteria from the day before.
I would strongly recommend that you wash your pans after use. You don't need to use soap if they're cast-iron; a good scrub with salt and hot water is fine. But please - especially if you're going to be cooking for guests - don't just leave the dirty pans sitting there and use them again the next day without washing them first. It's not safe.
P.S. I'd like to draw attention to the fact that of all of the items in the list above, botulism, despite having some of the worst symptoms, is generally the least of your worries. Both the bacteria and the toxin are easy to kill, and the spores are harmless to healthy adults. So if you hear people on this site or any site talk about food safety entirely in terms of botulism and the temperatures required to kill/inactivate the bacteria/toxin, ignore them, because they haven't done their homework.