Would you wash your hands after handing chicken poop? How about if you were digging around in the straw or dirt where chickens live? That's where your eggs come from.
Contrary to popular belief, chicken eggs generally aren't contaminated by feces before they exit the chicken -- the two pathways by which feces and eggs come out do intersect near the "exit," but because of the geometry, it's uncommon for feces to get on the egg there. Nevertheless, if the chicken has feces on its feathers as it moves around while laying, or if feces are around the place where it's laying, there could be contamination. In close quarters where most laying hens live, it's quite common. Even in so-called "free range" situations, studies have indicated that around 30% of eggs are contaminated on their surfaces by something undesirable, from feces to residue from previously broken eggs in the area. If any of the chickens there happen to be sick with Salmonella, it's now on the eggshell.
In the UK, most eggs are not washed. Therefore, when you handle intact eggs, you are touching whatever materials the eggs touched. Just because there isn't visible soiling on the egg doesn't mean there isn't a thin film of Salmonella or other bacteria present. And in fact it is well-known that Salmonella are commonly found on unwashed eggshells.
Now, is it likely that you could get sick just from those bacteria on your hands? Probably not. The concentration is likely nowhere high enough. The main problem could occur if you unintentionally transfer some of that bacteria to a better growth medium. Studies have also shown that it is possible for bacteria on the external eggshell to contaminate the contents of the egg while breaking, and if the raw eggs are left to sit for a long time or combined with something where the bacteria can grow, illness could result.
If such things can happen during minimal and unintentional contact between egg shells and their contents during breaking an egg, it's certainly possible for your hands to also transmit bacteria (cross contamination), particularly if they were wet before you handled the egg or something and you then touch another surface or some food where the bacteria can start to multiply.
Is it likely? Probably not in most cases. In terms of "risky" kitchen behavior, it's relatively low on the list. The example I'm giving here includes a worst-case scenario. But personally I'd treat it as if you were picking up some random item dropped on the ground in a chicken coop. If you would wash your hands then, you should probably wash your hands after handling (unwashed) eggs.
But don't listen to me. Here's some info from the UK Food Standards Agency:
Remember, salmonella bacteria can be on the shell as well as inside
the egg so, to help stop bacteria spreading, you need to be very
careful how you handle eggs, both when they are still in the shell and
after you have cracked them open.
Keep eggs away from other foods. And always wash and dry your hands,
and clean surfaces, sinks, dishes and utensils thoroughly, after
working with eggs.
Also, by the way, the question mentions removing eggs from a box to store in the fridge. Most food safety organizations don't recommend that for the same reasons -- they could come in contact with other foods and/or leave residue on surfaces in the fridge. It's better to keep them in the box they came in.