Yes, it matters a lot. When you are separating egg whites, it is for whipping them into a foam. This foam is a protein-based foam, relying on protein ends hooking into each other. Even small traces of fat will prevent the foam from forming. Egg yolks contain high amounts of fat. Once an egg yolk breaks in your whites, you have to start the separation anew, because it can prevent your foam from forming. Also, don't use plastic bowls for whipping egg whites (their surface retains some fat molecules even after washing, giving you a less stable foam) and only whip with a cleanly washed whisk or mixer attachment (not one you have just used for something else, not even if you wiped it clean).
To prevent big mishaps, just separate each new egg in a teacup and only add the new white to the old whites after it has separated cleanly. Else you are in big trouble if you are separating a lot and the last egg contaminates the whole whites with yolk. And a single contaminated egg is easily reused for a quick egg-and-feta sandwich or something else.
Contamination the other way round isn't so worrisome. You still want to work as precisely as possible, as yolk-only recipes will often have somewhat worse texture if eggwhite is included, but small contaminating amounts are usually not noticeable in the finished product, even in foams (zabaglione, mayonnaise). This is because yolk foams are fat-based, and small amounts of protein don't prevent a fat foam from forming.