I've followed the recipe for an opera fudge (and several others) as close as I can, yet my fudge always turns out grainy and crumbly. The best results I've gotten have come from not washing down the sides of the pan, but transferring the mixture to a new pan instead.
In Scotland we make a kind of fudge that is deliberatly hard and crystallised known as Tablet. This was a popular treat when I was growing up. Essencially the recipe for tablet, soft fudge, toffee and caramel are quite similar. The difference is made by how you cook and treat the mix as it cools.
Essentially you need to know about sugar boiling points. There are two important levels used: soft boil, and hard boil. My cook books suggest using a sugar thermometer to get the perfect boil but I've never found a sugar thermometer in the shops. Instead I rely on a great deal of practice, the colour, the texture and the drip of the mix to gage how hot the sugar is.
In a soft boil (234–240 °F or 112–115 °C), if you drip the mix onto a very cold surface or into cold water, then touch the drip with your finger, it will be soft like caramel. Additionally the colour will change from cream to yellow/tan.
In a hard boil (250–266 °F or 121–130 °C), dripping again onto a cold surface or into cold water, then touching the drip with your finger, it will be firmer or even hard. The colour will darken slightly to a dark yellow or light brown.
For caramel and fudge you go to a soft boil then cool. For tablet and toffee you go to a hard boil.
The next part is the cooling. This is equally important as the boiling. The faster you cool the mix the smaller the crystals become. The slower, the larger. If you want a caramel or a toffee, you must avoid agitating the mix as it cools. For fudge you should stir the gently as it cools slightly before pouring onto the try. For tablet, the mix should be vigorously stirred until stiff then poured onto a try.
What you're seeing is crystalization. Fudge is such a picky animal. Here is a site called the Science of Cooking that walks through exactly how to avoid grainy fudge: