When it comes to pairing specific ingredients, the general thinking seems to be that ingredients that share "flavour notes" in significant quantity will go well together.
In practice it goes much deeper than that, because in addition to taste, foods have very specific and recognizable aromas which affect the perception of flavour; wine is mostly sweet and sour, but you wouldn't substitute ordinary grape juice for it; the aroma of the alcohol is unmistakable, and even between different wines you will have varying levels of "fruitiness" or "woodiness".
But let's start with tastes:
Chocolate and coffee go well together because they share characteristic bitter notes. The roasting of the beans also makes a difference; the pairing works best when the roasting is enough to thermally degrade the bitterness without losing too much sweetness due to caramelization. This emphasizes the sweetness and de-emphasizes the bitterness and acidity of the coffee, making it closer to the flavour profile of chocolate.
A traditional stock combines a mirepoix (onions, carrots, celery) with meat (well, bones), both very high in umami. The end product may have varying notes of bitterness or saltiness from the vegetables but is basically one huge kick of umami. A lot of people also enjoy peanut butter and bacon together, which combines the significant umami and saltiness in both foods.
Beets and sour cream are a traditional (or so I'm told) Polish dish, both obviously sharing strong sour notes. Often chives are added, which are mildly pungent in nature and pungency tends to go with sour or bitter (since it is created by sulfur which is perceived as sour, bitter, and also "metallic").
This tends to work reasonably well for basic pairings, but another very strong element of this is balancing all of the different flavours in a dish.
Peanut butter and jam don't seem to have much in common, flavour-wise. But when you combine them together in a sandwich you have the high umami and saltiness of the peanut butter, the sweet and slightly sour notes of the jam, and the hints of bitterness and/or sourness in the bread. Together you have strong notes of all five tastes and the result is appealing to most people.
Five Spice is usually a mixture of cinnamon (sweet), star anise (mildly sweet and bitter), fennel (mildly sweet and fairly savoury), ground cloves (pungent and somewhat bitter), and Szechuan peppercorns (salty and pungent with a sour aftertaste). The quantities are generally adjusted to balance out the tastes; for example, fennel is the only really savoury ingredient so Five Spice tends to have more of that than other ingredients. Anyway, this is used all over Chinese cuisine as a sort of "wonder spice".
A really good roast chicken starts off high in umami and is then brined (salty), buttered (sweet), and roasted which involves the Maillard reaction (bitter). This is missing sour, which is why a lot of cooks will throw a few lemon slices in there.
But truthfully, even though this all sounds plausible, it's really so much more complicated than that. Even though there's some science at work here, the five tastes are a little like the four elements or humours; it's an almost archaic way of thinking about food chemistry, because our mouths and noses can detect so much more than that.
If you want to see just how little we actually really know about flavour today, check out They Go Really Well Together on Khymos, linked to from one of the related questions. There you'll see totally nonsensical combinations like chocolate and garlic or mint and mustard that actually - with the right preparation of course - produce pretty pleasant flavours/aromas.
The truth is people are still gathering data on this and we don't really know, exactly, what makes some flavours work together. At least we're starting to approach this scientifically and actually experiment and document this stuff, but there's not enough data to form a coherent theory yet. Check the answers to my question about flavour pairings and the other question on pairing resources if you want detailed data on which flavour pairings can "work", although you might not get much of an explanation for why.
