The most significant factors are probably:
- Muscle function and activity
- Fat content and distribution
- Diet of animal
Basic muscle structure in most vertebrates is somewhat similar, so taxonomic relatedness is often not paramount in matching flavors. (There appear to be some phylogenetic connections in flavor, particularly among many "beefy" animals, though.)
There's a reason that many people say lots of random meats taste like chicken. Since muscle structure is often quite similar even in very unrelated animals, a lot of reptile and amphibian meat can "taste like chicken." The texture is often different, because muscles may be used in different ways, but if you put the meat through a meat grinder, the flavor wouldn't be as varied. (Thus alligator sausage is not as adventurous or odd tasting as you might think at first.)
What makes "light meat" vs. "dark meat" often has a lot to do with muscle function/activity, and if you try similarly colored meat from different animals, it often has similarities in flavor. Ostrich meat tends to be dark and "beefy," while many parts of pigs are advertised as "white meat" which is presumably closer to birds. I remember my first experience a number of years ago eating a smoked turkey leg -- it was smoked for a very long time and cooked slowly for many hours. It tasted almost exactly like ham. And it was like ham: a leg muscle from an animal with similar fat distribution in that area which was processed in the same way.
The "gaminess" of wild animal meats often has to do with (1) increased age, (2) different/more varied diet, and (3) more muscle activity than most domesticated animals get. In my experience, for example, farm-raised venison is distinctly less gamey than wild venison. Moreover, most of our farm-raised animals these days tend to be slaughtered rather young (generally as soon as mature), so as to maximize profits. If you encounter older meat from old farm animals which are slaughtered late in life, it often has "gamier" or "deeper" flavors than the young animals we are now used to. We also tend to breed domesticated animals to maximize muscle content in certain desirable areas of the animal (e.g., chicken breasts) which often tends to make them more "bland" and less gamey, because the animals don't generally need so much muscle in those areas and thus never use them much.
(It's notable that tastes have changed significantly over time. Years ago, it was common to hang wild game for days or weeks until it began to rot in a process called mortification. While it tenderized the meat, it also had the effect of heightening the "wild" gamey flavors of each individual meat, which was considered desirable by many. In most industrialized cultures today, the trend is toward blander and blander meats, with young, lean animals the norm, and the boneless skinless chicken breast -- with the flavorful fatty skin and bones/connective tissue removed, produced from chickens which never exercise and eat uniform bland diets -- as one of the most common meat sources.)
Muscle texture and fat distribution can also create distinctions, even if flavors are not that different. Most muscle tastes somewhat similar, so it's often the fat that makes the big difference, since the fat tends to store energy from whatever food sources the animal has. As Harold McGee describes it in On Food and Cooking (p. 134):
Fat: The Flavor of the Tribe The machinery of the red and white muscle fiber is much the same no matter what the animal, because it
has the specific job of generating movement. Fat cells, on the other
hand, are essentially storage tissue, and any sort of fat-soluble
material can end up in them. So the contents of fat tissue vary from
species to species and are also affected by the animal's diet and
resident gastrointestinal microbes. It's largely the contents of the
fat tissue that give beef, lamb, pork, and chicken their distinctive
flavors, which are composites of many different kinds of aroma
molecules. The fat molecules themselves can be transformed by heat
and oxygen into molecules that smell fruity or floral, nutty or
"green," with the relative proportions depending on the nature of the
Thus, wild or older animals are more likely to have different sources for fat (and more marbled fat mixed into muscle), which gives it a more distinctive ("gamey") flavor. Fish meat tastes a lot different mostly due to fats/oils distributed throughout (instead of in pockets like most higher vertebrates), and the structure of the muscles tends to create a flaky texture due to different usage and less connective tissue. There are also distinctive chemicals (particularly in the fat) that give fish its characteristic "fishy" flavor, but when you encounter very "mild" fish, it's the texture and fat distribution that makes it most different from another white meat.
One last thing I would mention is expectations. Many, many studies have shown that pre-existing assumptions can have a great impact on our experience of flavor in food. (Expensive food tends to be rated as "tasting better," for example, even if the same exact dish is sold at different prices.) The combination of a "weird meat" expectation and a slightly different texture can often make the distinctions between meat flavor seem greater than they are. If you put the various meats (of similar color) through a grinder and then prepare them the same way, you might be surprised at how small the differences are between various animals.