Butterfly pea flowers will give a nice blue or indigo color - it is sold as dried whole flowers for making tea with (out of Thailand), and also marketed as a natural blue food colorant or dye. The tea itself is interesting and pleasant, but also mild and herbal in flavor - so should be easily incorporated into or even masked by whatever other flavors you want in your dish. The tea will also turn purple in the presence of acid (like, say, lemon juice) - so you might be able to get your purple coloration from that, if you don't mind adding a lemon taste.
Blue is tricky, at least if you're trying for a taste more neutral than, say, blueberries - so the pea flowers are a good find. And you can brew much more strongly than the picture for a deep color without too, too much extra taste.
If you want a red color (which you can also add to blue for purple) I find hibiscus usually works, it gives a more vibrant red color, while beets usually end up coloring something more pink or itself appearing a duller red, almost maroon, to my eye.
Turmeric or saffron will indeed give yellows and oranges (depending on concentration). Turmeric is also used as a yellow cloth dye, so should have really good staining power for whatever you're coloring. Saffron will get to orange tints if the concentration is quite high (but it is expensive, so only if you already want the strong saffron taste if you're using that much). You might actually try carrot juice for a more consistent and less expensive orange color (or tint your red beet or hibiscus red with yellow, depending on what you want or what you have).
The combination of saffron and butterfly pea flower makes some really beautiful and vibrant greens, from teal to deep emerald - and also a tasty tea, which is what I was doing with it. I had also heard that red onion skins will create a green dye. Other answers mentioned spinach juice and spirulina and the like, so I'll leave them be.
