Everything I bake with baking powder is bitter. If I use self rising flour or corn meal there is no bitter taste. So why do so many recipes call for baking powder?
Baking powder is a mix of a powdered acid and base. It's stable when dry and cool, but in the presence of heat and moisture they will react with each other, creating gas. This gas gets trapped in the structure of a dough or batter as it is baked, causing it to rise.
Self-raising flour has baking powder mixed in it, so when you use self-raising flour you are using baking powder. I suspect your problem comes from one of two things, you either have bad baking powder, or you are using way too much. I would buy fresh baking powder of a different brand, and check to make sure you are using the right amount for your recipes. Check to make sure you have the right measuring equipment. 1 teaspoon (tsp) is 5ml volume, and 1 tablespoon (tbsp) is 15ml. If you added 1 tablespoon instead of 1 teaspoon of baking powder you would get a result exactly as you describe.