I follow all the recipe guidelines, but very often my cakes crumble and tend to be either moist of heavy inside although they appear nice brown and light outside and the tester comes out clean. I have a doubt that it is because I remove the cakes from the oven immediately as they are cooked.
Most cake recipes I have seen give an approximate baking time and also instructions for determining doneness by observation (the "toothpick comes out clean" sort of guideline). When the criteria for doneness are met, it is assumed you will take the cake out of the oven immediately.
Very often there are further instructions about cooling, removing from the cake pans, frosting, etc. But they don't expect you to leave the cake in the oven (whether turned "off" or "on") past when it is done. Leaving it in the oven while the oven is cooling down would just introduce too much variability as ovens would cool at different rates.
I agree with your speculation that cake problems must be from some other cause.
Recipes will usually specify a timing that assumes the cake will be removed from the oven and left at room temperature to cool, unless otherwise stated. Anything else would yield results vastly dependent on how quickly a given oven cools down - which is again dependent on oven volume, insulation and design. Also, in case of a convection oven, you would create an even more complex temperature profile because you would bake with convection but finish in a slowly cooling still oven.
When it's cooked it should come out. Otherwise it's just over ooking. Best of all, it should come out marginally before its finished and continue with risidual heat but that's if you really want to take things to the next level.
Yes, you should take the bread out as soon it is baked. Let it cool for 5 minutes and de-mould it. Keep it on a wired rack otherwise the base will become soggy.