I am experimenting with increasing the amount of carrot in carrot cake. The original recipe uses 3 cups of carrots to 2 cups of flour; how much more carrot can I add in addition to the 3 cups without turning the cake into a dense carrot bar?
There is no exact limit, it's a subjective/linguistic question. With every carrot you add, the proportion of people who recognize your result as "cake" will go down and those who recognize it as something else will go up.
That being said, I now checked my preferred books for recipes that do quickbreads and muffins by the volume, and they tend to have a spread between 0.5 and 1.5 cups of add-in per cup of flour, with Ruhlmann's ratio also containing a fritter (not quickbread!) recipe at 2 cups of zucchini to 1 cup of flour. This means that, for typical tastes, your recipe is already on the high side of carrot. If you want to add more than that, do so incrementally, and don't be surprised when others dismiss it as not being a cake any more.
As mentioned in comments, you will be able to have a stronger carrot taste with a better texture if you would replace the whole fresh carrots for something more concentrated, e.g. partially dehydrated carrots, macerated carrots, or you can search for prepared carrot flavoring (some molecular gastronomy shops carry all kinds of flavoring) or also use something that is strongly reminiscent of carrots, such as a small drop of nigella oil.