Should you ever have your torch face downwards? When I do that I get huge flames. I try to torch it from the sides, but the sugar caramelizes very unevenly.
There are good videos on the internet showing how to properly caramelize the sugar; are you doing things differently ?
Most of them just have the torch a couple of inches facing down over the sugar and in a slow side movement, caramelize the sugar.
The torch flame needs to be "blue" so that it is at its hottest.