Since you didn't say you have a problem with the first rise, I will assume that your yeast weren't simply dead.
There are a few factors that go into rise times:
- How much yeast?
- How much water is available to the yeast?
- How much of yeast-inhibiting ingredients such as salt and sugar?
If there isn't enough water in the recipe or if the water is tied up by salt and sugar the yeast will only be able to divide slowly.
- How warm is the dough?
Yeast, like everything else, work faster in a warm space.
Therefore, to increase the rising speed, you could make a wetter dough, add more yeast, use warm milk, put the dough in warm oven, reduce the sugar, etc.
It may be that your recipe just takes longer to rise.
Alton Brown's cinnamon roll recipe has a lot of fat and sugar and not as much water. The recipe recommends making them the night before and letting them rise overnight. Dough that rises longer generally tastes better.
I would recommend letting the dough go in a warm place and see how long it takes to reach the fluffiness you like. If that takes too long you could start tweaking the variables above.