Milk contains [lactose](https://en.wikipedia.org/wiki/Lactose) naturally, so there will always be a minimum amount of sugar that any condensed milk will contain.

Having said the above, the recipe for making lowest-possible-sugar condensed milk, *is evaporated milk* (which still contains all of the lactose), so that's still sweet as well, but not overpoweringly so like condensed milk. 

If evaporated milk is *not sweet enough* to your taste, keep adding vanilla sugar to it while heating gently and tasting frequently till it reaches the exact taste you want!

<sub>Actually, condensed milk and evaporated milk are synonyms semantically. What you're *really* talking about is "condensed *sweetened* milk" **;-)**</sub>