Milk contains lactose naturally, so there will always be a minimum amount of sugar that any condensed milk will contain.
Having said the above, the recipe for making lowest-possible-sugar condensed milk, is evaporated milk (which still contains all of the lactose), so that's still sweet as well, but not overpoweringly so like condensed milk.
If evaporated milk is not sweet enough to your taste, keep adding vanilla sugar to it while heating gently and tasting frequently till it reaches the exact taste you want!
Actually, condensed milk and evaporated milk are synonyms semantically. What you're really talking about is "condensed sweetened milk" ;-)