In terms of food safety you can leave the door open for the same amount of time that your food can safely sit on the counter when taken out of the fridge. This is a couple of hours for most potentially hazardous foods as indicated in the answer to the linked question. This a cumulative time, so if you leave the door open for an hour three days in a row you're going to have to throw out a lot of food.
Of course this isn't really what you meant. If you're just leaving the door open while you pour yourself a glass of milk, the temperature of the food in the fridge isn't likely to reach the danger zone. How long that takes will depend on factors like the nature of the food and its location in the fridge, but I wouldn't worry about if you unless have it open for more than minute or so.
So from a food safety perspective you don't have to worry too much about leaving the door open. From an energy savings perspective you do however want to keep the length of time of the door is open to the minimum. The temperature of the air in the fridge will rise faster than temperature of the food, so every second you have the door open the fridge has do that much more work to cool it down again. This is probably the real reason why your parents told you to close the door immediately.