It makes a big difference what you are baking as to how much impact opening the door will have. A victoria sponge does not have any chemical leavening agents in it, all the lift you get comes from the expansion of air and moisture from the heat of the oven. When you open the oven door you let in cold air and lose some of your moist, hot air. Worst case is that the cold air could cause the cake to contract or possibly even collapse, best case is that you lose some of your lift and get a denser cake. The moisture loss makes a hard crust form sooner, constraining the lift of your cake and/or making the crust thicker and tougher. It works the same way for bread. Cakes/pastries which are chemically leavened are a bit less sensitive but they still get part of their leavening from the expansion of air.
Some things are much more sensitive to the door opening than others, opening the door too early with popovers and yorkshire puddings is a great way to ruin them, for example. Opening the door after yorkshire puddings stop rising helps them form a better crunch (that's possibly controversial, some say a crunchy yorkie is heresy). Cornbread is less sensitive, I've opened the oven plenty of times when I'm baking that in the oven with other things and it's always come out well because it's relatively dense to begin with.
There are a few cases where opening the oven door is beneficial early or mid-way through the process, one being bagels. Because bagels are boiled in water before being baked their structure is pretty fixed before they go into the oven, so opening the oven halfway through won't hurt, in fact I've found that a quick opening of the door can help form a better crust on them. When baking focaccia I often open the door and spray water into the oven to keep the crust from forming too quickly.
So in general it's good advice to keep the oven door closed as long as you can when baking most types of cakes, pastries, and bread. Once a crust forms and the rise stops it's safer to do, but limit it if possible to prevent uneven baking.