As an addition to @GdD's answer.
Refridgeration slows down the mentioned processes, but it doesn't stop all of them, so the time in the fridge does count towards spoiling, though more slowly than time outside the fridge.
Assume some food lasts 10 days in the fridge or 1 day without refridgeration. It may be that it will last half a day out of the fridge if it spent its first 5 days in the fridge.
It may be that this is simply what is meant by your family wisdom: the food isn't as fresh after refridgeration as it was before.
In reality, the relationship between spoiling processes and time is not linear, and highly dependent on the process we're looking at. So the result could be anywhere between time in the fridge does not shorten shelflife out of the fridge at all and there's almost no time left until spoiled:
- extreme example: an unopened pack of food such as UHT milk or a tetra pak of tomatoes that could be stored at room temp almost indefinitely does not suffer in the fridge)
- simple chemical processes like fat going rancid are often slowed down to ≈ half speed by decreasing the temperature about 10 °C. So if you keep your butter at +8 °C in the fridge for a week, it will have gained approximately as much rancidity as if you'd kept it 3 1/2 days at 18 °C. And every day in the fridge will count roughly like 12 h out of the fridge.
- microbial spoiling is highly nonlinear with temperature.
I cannot think of any situation where refridgeration would prolong shelf life at room temperature after taking the food out of the fridge.
A different point is that food you put into the fridge (e.g. leftovers) is not immediately down at fridge temperature, so there will be a "history" of room temperature and cooling, which further decreases remaining shelflife.