You're asking about a few different attributes here:
safe/unsafe - food is safe when it's essentially guaranteed not to cause foodborne illness. Nothing is perfect, of course, but if a food safety agency says something's safe, it means there's a sufficiently small chance of illness that they're fine with telling everyone that it's okay to eat. So if something is deemed unsafe, that doesn't mean you will get sick, it just means there's some chance.
stale - stale bread is dry and leathery or tough. It's not unsafe or spoiled, just much less pleasant to eat - it's about quality, not safety. Refrigeration does indeed accelerate this process.
spoiled - food is spoiled when it's obviously inedible. That might mean it's unsafe (e.g. moldy or rotten), or it might mean it's just not something you want to eat, like really overripe fruit. Since different people have different tastes and standards, that means spoilage can be a bit subjective.
In general, leftover cooked food is safe for 3-4 days, and garlic bread is no exception (if anything it'll be safe for longer), so you were fine. Again, "unsafe" isn't a guarantee, so longer than 3-4 days doesn't mean you will get sick.
That just leaves quality, and apparently it was good enough for you to eat, so... there you go. If it was stale it wasn't bad enough to stop you, and it certainly wasn't spoiled by your standards.