First off I want to point out the term "fresh". While some containers might keep milk from spoiling for longer, it may not taste as nice.
Several things might be why:
1.) Plastics leach flavor and odor into the milk. Cardboard cartons are also lined with plastic, not wax since about the 1940s. I would say this is likely the biggest impact-- I've always found plastic containers make drinks and food taste downright funky.
Plastic milk jugs are made of high-density polyethylene. The FDA considers this material safe, but just because something is safe doesn't mean it doesn't impart a flavor.
Study from Journal of Dairy Science showing glass having least effect on taste:
"...overall, the development of water off-taste occurred least in
glass containers and most often in high density polyethylene
containers..."
2.) Glass bottles feel colder. Ice cold milk tastes better(fresher) to most people.
The thing is, it's not actually colder while in the fridge. Glass has a higher thermal conductivity than most plastics. That means heat can transfer through it faster than plastic. The second law of thermodynamics demands equilibrium, so while your hand makes the milk hotter, the glass of milk makes your hand colder. Because of glass's higher thermal conductivity, it does this faster than plastic and we sense the glass as being colder.
So while it's not -actually- colder and wouldn't keep the milk from spoiling for longer, it would affect our perception of how cold it is and therefore also the taste.
Packaging of milk varies so a thicker glass bottle might also keep the milk colder while it's sitting on the counter vs most thin plastic milk jugs. There are a lot of variables with packaging so it really depends on what you're comparing.
3.) Oxygen Permeability: Glass has no O2 permeability whereas most plastics do.
Starting in 1937 the Methylene Blue Dye Reduction Test used to be one of the main statutory tests for checking if milk was bad by checking how much oxygen is present in the milk. This lets you know how much aerobic bacteria is in the milk (bacteria that requires oxygen). This is actually a science experiment you can do yourself. While we do have newer, better tests for handling milk in bulk refrigeration these days, it's important to note for a long time this was a large part of measuring milk quality.
4.) It just takes time to process, test, and ship milk. You mentioned it yourself-- usually to get a glass bottle you have to get it from a dairy, something local and closer to home. So it hasn't traveled as far.
Also, while it's not legal everywhere, some people will sell raw milk in glass bottles so there isn't even time wasted processing or testing the milk. However, raw milk goes bad faster. Still, some people might prefer the taste while it's still good. Processing is done to kill bad bacteria(so it's not a bad thing), but it does take time that eats into shelf life. Milk delivered to you may only be a few hours old. The processing itself can effect the flavor of the milk. Processing plants can hold milk for up to 3 days.
5.) In many areas, the companies that bottle with glass are the ones that are making higher quality milk in the first place. Glass is expensive, and usually only people who are willing to pay more for higher quality milk are going to be the same ones who pay for a higher quality bottle. Glass bottles are good for the environment and so some people are willing to pay more for that. Refillable glass uses about half as much energy vs plastic counterpart. Note here that glass doesn't automatically mean "better milk".
6.) The way food looks affects our perception of taste.
In a 1980 study, subjects were blindfolded and asked to tell whether
the beverage they were drinking was flavored orange. Only one in five
could. But when they were allowed to see what they were drinking, each
of them identified the orange flavor. And when a lime-flavored drink
was colored orange, nearly half of respondents thought it was flavored
orange
You also see this with high-end restaurants-- they put a lot of value in how the food is plated, going as far as using a ruler to get a perfectly straight line, and wiping up small drips to make the food look perfect. Our vision influences our taste.
Since most glass is translucent and most plastic containers are not, there's a good chance this influences taste as well.