Hah, I get to cite my new copy of McGee's "On Food and Cooking" for the first time! There are several things going on here (all of which can be found in the 2004 edition of McGee, most on page 50).
Firstly, as Nathan indicates in his answer, most of the liquid that is sold as buttermilk these days is in fact not "real" buttermilk, but so-called "cultured buttermilk", made from ordinary skim milk and fermented until acid and thick.
Secondly, even traditional buttermilk was somewhat sour, though less than cultured buttermilk. To find out why, let's examine the processes that were involved in making it. If you want to make butter, you start by separating milk into cream and whey. Since the 19th century, we typically do that with a centrifuge and it goes quite quickly, but before then, we would leave it to gravity. That would take a while, and the milk would start to ferment while it happened - especially in warmer environments. This fermenting is essentially lactic acid bacteria converting lactose into lactic acid, which sours the milk a little.
So in either case, you would have more lactic acid in butter + buttermilk than in butter + whole milk.