Classic Chevre uses a bacterial culture as well as rennet to coagulate the cheese, but that is not the only way to make a goat's cheese. The recipe you linked to doesn't have a long incubation time, so I doubt there's any intention that the buttermilk is inoculating the cheese and there's no rennet. So I think the "bacteria" idea is a red herring.
A very simple goat's cheese can be made with goat's milk and lemon juice. I have had great success with it. It does make a fairly soft cheese, though how soft depends on how long its left to drain.
The recipe I use has 1/3 cup (US) lemon juice to 1 quart milk. I'm in the UK so this works out as 2 lemons per litre. The result is quite "lemony" and you might want to use another acid source (white wine vinegar for instance) but if you keep trying you can home in on what you like.
Simple acid cheeses like reasonably high temperatures. My recipe uses 180 - 185F (which may be hotter than you have used) before adding the lemon juice.
The other thing is, its generally much easier to just let the milk sit after acidification so the curd can develop. 10 minutes is usually enough, but you can always leave it 20 or 30 minutes if the curd is slow to set. The guilty kitchen recipe goes straight to ladling out the curds. That is something you do for a ricotta or high acid cheese (which may sort of be what they are aiming for - I don't cook with buttermilk myself so don't have the experience) but I'd want to let things set a bit first with a simple goat's cheese.
So: warm slowly to 180-185F, add lemon, sit, strain through cheesecloth to the texture you like.
Crumbly may be an indication that its not setting long enough?