Perhaps we should re-direct this conversation to be closer with the OP's intent. The OP's question is:
"if this is a real problem and if there are ways to avoid it besides buying from them"
The first question can be paraphrased as "Do mycotoxins present in brewed coffee affect mental and physical performance enough that I should spend extra money to buy mycotoxin-free beans?".
First off, the question is highly subjective - is a potential increase in mental performance worth the extra money to the OP?
There are no studies on is whether the level of mycotoxins present in coffee has any measurable effect on a person's mental and physical performance. Asprey has a theory based upon personal experience, therefore they may be worth something, but should also be taken with a grain of salt. The lack of studies on this does not mean Asprey is a fraud or a con-artist, only that his assertions should be taken for what they are - personal experience. His opinions should not be dismissed out-of-hand.
We therefore have to break the question down. Does mold exposure impact mental performance?
As far as I know, there are no easy answers here, but this study does imply a link between mold exposure and mental performance:
https://www.atlanticlegal.org/pdfs/baldo.pdf
Second, does the level of mold / mycotoxins present in coffee affect mental performance? There are no studies about this, so your only source will have to be the personal experience of people trying this for themselves (aka Asprey and his readers).
Separately, the moderators and other contributors have made it clear that the mycotoxins present in coffee (beans or brewed) fall under the FDA's limits. However, Asprey's blog and assertions are about maximizing mental & physical performance. There may very well be a different threshold between what the FDA considers a "safe" level of mycotoxins, and the level of mycotoxins capable of affecting your mental performance. It would be illogical to conclude that the two thresholds are the same. Keep in mind that the degree of change Asprey is likely talking about may be the difference between getting an A+ vs. an A on whatever measure you consider to be relevant.
In short, there are no hard & fast answers to the OP's question because no relevant studies exist. We can safely say that yes, mycotoxins are present in coffee. Do said mycotoxins affect mental performance? Unknown, but it is supported / suggested by anecdotal evidence. Is anecdotal evidence sufficient for the OP to spend extra money on coffee beans? This is the real question.