The recommendations (in the US at least) are based on a risk model, which takes into account a number of factors:
- frequency of outbreaks and occurrence of illnesses
- severity of illness, taking into account illness duration, hospitalization and mortality
- likelihood of contamination
- growth potential/shelf life
- manufacturing process contamination probability/intervention
- consumption
- economic impact
Some of these factors are statistical (frequency, economic, for example), and some are measured (growth, aspects of severity). For the measured aspects, a large number of methods are employed, including animal testing, human histories, and toxicology reports (including LD measures).
Primarily, most home cooks should follow the FDA guidelines as the recommendations take into account the supply chain and common outcomes for most pathogens (for a given food). It's important to note that live pathogens are not the only risk in food safety, also toxins (as produced by the pathogens) are an important factor depending on the specific life form. These toxins are not mitigated by cooking (generally), and some pathogens are not completely mitigated by heat or cold. The risk is related to the combined risks, time, and supply chain.
The FDA site contains the regulations, methods, and sampling guidelines for various stages of food production and food safety.
This is not my speciality, but I did teach food safety many years ago (please refer to the current guidelines and methods for current recommendations).
References:
- Designating high risk foods
- Home food safety cheatsheet
- Example toxicity studies - I'm not certain these feed into these specific guidelines, but the methodology will be similar