In 2005, epidemiologist and physician John Ioannidis published the controversial and famous paper, Why Most Published Research Findings Are False in PLOS Medicine. He showed there were sound reasons to be skeptical of most published research, especially those results that rely on statistical analysis to draw conclusions about the relationship between two phenomena. Familiar examples:
• How effective is a drug?
• Does a substance in the environment increase the incidence of cancer?
• What is health effect of a food?
As he suggests, the science in most of these cases is far from settled.
By 2014, Ioannidis’s fame had grown to pop-science level. He was invited to do one of the Talks at Google, where he was given a fawning introduction by the earnest and callow member of the “People Operations Team.” He can do no wrong.
The Google talk is a nice introduction to the replication crisis. Starting at 7:30, he goes over some of the results in nutritional epidemiology. Of fifty randomly selected foods, almost all showed both increased and decreased cancer risk, often by remarkably large amounts. Eating one serving of potatoes per day could double your cancer risk, or decrease it by half. The relative risk (RR)* values for various foods spanned the range from 0.1 to 10 in his survey of fifty foods. He claims that the best studies find RR values almost indistinguishable from 1.0 (e.g., 0.998). While such a tiny effect is important for a large population as a whole, it is meaningless for any individual.
Flash forward to the year 2020. Ioannidis published an article in Stat in mid-March questioning the data quality for the WuFlu and the measures being taken in response. He noted that we didn’t really know the denominator in calculating the fatality rate for WuFlu.
The data collected so far on how many people are infected and how the epidemic is evolving are utterly unreliable. Given the limited testing to date, some deaths and probably the vast majority of infections due to SARS-CoV-2 are being missed. We don’t know if we are failing to capture infections by a factor of three or 300.
This evidence fiasco creates tremendous uncertainty about the risk of dying from Covid-19. Reported case fatality rates, like the official 3.4% rate from the World Health Organization, cause horror — and are meaningless.
Based on limited information available in March, Ioannidis conjectured that the fatality rate would be far lower, 0.125% with a large confidence interval based on “extremely thin data.” For this, he was vilified as a COVID denier or COVID minimizer. The evidence is mounting for his hypothesis based on results of antibody testing at various locations across the US, including work by his group at Stanford. The infection fatality rate in the latter work was estimated to be 0.17%, not far from his original conjecture. He summarizes his results and addresses some objections in this short video from two weeks ago.
Most of you have probably already heard of the work on antibody tests, which imply higher infection rates and consequently lower fatality rates. The point of this post is not to rehash that tired topic. The unifying theme is that Ioannidis follows where the data lead him. Conclusions are subject to revision based on new information, which is precisely what is not happening in the response to the WuFlu. Policies are still based on the information available in March.
The science is never settled — except possibly in this case.
*Relative risk (RR) of 2 means double the risk of the disease; RR of 0.5 means half the risk.