Skeptics have undermined the public-health response with dubious research
The chart above illustrates that COVID-19 is much deadlier than the seasonal flu, as it features faster transmission and higher death rates. This has been the general consensus among public health experts and epidemiologists since the early days of the outbreak. However, a vocal minority of naysayers have steadily worked to undermine this consensus, fueling resentment towards stay-at-home orders. This chorus includes politicians, academics with no background in epidemiology, smart people with little sense, and even doctors and researchers who ought to know better.
Lockdowns speak louder than words
The Chinese witnessed the virus’ deadly effects firsthand after COVID-19 spread largely unimpeded for at least a month in Hubei province. China responded aggressively by imposing a strict, military-backed quarantine of the region in late January.
News of a lockdown in China wasn’t enough to concern President Trump, who said on February 26 “This is a flu. This is like a flu….we’ll essentially have a flu shot for this in a fairly quick manner.” By this time, Italy’s outbreak was well underway and just over a week later, the Italians locked down their most economically productive region and soon after, the rest of the country. This was no easy decision for Italian authorities, as the country has suffered lackluster economic growth over the past decade.
Skeptics: cases are missing, don’t believe what you see
Despite such drastic actions, skeptics remained unconvinced about COVID-19’s deadly nature. Some pointed out the severity bias, the fact that new stories and data are more likely to showcase many of the worst case scenarios. Other discussions revolved around how case fatality rates (confirmed deaths / confirmed cases) are biased because we can’t be sure about the number of cases in the denominator if testing is hard to come by and many cases are asymptomatic.
On March 17, Dr. John Ioannidis wrote:
The most valuable piece of information…would be to know the current prevalence of the infection in a random sample of a population and to repeat this exercise at regular time intervals to estimate the incidence of new infections. Sadly, that’s information we don’t have.
In the absence of data, prepare-for-the-worst reasoning leads to extreme measures of social distancing and lockdowns. Unfortunately, we do not know if these measures work.
It’s true that deaths are more likely to be recorded than cases, but given what we witnessed in Italy with doctors being forced to decide who gets treatment and who doesn’t, it also seemed unlikely that we’d miss so many cases that COVID-19 ended up being merely as deadly as the flu.
Skeptics collect their own biased data
In early April, Ioannidis and a team of researchers (including one of the founders of the health care analytics firm I used to work for) took it upon themselves to see just how prevalent COVID-19 is within Santa Clara County, CA, one of areas with a known early outbreak. Their research found that the prevalence of COVID-19 was low in the county and that there are likely many uncounted cases, leading them to wrongly conclude that the infection fatality rate (true deaths / true infections, not just those confirmed) is just higher than the flu.
There are some well reported issues with this study and some that haven’t been touched as of yet. The first issue is that using Facebook to ask people if they’d be willing to be tested is likely to attract a high proportion of people that think they were exposed to COVID-19, making it more likely to generate positive results. This a far cry from the random sample Ionnidis demanded in his earlier writing, though the researchers try to adjust this sample by controlling for whether people reported coughing/fever symptoms. Second, since the prevalence of COVID-19 was estimated to be low in the area (2.8% weighted, with some adjustment for test performance), it’s possible that false-positives can outweigh true positives.
The study’s test kit is stated to have 3/371 false positives (the initial publication only listed 2/371), which gives a nearly 0.8% false positive rate with a 95% confidence interval of 0.2% to 2.3% (author’s calculations). That means, it’s possible that many, if not most of the cases they detect could be false.
What hasn’t been touched on is the fact that antibody test-kits have not been sufficiently validated. Nearly half of the U.S. population has been exposed to cytomegalovirus (CMV), and it’s one of the factors known to trick COVID-19 tests into thinking they’ve found a positive case, but manufacturers haven’t published results based on sufficiently high proportions of samples with CMV. Indeed, researchers have found that many commercially manufactured antibody tests have far higher false-positive rates than they let on.
What does this all mean? The denominator (number of cases) in the infection fatality rate calculation from the Santa Clara study is unreliable. But what the authors also haven’t touched on is that the numerator (the number of deaths) is also missing data. They account for deaths that are likely to occur though haven’t happened yet (time lag), but they don’t account for the odds that deaths are counted at all in real time.
The New York Times and the Economist have reported that there are significant numbers of deaths above what is expected on average (deaths in excess of the baseline). The New York Times’ count has the U.K. missing 34%, Spain, 33%, and France 31%. The Economist’s data shows Northern Italy is missing 48% of the true deaths. In New York City, that number is 29%.
The Santa Clara study noted that 94 deaths were counted by April 22nd, three weeks since the initial study (which accounts for time lag) and they estimate that 54,000 residents had been infected (with 95% probability the true value is between 25,000 and 91,000). This translates to an infection fatality rate of 0.17% (95% CI, 0.1%-0.37%). Their 0.17% estimate is 1.7 times that of the seasonal flu (0.1%). But what about accounting for deaths that likely won’t be counted in real-time?
Let’s assume that the rate of uncounted (missing) deaths ranges from 15%-50%, with a reasonable value being in the 30% range. This all depends on whether doctors are looking out for COVID-19 and whether there exists an ability and willingness to test people. That means on the low end, the true deaths are 111 and at the high end, 188, with a reasonable value of 134. This would alter the infection fatality calculation to be 0.25% (134/54,000), with a low of 0.12% and a high of 0.75%. Notice the range of values — 0.12% is on par with the seasonal flu, 0.25% means 2.5 times the flu, 0.75% means 7.5 times more deadly than the flu.
How can policy-makers make responsible decisions using data from a study with such a wide confidence interval? The first step would be to compare these numbers to other research for external validation. New York City has also also recently undergone testing that suggests that around 1 in 5, or 2.7 million have been infected by COVID-19. That is an infection fatality rate of 0.63% (12,000 confirmed deaths + 5,000 probable deaths over 2.7 million cases), which is 6.3 times the flu and far higher than the reported numbers in the Santa Clara study. Given that the prevalence is so much higher in New York, the 0.63% estimate is more reliable than the Santa Clara study and less likely to be wildly affected by issues with test kits.
Infection fatality rates don’t capture the full threat
Infection fatality rates say how likely one is to perish given they were infected. But we know that COVID-19 is far more transmissible than the seasonal flu, which means many more people are at risk of being infected by the coronavirus. CDC researchers estimate that on average, 8.9% of the population gets the flu every year, and that number jumps to 10.4% without a vaccine (author’s calculations using CDC data).
Even with most of the country in lockdown for the past six weeks, it’s estimated that 13.9% of New York state has had COVID-19 . That means, if the lockdown had any significant impact on transmission (this is addressed below), it is likely that COVID would have infected at least twice the number of people that get the flu annually. Thus, even if COVID-19 is as deadly as the flu on a case by case basis, it would still kill at least twice as many (likely more) people as the seasonal flu because so many more people would be infected.
To close the loop on transmission rates, one measurement of lockdown impacts is people’s rate of mobility, measured by how often they ask for walking directions on their phone (walking is more consistent across countries than driving or transit directions). On average, people in the U.S. have reduced their mobility and potentially, transmission, by 60% versus baseline (lower right panel). This is consistent with other reporting on the subject.
Proclaiming dubious results to the world
In 2018, MIT researchers conducted a study that showed false news spreads many times faster than the truth, likely because of the novel (made-up) nature of such information. The COVID-19 epidemic has been fraught with the same dynamics. In the wake of the Santa Clara study, doctors with no epidemiological background chose to run with these results (nearly 1 million shares), claiming in spite of all we have seen that COVID-19 is just the flu (5 million views). They were wrong but their views were spread widely as more evidence that shutting down public life was a foolish decision.
Real-time science is difficult
I’ve spent most of my career as a researcher and data scientist. Even when moving at the glacial pace of academic research, it’s incredibly difficult to do everything just right to design a study and gather/analyze data appropriately. This task becomes exponentially harder in-real time when there are life-and-death policy decisions to be made and there are powerful political forces at play. This inherent difficultly means that every piece of news needs to be viewed with a healthy dose of skepticism, with a backdrop of what makes sense. A single study or report that flies in the face of common sense and established, scientific consensus should not be enough to overturn policies aimed at saving potentially hundreds of thousands of lives.