The Limits Of Evidence And The Need For Open And Robust Scientific Debate
With The Benefit Of Real Life Examples
Covid cases are currently plummeting in the US as the Omicron Surge wanes. While absolute case numbers remain high, we are seeing a nearly 50% week over week decline. This is happening all over the nation. It is happening in places with mask mandates and places without. It is happening in places with high vaccination rates and those without. It is happening in places with proof of vaccination requirements and places without.
It is happening whether you have a school mask mandate or not. It is happening whether you provide random testing or not. It is happening whether you have virtual or in person learning. It is happening whether you are contact tracing or not.
What I am about to say, I say with no compelling statistical analysis beyond what is right in front of your eyes:
Omicron has broken through every line of defense implemented with the idea that it would be controlled and then subsided regardless of whether anyone cared to try and stop it. There are now many places with low vaccination rates, no meaningful mitigation whatsoever, and minimal if any deviations from normal behavior. These places are still declining.
This trend is almost undeniable at this point. The biggest surges up front were in NY, which has some of the most aggressive interventions in the country. It then surged in Florida, which has some of the least aggressive interventions. It then jumped all across the nation in a fairly predictable fashion regardless of state interventions. As of yesterday, there were more cases per capita in San Francisco with its’ mask mandates, proof of vaccination rules, high school vaccination rates, and aggressive contact tracing than there were in my rural Pennsylvania county with a vaccination rate of less than 40% and none of those other things.
The Limitations Of Public Health Intervention
We have made a lot of mistakes in our pandemic approaches. When I look back, I can largely break the most egregious in to three categories. Errors in the first two frequently drove errors in the third. These can be broken down in to errors in the application of evidence, errors in the assessment of harm from an intervention, and political errors in the mechanisms of scientific debate.
Errors In The Application Of Evidence
Modern medicine thrives on evidence. Every medical student takes some sort of course (or courses) on the interpretation of data. Those principles are then further ingrained with the review of scientific papers on different clinical rotations. This application is often referred to as evidence based medicine.
Evidence based medicine has led to a largely positive revolution in the field. This has allowed practice to follow data that should theoretically help reduce bias by provide an objective standard. I want to be clear that my following statements are not an argument against evidence based medicine. They are an argument against the misapplication of evidence.
When one designs a study, it is typically set up to answer a specific clinical question. In fact, the more questions you try and answer with one study, the less likely you are to end up with an effective study. This is important, because it also means that the value of the data obtained may very well be limited to the clinical question the study is designed to answer.
Now, a study may very well give some insight in to questions that go beyond the study population or question. We often ask whether a study can be generalized to the clinical scenario or population in question. Unfortunately, this isn’t always the case.
The Most Common Errors In The Application of Evidence
One of the problems we have consistently encountered during the pandemic is that we have no history with this virus whatsoever. All studies are limited by the timeframe over which the study has taken place. This has led to a lot of evidence that can only truly be applied to a short time period.
I’ve written in the past about this limitation, but I think we really need to look at it closer. Early justifications for everything from lockdowns to masks to school closures were all based on studies that looked at relatively short time periods. What we’ve of course learned is that a lot of these interventions have a limited life span. I’ve previously posted links to the Bangladesh Mask study for example that showed that the effectiveness of the intervention tended to drop off over time. Lockdowns were unsustainable, and unless the virus was entirely eliminated from a nation, it came roaring back the moment they were undone.
When we see a study that says that “intervention X saved 6,000 lives,” what we are really saying is that we modeled that there would be 6,000 fewer deaths over a certain timeframe based on the observation of a certain population. Even if the modeling were sound, it still says nothing about what happens once that timeframe ends.
Another common problem with the way we interpreted evidence was a tendency to improperly generalize data from one group to another. This error shines most egregiously in our original vaccine studies. We specifically excluded individuals with prior infections from those studies, and we thus have zero randomized controlled trials on the benefit of vaccinating those with a prior infection. Nevertheless, we’ve generalized that benefit repeatedly, often looking at low quality studies that show marginal absolute benefits. We will come back to this, but this may end up being the single most egregious error that we’ve made.
We’ve similarly tended to cohort risk groups. When we are looking at relative risk reduction, we need to have a general understanding of what the absolute risk is. If there are no deaths in a certain risk cohort, you cannot reduce deaths in that cohort no matter how good your intervention is. Similarly, if the mortality is very low, the potential reduction is also very low.
I’ve often used the example of Aspirin to reduce the rate of heart attack in coronary artery disease. If I gave everyone with coronary artery disease an Aspirin, I’d likely reduce the rate of heart attack. If I gave it to everyone over 50, I’d probably reduce the risk of heart attack by some additional amount, but I’d substantially increase the likelihood of a side effects. If I gave it to everyone over 18, I’d likely show a lower rate of heart attack, but it would be no better than just giving it to everyone over 50, and I’d subject every person between 18 and 50 to the potential for side effects with limited benefit.
We’ve seen the above a lot in our pandemic calculations. We take large groups of people and cohort them together. A healthy 18 year old who has already survived Covid has a mortality rate that is likely less than 1/10,000th of that of an unvaccinated 80 year old, but we often put them together. They are not the same.
Errors In The Assessment Of Harm
It is important to point out that most of our COVID pandemic interventions were entirely unprecedented. We never had widespread worldwide lockdown. We never masked school children for months to years on end or closed all learning institutions for months. We never mass vaccinated the world population against anything within a year. We never vaccinated a human with an mRNA vaccine before.
The reason that this is important, is that we had no real understanding of the long term harm from any of these interventions when we rolled them out. We did have some data from prior studies that might have allowed us to get at it, but even these were largely ignored. A lot of the time, we would treat the absence of evidence of harm as the absence of harm. When one has never before sought evidence, this of course is not knowable.
It takes time to understand harm, and sometimes harm doesn’t manifest right away. If a school intervention leads to a drop in productivity in adulthood, we won’t know that for years. If a novel therapeutic is associated with the development of some pathology down the road, there is simply no way to know that. Let me state the obvious: we still don’t know the extent of harm from most of our interventions.
Political Errors In The Mechanism Of Scientific Debate
Let me again state the obvious: trust in our public health institutions is at an all time low. Most of our institutions have backed themselves in to a corner where there is no way for them to legitimize themselves without angering the portion of the population that still supports them. This requires a closer look, because it probably has little to do with the specific recommendations they made at the beginning.
There are now numerous well documented attempts to block counter arguments from qualified scientists. I wrote a piece a while back titled, “Science As Propaganda,” in which I spoke about the seriousness of the shutdown of minority voices in scientific debate. It often looks like we determine what we want to do first, and then try to justify it with science. This is of course entirely backward. We should first try and understand the science. The science then plays a role in the response, along with other political and ethical concerns that we all understand are part of every intervention.
By pretending that interventions are entirely based on science when we know that they’re not, we don’t make the interventions appear more legitimate. In fact, we tend to delegitimize the scientific entities. By doubling down with poor scientific studies, we worsen the problem. We’ve dug a deep hole, and we just keep digging.
Attached is an editorial addressing just one example of the attempt to politically squash legitimate scientific debate:
https://www.beckershospitalreview.com/hospital-management-administration/viewpoint-dr-fauci-and-dr-collins-shut-down-covid-19-debate.html?fbclid=IwAR3KUgk2lnIy7r-7FfRlhNJVPpd9KqHgKzGp3b6gQ4HMgdN7f46Z--NUMH0
Below here is an example of a previously respected physician tearing apart a low quality CDC “study.” He actually calls it borderline propaganda in the takedown:
These studies that the CDC publishes have done very little to convince anyone of anything. Those who already believe strongly in the value of the intervention still agree with it. Those who are skeptical already are pushed further out the door. They then lose credibility when they endorse a well done study.
Similarly, at least in America, we’ve managed to turn one’s position on these scientific matters tribal. We’ve taken things that should have been publicly debated from 100 perspectives and broken them down in to left-right politics.
Let’s Look At Some Real Life Examples
I am going to attach a number of news articles or studies that are directly related to the above concerns. While one might be able to guess my personal opinion on a lot of these issues, what I really want to point out is that they were profoundly more complex than they were made out to be.
Did Lockdowns Actually Work?
At least in the US, no one has been talking about lockdowns since the Winter of 2020. Even then, we actually saw some targeted business closures in select states. Nevertheless, substantial economic interventions persist even in the US. They are typically much more aggressive in other parts of the world. The following study attempts to answer that question:
https://sites.krieger.jhu.edu/iae/files/2022/01/A-Literature-Review-and-Meta-Analysis-of-the-Effects-of-Lockdowns-on-COVID-19-Mortality.pdf
This is a multinational study. It essentially argues that the mortality benefits over time to lockdown policy were extremely limited, while the costs were astonishingly high. Most of our studies on lockdown suffered from both errors in the application of evidence (short time frames, poor generalizability) and errors in assessment of harm. It is now irrefutable that lockdown policy has led to substantial economic harm.
Now, this study has been making the rounds, and there are some criticisms. A common criticism is that it was put together by economists. I’ve never understood why public health officials were given broad license to comment on economics, but economists have been given no quarter to comment on public health. I think this particular argument simply falls flat.
On the other hand, a legitimate potential argument would be that this study fails to account for the timing of lockdown, and early intervention may very well have led to lower mortality in certain locations. That may be, although it would then require us to understand where that timing cutoff is. In other words, it would require an even more complex analysis.
The harms are much harder to ignore. Even if we accepted the limitations to the mortality assessment, it is very clear that economic harm has occurred. Thus, even a more significant mortality benefit would need to be weighed against those harms.
I personally took a lot of flak early on in the pandemic for this very concern. In fact, any concern over lockdown was shot down politically. Legitimate scientists with concerns were frequently censored or had their reputations attacked. It is very clear we never allowed for open debate on these profoundly impactful policies. The seeds of distrust were sown all the way back here.
Vaccine Immunity Is Not Superior To Naturally Acquired Immunity
I’ve already spoken at length about this topic, but I’m attaching another news article that just came out yesterday that addresses the durability of natural immunity:
https://www.npr.org/sections/goatsandsoda/2022/02/07/1057245449/the-future-of-the-pandemic-is-looking-clearer-as-we-learn-more-about-infection?fbclid=IwAR1lkfj95FUQNdKJxXbTRxYYfETwyjbwmDnUQOPO_cWlHCbfsY7_Pg6JsRQ
There is a legitimate debate over whether the addition of some number of vaccine doses enhances the immune impact of a prior infection, but it is also pretty clear that any prior infection leads to durable immunity in otherwise healthy people. That immunity now according to numerous studies is both longstanding and more robust that vaccine immunity alone.
Now, this doesn’t mean that people shouldn’t be vaccinated. It doesn’t negate the potential benefits of vaccination in those without a prior infection. It doesn’t say that one might not obtain some small additional absolute benefit if previously infected. It simply doesn’t answer any questions about durability beyond a couple of years. It also doesn’t necessarily tell you about the immune benefits over time to variants.
So of course, you’ll still need a variant specific vaccine right? Maybe. The following study suggests that the original vaccine is just as effective against Omicron variant as an Omicron specific vaccine in animal models:
https://www.biorxiv.org/content/10.1101/2022.02.03.479037v1?fbclid=IwAR38m_DZvc-pTxWSkyXS7bt2pKehpnFMcCEGeNrmEi0NiIMg6nfnQGxEEvQ
And just to be clear, this is the same vaccine that generates less durable immunity than a prior infection. Right on cue, we also have a number of studies now showing the reduction in severe disease and reinfection from a prior infection is at least as robust as vaccination alone.
For me to say that six months ago was borderline heresy. You’d be at some risk for losing your medical license. You were certainly blacklisted. It is now a mainstream opinion. What did that do for the legitimacy of the entities trying to punish those who held it six months ago? A lot of people haven’t forgotten.
What Are The Value Of Alternative Treatments?
I’ve largely remained out of the debate surrounding therapeutics like hydroxycloroquine and ivermectin. I have no personal experience with these drugs, and support tended to rely on low level evidence. I still don’t have a strong opinion at this point.
That being said, I’m attaching a study out of Brazil that argues that ivermectin use can reduce mortality by 90%:
https://www.researchgate.net/publication/358386329_Strictly_regular_use_of_ivermectin_as_prophylaxis_for_COVID-19_leads_to_a_90_reduction_in_COVID-19_mortality_rate_in_a_dose-response_manner_definitive_results_of_a_prospective_observational_study_of_a
There are a number of limitations to this study. Every population level retrospective study has a number of limitations. Of course, these limitations are no greater than the limitations we often see with studies on the value of everything from masks to booster doses of vaccine. We’ve subjected these therapeutics to a different standard than more favored drugs and interventions. This has made it very difficult to properly study and debate.
This is an egregious error for a couple of reasons. If ivermectin really is this effective, then we’ve let people die for no reason. That alone is egregious. If it isn’t this effective, then our aggressive shutdown of those arguing in favor of it has convinced large numbers of people that there is a conspiracy to shut it down. This has led to people taking inappropriate formulations and made proper treatment more difficult. That is still egregious. Our entire handling of it has been egregious.
We Continue To Ignore Minority Harm
I’m attaching a new story about fentanyl deaths. This is important, because in spite of all of the focus on Covid, more young people are dying of opioid overdoses than have ever died of Covid:
https://www.usatoday.com/story/opinion/2022/02/06/fentanyl-deaths-young-americans-fear-and-depression-drive-overdoses/9314171002/?gnt-cfr=1
When we initially implemented our lockdown policies, we continuously ignored clear and obvious spikes in suicide, suicide attempts, and drug overdoses in young people. In fact, every single marker of health in young people has been negatively impacted by our pandemic interventions. We then tended to try and shut down those pointing out this objective truth by saying that there was no rise in suicides in the US. That’s true, unless you look at young people. There was no rise in excess deaths in youth during the pandemic, but that didn’t stop us from being concerned about older cohorts.
Why wouldn’t we also be just as worried about young people? Why aren’t we MORE worried about harm coming to young people? A death in a young person loses a lot more years of life.
It is also a bit ironic that more youth are dying from opioid overdoses than Covid. This started as another medical intervention to which harm was initially ignored or downplayed. It was another intervention in which those attempting to warn about the risks were deplatformed. It is another intervention where the harm didn’t become clearly visible for many years.
Of course, opioids aren’t the only source of harm to youth. Here is another study showing profoundly negative cognitive impacts on children related to pandemic interventions:
https://www.medrxiv.org/content/10.1101/2021.08.10.21261846v1.full.pdf?fbclid=IwAR2XN8IHabLr5Mu6ZbQdAGznS3iiCQZydTmPRpgqGeE1a8vNH-XlT0FIBgY
This is another concern that was brought up early on and largely suppressed. It was another harm that was ignored. It was another harm that took time to be visible.
We Are Attempting To Change How We Calculate Covid Deaths And Hospitalizations
I’m attaching an article that addresses a number of changes in our pandemic response:
https://www.politico.com/news/2022/02/07/biden-covid-hospitalization-data-recalculate-00006341?s=04&fbclid=IwAR282jimjvfK5VLVDjsitffnehdIrlZnhbxSDGXMBHEu2u1Wah9hQD2iRPQ
I applaud this, because we all know numerous cases where a patient is admitted for something else and tests positive for COVID. Many hospitals test every admission. There is a non-trivial number of COVID hospitalizations that have been recovered for weeks and are still persistently testing positive when being hospitalized for completely unrelated things. Some simply are never symptomatic.
It Doesn’t Stop Here
We are now seeing changes in guidance on everything from cloth masks to school interventions. I applaud a lot of these changes. The thing is, that a lot of the data used to justify these changes has been known since near the beginning of the pandemic. Cloth masks have always had limited utility. School closures have always threatened harm to children. Extended quarantines have always been disruptive. I could go on.
Perhaps we should have allowed those concerns in to the public debate from the beginning. If we ever want to bring legitimacy back to public health, we need robust debate. We need a real mea culpa.
We also need to stop with the endless barrage of new mandates and interventions. Current mandates tend to focus on vaccines. We are seeing the same pattern of looking at short term data, failing to look at risk cohorts, ignoring harm, and then politicizing the data while stifling dissent.
Early on, I said that I would come back to our failure to recognize natural immunity in vaccine studies. It has been clear from the beginning that natural immunity led to substantial protection from reinfection and severe disease. This has in fact never really been a question. Reinfection does clearly occur, but that is also typically true for coronaviruses. The NPR article that I posted above references a number of insights in to the fact that waning antibody levels are normal. In fact, immunity as it relates to SARS-CoV-2 has been really boring and predictable. It has been similar to just about every other coronavirus.
Failure to recognize natural immunity has led to a number of egregious harms that are going to be difficult to undo:
Having had a prior infection is almost certainly the number one reason why healthcare workers haven’t voluntarily been vaccinated. We have fired thousands of workers in a crisis who were objectively LESS likely to spread the disease than the vaccinated workers for failing to take this medical intervention.
Failure to recognize prior immunity makes this look much more like a conspiracy to make people get vaccinated than a genuine public health campaign to reduce risk. It may be bureaucratically more difficult to count a prior infection, but it would legitimize the process a whole lot more. This has led a lot of people who would likely benefit from vaccination to avoid it over this concern.
Recognizing a prior infection would have been the obvious move to reach across the aisle politically on vaccine policy. Failing to do so will inevitably entrench political tribes on this issue. Of equal importance, the tribe arguing against vaccination actually has plenty of science on their side. This means that individuals with some reasonable sophistication may come to the conclusion that they need to support entities with additional goals that they would otherwise support less over this issue.
It really does violate all prior understanding of medical ethics to try and force someone to take a medical intervention that they don’t clearly need. Remember that those with a prior infection were excluded from the original vaccine studies. We still have no prospective randomized studies on this population at all, and our retrospective studies suggest that they are low risk.
In Conclusion
We are still making the same mistakes that we made at the beginning. We are ending up with the same results. We have all of the economic harm of lockdowns. We have all of the morbidity and mortality of Covid. We have profound political division. We’ve additionally manage to delegitimize our public health institutions, which leaves us vulnerable in the future.
It is beyond time to acknowledge these failures. Evidence based medicine has limits prescribed by the studies performed. Minority opinions are sometimes correct. Harm may occur with any intervention, and that harm may not be initially obvious.
Without these acknowledgements, we are never going to break the division that has led us to what is almost certainly an outcome consistent with the worst of all worlds. A good solution starts with the truth.
The majority of people hunger for the truth. Excellent article and I appreciate it.
Excellent article. Thank you for everything you do!