Debunking THAT Cochrane Review article on masks

Mar 03, 2023

white and yellow plastic pack — Photo by Jonathan J. Castellon on Unsplash

Welcome to the Covid-Is-Not-Over newsletter! My hope with this newsletter is to help the Covid-aware community with the resources we need to navigate an increasingly Covid-denying world. In other words, things to share with our networks and things to read ourselves to prepare for difficult conversations.

Today it’s all about masking. To be precise, the infamous Cochrane Review article from a month or so ago that’s basically been interpreted in the mass media as proving that masks are useless on both the individual and population level. Bret Stephens from the New York Times is perhaps the most notorious example.

This is clearly wrong. Masks work, good masks work better. The science is definitive on this. Mandates and policies where people wear good masks all or most of the time will work in reducing the burden of Covid on society and for individuals. Mandates where people aren’t consistently wearing good masks in all contexts and spaces aren’t going to be as effective as they could otherwise be. If you wear a mask, there will be less of a chance of catching any airborne disease like Covid or the flu. (See my blog for the science, here and here)

The solution isn’t to give up on masks, it’s to get people wearing better masks more consistently in a wider variety of contexts. The challenge is that mask mandates and policies have non-scientific challenges. If there’s a mandate or policy in place, the effectiveness is related to how people follow the mandate or policy. If there’s low compliance, it won’t be as effective in lowering overall transmission. However, if compliance is high, there will be a decrease in transmission.

So: Don’t give up on mandates or policies. If you have a one in your organisation (or even a “strong recommendation”), work towards building compliance. Work towards getting people to wear better masks. That’s the challenge.

Below are a good selection of the articles I could find debunking or criticising the Cochrane study. It’s quite a long list, to say the least. Sorry if there’s more links and excerpts that I would normally want to include in one of these newsletters, but I think it’s useful to present a wide variety of viewpoints. I hope that among all the various points I’ve highlighted, you’ll be able to find the information you need to counter misinformation in your networks.

I also want to mention a couple of excellent twitter threads discussing the Cochrane Review article by Trisha Greenhalgh and Tomas Pueyo. There are also a couple of resources on why randomised controlled trials aren’t that appropriate for interventions like masking here and here.

Good luck!

Don’t believe those who claim science proves masks don’t work by Lucky Tran / The Guardian

The analysis is flawed because it compares apples to oranges. The paper mixes together studies that were conducted in different environments with different transmission risks. It also combines studies where masks were worn part of the time with studies where masks are worn all the time. And it blends studies that looked at Covid-19 with studies that looked at influenza.
If apples work and oranges don’t, but your analysis mixes them together, you may come to the false conclusion that apples don’t work. Out of the 78 papers analyzed in the review, only two actually studied masking during the Covid-19 pandemic. And both of those found that masks did protect wearers from Covid-19. But these studies are drowned out by the greater number of studies on influenza included, where the benefit of masking is harder to detect because it’s a far less contagious virus than Covid-19.

Yes, masks reduce the risk of spreading COVID, despite a review saying they don’t by C Raina MacIntyre, Abrar Ahmad Chughtai, David Fisman, Trish Greenhalgh / The Conversation

The authors of the Cochrane Review acknowledged compliance with masking advice was poor in most studies. In the real world, we can’t force people to follow medical advice, so RCTs should be analysed on an “intention to treat” basis.
For example, people who are prescribed the active drug but who choose not to take it should not be shifted to the placebo group for the analysis. But if in a study of masking, most people don’t actually wear them, you can’t conclude that masks don’t work when the study shows no difference between the groups. You can only conclude that the mask advice didn’t work in this study.
There is a great deal of psychological evidence on why people do or don’t choose to comply with advice to mask and how to improve uptake. The science of masking needs to separate the impact of the mask itself from the impact of the advice to mask.
Mask-wearing goes up substantially to over 70% if there is an actual mandate in place.

Q&A: What a major review does and does not tell us about masks and COVID-19 / Healio (Interview with Dr. Linsey C. Marr)

Healio: What does the review not tell us?
Marr: The review does not tell us how well these measures work on an individual basis. It doesn’t mean that masks don’t work or that hand-washing doesn’t work for an individual. The studies instructed people to wear a mask or wash their hands, and they did not actually ensure that subjects did it properly and all the time.
A major limitation is that many of the mask studies took place in health care settings, and the health care workers wore masks only when interacting with patients. The health care workers probably did not wear masks at other times, like at home or when interacting with people in the community — for example, restaurants, shopping, socializing — and they easily could have picked up infections in those other settings.
Thus, the effect of wearing a mask would be diluted, like if you told people to use condoms occasionally but not always. An individual can achieve a much greater reduction in risk than suggested by this study.
One of the authors’ key conclusions that I want to highlight is “relatively low adherence with the interventions during the studies.” This means that people didn’t actually wear a mask or wash their hands as instructed. I’m fairly certain there would have been a larger effect if adherence had been better. So we can conclude that low adherence to mask wearing means you won’t see a benefit populationwide. Surprise, surprise.
Healio: Do you think masks protect people from SARS-CoV-2 infection?
Marr: There is no question that a high-quality, well-fitting mask can reduce the risk of SARS-CoV-2 infection if worn properly and consistently. We know this from a study in California by the CDC that found that individuals who consistently wore an N95 or KN95 had an 83% lower risk of testing positive for COVID-19 compared with those who didn’t mask. Those who wore a surgical mask had a 66% lower risk, and those who wore a cloth mask had a 56% lower risk — although there was enough variability with cloth masks that the difference might not be significant [because] some cloth masks are better than others.
We know from physics that a good quality mask acts like a filter, trapping almost all the virus particles in the air that you would otherwise breathe in from the air around you. This reduces your dose of virus and reduces the chance that you might become infected.

COMMENTARY: Wear a respirator, not a cloth or surgical mask, to protect against respiratory viruses by Lisa M Brosseau, ScD, CIH C. R. MacIntyre, PhD Angela Ulrich, PhD, MPH Michael T. Osterholm, PhD, MPH / CIDRAP

The Cochrane review by Jefferson et al states that respiratory viruses spread as follows: "People infected with a respiratory virus spread virus particles into the air when they cough or sneeze. Other people become infected if they come into contact with these virus particles in the air or on surfaces on which they land."
This is the classic definition of droplet transmission, which focuses solely on symptomatic coughing or sneezing that produce large droplets propelled into the face of someone nearby. This review was also focused on contact transmission, which has been ruled out for SARS-CoV-2 by scientists and public health authorities like the CDC.
There is no mention of airborne or aerosol transmission, the former being defined as inhalation of "droplet nuclei" at long distances from a source, and the latter representing a more up-to-date understanding of infectious particle inhalation both near and far from a source. An RCT by MacIntyre et al showed that even for infections assumed to be droplet transmitted, N95 FFRs prevent infection, while surgical masks do not. This again points to the droplet paradigm being incorrect.

Multiple studies show that face masks reduce the spread of COVID-19; a Cochrane review doesn’t demonstrate otherwise / Health Feedback

The variable quality of the RCTs included in the Cochrane review mainly results from the difficulty of conducting robust randomized controlled trials on mask effectiveness. On the one hand, trials conducted during periods of low virus circulation—like many of the RCTs included in the Cochrane review—are generally underpowered because a low number of infections make the statistics less reliable. But on the other hand, preventing a group of people from using face masks amid a pandemic to evaluate the effectiveness of this measure at preventing infections would be considered unethical.
The problem is that low-quality or inadequately designed RCTs are less likely to detect an effect, which might be misinterpreted as evidence that masks don’t work. That is why some epidemiologists consider that a mix of well-conducted and more comparable RCTs and observational studies might produce more reliable results than pooling data from variable-quality RCTs addressing “quite different questions”

Revisiting the Science: Do Masks Stop COVID-19? by Rebecca Watson / Skepchick

The Cochrane authors’ conclusions are valid, if your MAGA uncle would have bothered to read that far: “The high risk of bias in the trials, variation in outcome measurement, and relatively low adherence with the interventions (i.e. people just didn’t wear masks) during the studies hampers drawing firm conclusions.” And “There is a need for large, well-designed RCTs addressing the effectiveness of many of these interventions in multiple settings and populations, as well as the impact of adherence on effectiveness, especially in those most at risk of (acute respiratory infections).”
It’s absolutely okay if the end result of a systematic review is “Hey, we need more data to fully understand the efficacy of this intervention during a pandemic,” and I really wish they had read the room and made that the obvious takeaway, here. I wonder how much of this review was influenced by past criticisms from scientists who say Cochrane takes far too long to make useful recommendations: personally, I’d rather they take their time and get it right rather than cobble together studies that aren’t actually relevant to our current situation. We were already facing an uphill battle when it comes to convincing people to do the very basic, obvious good thing of just putting a damn mask on when you’re inside public places where case counts are high. If we can’t even get the “gold standard” of systematic reviews to understand the problem, I’m afraid this battle is lost.

Do masks work? by Katelyn Jetelina and Kristen Panthagani, MD, PhD / Your Local Epidemiologist

Only included randomized control trials (RCTs). This is typical for meta-analyses as RCTs are the “gold standard” for scientific questions. But these RCTs had a number of problems and, given the limited number of RCTs on COVID-19, do not represent the totality of evidence (i.e., see all studies above).
Combined different viruses. When a virus is less contagious, an effect is harder to detect. Many of the RCTs evaluated influenza, which is far less contagious than COVID-19. This means that if we combine them, the impact of masks may be underestimated. (Another scientist, separate from this review, removed the flu studies and reran the meta-analysis. He found masks protected against SARS-CoV-2.)
Combined settings. Studies ranged from suburban schools to hospital wards in high‐income countries, crowded inner city settings in low‐income countries, and an immigrant neighborhood in a high‐income country.
Only asked one question. Does wearing a mask protect me? This ignores other important questions.

That Viral Op-Ed About Masks Is Based On Flawed Research, Frustrated Experts Say by Katie Camero / Buzzfeed News

But the experts we spoke to said these reviews aren’t immune to bias and can have limitations, especially this one.
For example, most of the studies included in the review were conducted before the COVID pandemic during “non-epidemic influenza periods,” such as the H1N1 pandemic and other flu seasons up to 2016. Just six studies were done during the current pandemic but spanned many different countries, including Mexico, Denmark, Bangladesh, England, and Norway, and many different settings, including suburban schools, hospital wards, and low-income neighborhoods.
It’s a classic example of “low-quality studies with poor conclusions getting lumped with higher-quality studies that should be valued more,” said Dr. Dean Blumberg, chief of pediatric infectious diseases at UC Davis Health.

The new scientific review on masks and Covid isn’t what you think by Kelsey Piper / Vox

The new Cochrane review paper strikes me, and may strike you, as something of a scientifically irresponsible way to represent these findings. It gets at one of the core challenges of science: There is no methodology that can straightforwardly find answers in messy study data without many judgment calls by scientists, who are humans with their own strength, weaknesses, and eccentricities. A meta-analysis, after all, can’t meta-analyze itself.
“I was really surprised that the Cochrane group let this go through,” Jake Eaton, a public policy and global health researcher who was the lead researcher on a Cochrane review of childhood nutrition, told me. “The fact it’s looking at masks across different settings and with different diseases makes it really tricky. Cochrane reviews are very good if you really want to assemble the most rigorous evidence and say, ‘Do we have a conclusive signal that this works?’ This is something of a perverse use of a Cochrane review.”
The big problem is the sheer scope of the question: not “does masking reduce Covid transmission during a pandemic?” but “does masking help against all respiratory illness across contexts?” Because of the questions it chooses to consider and the ways it chooses to aggregate its evidence, it has arrived at a mask-skeptical conclusion with limited applicability to the hot-button question we all care about: whether there’s evidence for masking during high-transmission periods in the Covid-19 pandemic.

Column: COVID deniers claim a new study says mask mandates don't work. They should try reading it by Michael Hiltzik / Yahoo!Finance / LA Times

Stephens tries to dodge the weight of empirical evidence by suggesting that politics inevitably made mask mandates in the U.S. "a bust."
He writes that "there was never a chance that mask mandates in the United States would get anywhere close to 100 percent compliance" — never mind that even compliance at far, far lower rates would help reduce the spread of the coronavirus.
He blames "American habits and culture," "constitutional limits on government power," "competing social and economic necessities" and "the evolution of the virus itself." Along the way, Stephens takes an ignorant and utterly unwarranted swipe at the CDC, which he calls "mindless."
This is all abject twaddle. Nothing about American habits would have interfered with more masking, if conservative politicians hadn't declared that it was a violation of American values. Nothing in the Constitution obstructs mask mandates any more than it obstructs seat-belt laws or, indeed, the Affordable Care Act's insurance mandates (the Supreme Court has said so).
What "competing social and economic necessities" or "the evolution of the virus" have to do with mask mandates, Stephens doesn't say. We can take that as a sign that he doesn't know. As for the CDC, to whom should you turn for professional advice about public health — Stephens or the CDC? Given Stephens' record of dispensing unfounded claptrap, detailed above, to ask the question is to answer it.

Thank you for reading The Covid-Is-Not-Over Newsletter. This post is public so feel free to share it.

My Blog has a bunch of COVID Information posts you can find here:

Oh, hey, and why not subscribe! See you again soon!

The Covid-Is-Not-Over Newsletter