Why it’s important to look for the limitations in coronavirus studies
This week, I got to talk about vaccines and vaccine information with Natalie Dean, and The Verge’s Nicole Wetsman and Nilay Patel on The Vergecast. It was a great conversation with Dean, who is an assistant professor of biostatistics at the University of Florida and an expert in infectious disease epidemiology who has worked on designing vaccine trials.
After the podcast, I got to ask something that had been on my mind: how did she — a person who works with scientific data every day — feel about the flood of early versions of scientific studies being posted online? And how much attention should the general public be paying to them?
Before we get to her answer, let’s take a quick step back to talk about what these early studies are. Scientists call them preprints, and they are studies that get posted online before they’ve been published in a scientific journal. They haven’t been fully vetted by other experts (a process called peer review). Preprints are helpful to scientists because they quickly put data out into the world that could be useful to other researchers.
The downside? They’re essentially first drafts, and conclusions could change after the researchers get feedback. Balancing the need for speedy data with the need for reviewing work before it goes public is especially tricky during the pandemic — and both the pace of publication and attention to preprints has definitely amped up in the past several months.
“There already were these preprint servers before, but they’ve never been used to this degree,” Dean says. For a long time, prestigious journals would refuse to publish a paper if it had been posted elsewhere, which meant scientists kept their research under wraps until they had their final draft.
That’s started to change over the last few years as journals have loosened their restrictions. Now there’s a flood of coronavirus-related papers available online, and some can attract a lot of attention — and confusion. Not all of that attention is a good thing, as Kelsey Piper wrote for Vox back in May:
The newer, faster pace could mean that badly flawed pre-prints get widely shared and covered in the media, fueling the spread of misinformation and forcing other scientists to waste valuable time by publicly debunking papers that would ordinarily be rejected in the peer-review process.
Piper pointed out that there were plenty of advantages to preprints, too — they force data to move at 21st-century speeds instead of staying stuck in the 19th century, and when lots of experts are actively participating in the process, they can cut out a ton of unnecessary downtime.
That’s especially true in a pandemic. “My feeling is the benefits outweigh the risks in the situation where time is of the essence, and we’re all learning from each other,” Dean says. And the pace of academic publishing is so slow, that I just don’t think it is still viable.”
But there’s still that one big disadvantage. “Scientists know how to evaluate the limitations of preprints. But it’s challenging for the public to do so,” Dean says.
She’s got a good tip to help readers start assessing news stories about preprints: look for the limitations. A story that tells you what the study doesn’t is helpful. This doesn’t just apply to preprints, either: it’s good advice for any story about a scientific paper. All studies have limitations; good reporters tell you what they are.
In other words, look for the peer review in a news story, especially when it comes to a preprint. Look for it on Twitter, where Dean and other researchers have thrown cold water on extraordinary claims. Look for it in dedicated volunteer efforts, like Johns Hopkins’ 2019 Novel Coronavirus Research Compendium which actively assesses papers that get a lot of media attention. Look for it at Rapid Reviews: COVID-19, an open-access journal explicitly devoted to peer-reviewing COVID-19 preprints. And most of all, as we keep wading through a flood of information, take a beat to look skeptically on claims that seem incredible.
“The more extreme a result seems, the more evidence we need to support that,” Dean says.
Here’s what else is going on this week:
Research
Around 2 percent of Red Cross blood donors have COVID-19 antibodies
Only a small number of blood donors had antibodies to the coronavirus, a study from the Red Cross found. That indicates “that very few people in the United States have been exposed to the virus that’s ravaged the country,” Nicole Wetsman reports.
(Nicole Wetsman / The Verge)
Coronavirus May Increase Premature Births, Studies Suggest
New studies from the Centers for Disease Control and Prevention found some evidence that coronavirus infections in pregnant women could increase the rate of premature births, but other experts say a lot more information is needed.
(Roni Caryn Rabin / The New York Times)
Development
Two Pharma Companies Released Their Coronavirus Vaccine Blueprints In A Bid To Regain Public Trust
This week, both Moderna and Pfizer released their “blueprints” for their respective clinical trials. The documents cover how the companies have mapped out their trials, and they provide a window into the vaccine development process at a moment when trust in that process is waning. (Stephanie M. Lee and Dan Vergano / BuzzFeed)
How to Ship a Vaccine at –80°C, and Other Obstacles in the Covid Fight
To get ready for a vaccine rollout, FedEx and UPS are installing freezers at hubs around the world. Also, there’s a dry ice shortage.
(David Gelles / The New York Times)
Lilly’s Covid-19 antibody helps some patients rid their systems of virus sooner in early analysis
Early data from a treatment study involving manufactured antibodies had some promising early results. But the results are very early and have not been peer reviewed.
(Matthew Herper and Damian Garde / STAT)
Perspectives
Pandemic isolation has killed thousands of Alzheimer’s patients while families watch from afar
In an effort to protect patients from the coronavirus, nursing homes have closed their doors to visitors. It’s been a devastating road for families trying to care for patients with Alzheimer’s or dementia. (William Wan / The Washington Post)
I Am Not a Brave Person. I Am Also Patient 1133.
Writer Molly Jong-Fast writes about her experience as a volunteer in Pfizer’s vaccine trial and why she chose to participate in the first place:
I am a normal, un-brave person. I am also a part of history now, part of a small group that may be protected from the deadly virus, or who may not be, but whose experience will have something to teach others. I no longer feel I’m simply waiting around to get sick.
(Molly Jong-Fast / The New York Times)
More than numbers
To the more than 30,316,394 people worldwide who have tested positive, may your road to recovery be smooth.
To the families and friends of the 948,367 people who have died worldwide — 198,306 of those in the US — your loved ones are not forgotten.
Stay safe, everyone.
This week, I got to talk about vaccines and vaccine information with Natalie Dean, and The Verge’s Nicole Wetsman and Nilay Patel on The Vergecast. It was a great conversation with Dean, who is an assistant professor of biostatistics at the University of Florida and an expert in infectious disease…
