Vote 2020 graphic
Everything you need to know about and expect during
the most important election of our lifetimes

Scientists Raise Concerns Over Data Used in Two Major Covid-19 Studies

A scientist in Panama handling lab samples while conducting covid-19 research
A scientist in Panama handling lab samples while conducting covid-19 research
Photo: Getty Images

Update Thursday 4 p.m. ET: Lead author Mandeep Mehra and two co-authors formally retracted the study they had published in the Lancet, which originally found no evidence for hydroxychloroquine’s effectiveness in preventing covid-19 deaths. Notably, this decision did not involve co-author Sapan Desai, the cofounder of the company Surgisphere that provided the hospital data now suspected of being fraudulent. In their retraction notice, the remaining study authors stated that Surgisphere refused to share its full data with independent peer reviewers, citing confidentiality agreements. As a result, the peer reviewers were unable to conduct a full investigation and subsequently dropped out of the process. “Based on this development, we can no longer vouch for the veracity of the primary data sources,” the authors wrote. “We deeply apologize to you, the editors, and the journal readership for any embarrassment or inconvenience that this may have caused.”

Advertisement

Update Thursday 4:25 p.m. ET: The New England Journal of Medicine study, which found an increased risk of covid-19 mortality in people with preexisting heart conditions, was also retracted by all of the authors this afternoon.

Original article appears below:

Serious credibility questions are now surrounding two coronavirus studies recently published in scientific journals, including one that found little evidence for the drug hydroxychloroquine working as a potential treatment for covid-19. Critics have argued that the data used by the same team of researchers in both studies could have been falsified by the corporation that provided it. Both journals have now expressed concern about the findings while asking for the authors to prove that their data is reliable.

Advertisement

The two studies were published in the New England Journal of Medicine (NEJM) on May 1 and in the Lancet on May 22. They were conducted by members of the same team of researchers, led by Mandeep Mehra, medical director of the Brigham Heart and Vascular Center in Boston, Massachusetts. Both papers were covered by Gizmodo soon after publication.

The NEJM study found evidence that people with preexisting heart conditions were more likely to die from covid-19 than those without them. It also found no link between an increased chance of death from covid-19 and patients taking certain heart medications known as angiotensin-converting–enzyme (ACE) inhibitors and angiotensin-receptor blockers (ARBs).

The more widely circulated Lancet study found evidence that hospitalized patients taking hydroxychloroquine or chloroquine (either alone or in combination with an antibiotic) did not survive with any greater odds than those not taking the drugs; rather, they seemed even more likely to die. The study, which was purported to involve over 90,000 patients, was quickly influential, with the World Health Organization and several countries citing its results in deciding to stop ongoing studies of hydroxychloroquine over fears of endangering patients. This afternoon, the WHO announced it would restart the hydroxychloroquine arm of its covid-19 treatment trials.

The two studies relied on anonymous patient data that was said to be collected from hundreds of hospitals by the company Surgisphere. One of the company’s founders, Sapan Desai, is also listed as a co-author in both papers. But soon after the Lancet study made waves, outside scientists and critics, including James Watson of Oxford University and Australian statistician Peter Ellis, said they spotted irregularities with Surgisphere’s data.

Advertisement

Simply put, the data seems too good to be true. Surgisphere claimed to have collected meticulous electronic records from hospitals in countries where there is relatively little digital health care infrastructure, particularly in Africa. Elsewhere, the data didn’t match up with other documentation. Initially, for instance, the Lancet study reported more deaths from Australia than the official count at the time. The study’s authors issued a correction for that error, claiming they had mixed in data from Asia, but said that the findings weren’t affected. Also suspicious is the high number of cases collected from their claimed sampling of hospitals relative to the total number of cases in a region. The study’s reported North American numbers were slightly higher than the total number of reported covid-19 hospitalizations in the U.S. at the time, for instance.

As others have noticed, Surgisphere also seems to have a remarkably small footprint for having been founded in 2007. Prior to the covid-19 pandemic, no published studies had used Surgisphere’s data. And despite claiming to have secured data-sharing agreements with hundreds of hospitals worldwide, the company only has a handful of listed employees. The company has so far not disclosed the names of any of these hospitals.

Advertisement

On Tuesday, hundreds of researchers signed an open letter to the Lancet and the NEJM highlighting these inconsistencies and demanding that Surgisphere make its data available to the public and that the journals conduct their own investigations. That same Tuesday, the NEJM issued an expression of concern agreeing with the critics and calling for the authors to provide their data; the Lancet followed suit on Wednesday, adding that the results of an independent audit would be out soon.

A third study conducted by Surgisphere cofounder Desai using Surgisphere data that is not yet peer-reviewed is also under question. It concluded that the antiparasitic drug ivermectin can substantially lower the risk of dying from covid-19. Despite not being published yet, the study may have influenced governments in South America, where ivermectin is commonly used, to sanction its experimental use.

Advertisement

A representative for Desai appeared to confirm to Science Magazine on Wednesday that Surgisphere’s data wasn’t shared with the other co-authors. The spokesperson told Science that the company would soon arrange “a nondisclosure agreement that will provide the authors of the NEJM paper with the data access requested by NEJM.” Mandeep Mehra, the lead author of the Lancet and NEJM study, has not responded to a request for comment from Gizmodo.

Other studies related to covid-19 have been criticized as deeply flawed and even occasionally retracted, but this is undoubtedly the largest potential scientific scandal to emerge so far. To be clear, though, even if these studies do turn out to be fraudulent, that doesn’t mean their basic conclusions are necessarily wrong, as other evidence has suggested that ACE inhibitors and ARBs aren’t dangerous for covid-19 patients, while the overall evidence for hydroxychloroquine’s effectiveness remains mixed (some studies supporting its use have faced their own concerns over reliability).

Advertisement

Though any scientific fraud is dismaying, this debacle highlights the willingness for self-criticism within the scientific community. It took less than two weeks for these studies to be torn to shreds once the first whispers of skepticism were heard. It also illustrates the importance for everyone (science reporters included) to not take any single study’s findings as definitive.

Science writer at Gizmodo and pug aficionado elsewhere

Share This Story

Get our newsletter

DISCUSSION

While happy that the tenacious, albeit clunky, process of scientific self-policing will likely win out once again—-I am disappointed by two things 1) The Lancet—-like a lot of other prominent medical journals lately—seems compelled to publish COVID-19 studies at an accelerated clip and I wonder what that’s doing to the peer-review process. Are their editors and associate editors seeking out top/relevant experts to do the peer reviews—or is it whoever they can round up on short notice? This is not my field, but it’s easy for me to tell when I get reviewer feedback on submitted manuscripts from experts on the particular topic my study is on vs. those from tangentially related fields. And I get how expert reviewers may be in short supply right now and/or too busy to agree to be reviewers. If the rumors and accusations for this current episode are true, that SHOULD have been caught by the first round of peer-reviewers and not left to the wider community to catch. I take peer-reviewing very seriously and it takes a of time to do it right. And sometimes it requires going the extra steps to research and spot check specifics that aren’t in your wheelhouse or that no one would know off hand without some additional work.  And 2) What kind of co-authors would sign off as being on a study without having access to the primary datasets? That’s complete bullshit. I would NEVER agree to be on a study if I didn’t have complete access to the raw data and was therefore able to conduct my own analyses. In fact, if that was a stipulation, it would make me immediately suspicious.