Here in science-writing land, when it comes to biomedicine we try hard to stick to rigorous, vetted evidence. That means studies published in reputable journals by, ideally, scientists with no financial or ideological stakes in what they’re investigating. Testimonials and anecdotal reports of patients who swear by some new remedy don’t count (except when we need a “real person” to liven up a dry medicine story).
So you can almost hear science writers emitting a loud collective groan this evening. It turns out that doctors and other researchers have been a trifle, well, selective in which studies they publish on antidepressants.
This conclusion emerged when scientists led by Erick Turner of Oregon Health & Science University and the Portland Veterans Affairs Medical Center compared studies in medical journals to studies submitted to the Food and Drug Administration. The difference? When a manufacturer asks FDA to approve a new drug, it has to submit all studies on the drug's safety and efficacy. But no such law compels anyone to publish those studies where you and I can read them.
Lo and behold, when Turner and his colleagues compared the two batches of studies—the uncensored whole, in FDA’s files, vs. the selected-for-publication subset—on 12 widely-prescribed antidepressants approved between 1981 and 2004, involving 12,564 patients, the mismatch was jarring, they report this evening in the New England Journal of Medicine. In the published literature, 94 percent of the studies concluded that the antidepressant worked better than a sugar pill. In the FDA files, 51 percent of the studies were positive. Of 36 studies that were not positive, 33 either were not published or were spun in such a way as to seem positive.
Turner put it this way in a statement: “Selective publication can lead doctors and patients to believe drugs are more effective than they really are, which can influence prescribing decisions." Based on all the studies and not the cherry-picked ones, each antidepressant was less effective than the published literature made it seem—and that stories in Newsweek and everywhere else that rely on that published literature conveyed.
It isn’t clear, says Turner, whether negative studies get deep-sixed because of the file-drawer effect (authors and sponsors don’t bother to submit manuscripts, for reasons I’ll let you infer), or because journal editors and reviewers decline to publish the negative studies that they do receive. I think he’s being too kind. With the proliferation of journals, virtually anything can get published somewhere—maybe not in the NEJM, but in some third-tier rag, which would still count as “published.”
The result of this selective publication is no less than a distortion of science and—since these are studies that drive what doctors advise their patients to do and what patients ask for—a perversion of the biomedical system in which untainted results are supposed to benefit public health. As Turner said, “doctors and patients must have access to evidence that is complete and unbiased when they are weighing the risks and benefits of treatment.” As things now stand, they do not. So next time you read or hear a story in the media about the wonders of a new drug, stop a minute and ask which contrary evidence might be moldering in a file drawer somewhere.