If there are any doctors who are congratulating themselves for basing their treatment decisions on rigorous clinical evidence--not the questionable studies on things like homeopathy that my colleague Jerry Adler wrote about in his health column this week--they can stop. Physicians and even biomedical researchers may say they rely solely on the gold standard of randomized clinical trials, but they’re as likely to believe much weaker studies as is the most desperate cancer patient combing the Web for laetrile.
So finds an alarming study published this afternoon in the Journal of the American Medical Association. Scientists led by John Ioannidis of the University of Ioannina School of Medicine in Greece find that well-known claims based on poor studies are repeated over and over again in medical journals even when better studies contradict them. The earlier, refuted conclusions continue to pop up in medical journals like satanic whack-a-moles.
It’s important to understand why some ways of doing a study are better than others. Less-good are observational studies. In these, you basically watch and measure a group of people who are doing something related to their health--women taking estrogen for menopause, say--and compare them to a similar group who are not doing that thing. In this example, observational studies concluded that estrogen reduces women’s risk of heart disease, Alzheimer’s and other bad things. But then a rigorous study looked at the same question (the effect of estrogen), by randomly assigning some women volunteers to take estrogen and others to take a dummy pill. Lo and behold, estrogen suddenly didn’t look like such a good idea: it raised the risk of heart disease, stroke and breast cancer, the Women’s Health Initiative reported in a blockbuster announcement in 2002.
Why the different results in the two kinds of studies? One reason is that in observational studies, women who chose to take estrogen are inherently different from women who did not. They were, de facto, seeing a doctor (for prescriptions if nothing else), and were likely wealthier and better educated. Those three factors alone, not the estrogen, might have made them healthier than non-estrogen women, distorting the results.
The well-publicized negative verdict on estrogen makes it hard for biomedical researchers to slip a sentence into their papers off-handedly saying estrogen protects against heart disease. Not so with other studies. To wit:
Two observational studies in 1993 suggested that vitamin E could decrease the risk of developing cardiovascular disease by some 40 percent. But randomized trials in 1996 and 2000 showed no benefit or even increased harm. Yet papers in medical journals continued to cite the dubious findings. In 2005, 50 percent of articles mentioning vitamin E and cardiovascular disease cited the favorable study, leaving readers with the erroneous impression that vitamin E helps prevent heart attacks and stroke.
A 1981 observational study claimed that beta-carotene can prevent cancer. Randomized control studies in the mid-1990s refuted that, finding that taking beta-carotene actually increases a person’s risk of developing cancer. Yet 10 articles that mention the positive 1981 study do so favorably, while three are equivocal and only three are (correctly) unfavorable.
A 1996 observational study claimed that estrogen can reduce older women’s risk of developing Alzheimer’s. A randomized control study in 2004 refuted that. Yet 29 articles that cited the 1996 study mentioned it favorably, while 14 were equivocal and only four were unfavorable.
It’s difficult to fathom why scientists can’t tell a rigorous study from a dubious one and keep citing the latter. It’s fine to start with observational studies, especially to generate hypotheses. But once a randomized control study shows that the compound or other treatment originally deemed beneficial is not, surely it’s time to stop saying otherwise. That’s especially true when studies are available to anyone with an Internet connection to PubMed. By continuing to make health claims for vitamin E and heart health, beta carotene and cancer, estrogen and Alzheimer’s, and who knows what else that has been refuted, scientists mislead the public and promote dangerous treatments.