In a post earlier this week on a study raising doubts about some high-profile studies in neuroscience, I was remiss in implying that the problem existed only in fMRI studies. As the paper’s lead author, Niko Kriegeskorte, reminds me, “this is not only about brain imaging (as your title suggests), but equally affects other fields of systems neuroscience,” including EEGs.
I also reported criticism that Kriegeskorte and his colleagues had not listed the studies they found to be problematic, which leaves scientists scratching their heads about what’s reliable and what's (maybe) not. The reason they did not make the list public, Kriegeskorte explains, is that their aim was “to educate, so that an alarming trend can be nipped in the bud before many incorrect claims accumulate in the literature” and not to accuse. “So we decided not to list papers. Every case is different and we could not have done justice to particular studies had we been more specific. We didn’t even want to give a percentage [of how many studies resort to the ‘double dipping’ they criticize], but one of the four reviewers and the editors were adamant about this. . . . We feel that starting a political fight with hundreds of authors is not helpful to our field—especially when most of the studies affected are likely to be correct in their overall conclusion. . . . A few bad apples can and should be listed. But literally thousands of overall good apples, each with a little brown spot, can and should not."
Since the original criticism of fMRI studies earlier this year by Ed Vul and his colleagues, neuroscientists seem to have gotten more aware of statistical pitfalls that can skew their results. But there is no question that passions are running high and that back-biting and defensiveness have set in. At least one attempt to get critics and criticized together in the same room blew up.