Voodoo That Scientists Do

Peer Review / by Jon Bardin /

When findings are debated online, as with a yet to be released paper that calls out the field of social neuroscience, who wins?

Page 2 of 2

Vul admits that his choice of language was intentional. “Some of the wording was probably a bit more provocative than was needed to draw people’s attention,” he says. Though reflective on the implications of the language, Vul is open about what his goals were. He and Kanwisher had previously written a similar paper discussing the statistical point on its own, and it went largely unnoticed. This time, he said, “we wanted to make the paper entertaining and to increase its readership. We wanted our paper to have some impact. If people don’t know about these statistical problems, nothing will be done to fix them.” As Vul points out, his approach seems to have worked: The paper is being widely discussed across the neuroscience community.

How much can we learn from an fMRI?

Since the debate erupted on the pages and in the comment sections of blogs and online newspapers, the editor of Perspectives, Ed Diener, in conversation with Vul and his rebutters, has decided to strike the word “voodoo” from the paper’s name. Yet for Wager and social neuroscientists, it feels like a hollow victory that’s come too late, and they find themselves wondering why Diener and the reviewers approved the title in the first place. According to Wager, the paper has made grant administration officers more wary, and it has affected the peer review process: “Everyone knows about the voodoo thing now, even though it’s getting taken out of the journal article,” he says. “The idea is out there, and it’s hard to correct.” Vul has a different take. Once their paper had passed peer review, Vul and his colleagues argue, it was the public’s right to read about it, and respond to it, however they chose — especially given that it sought to reveal flaws in publicly funded research that gets widespread media coverage.

More generally, Vul believes that the response to this article represents the power of 21st-century online media. The reaction to his findings signals changes in discourse, peer review, and potentially even funding. Most important, it has forced scientists to re-evaluate their methodologies, as even Wager’s rebuttal concedes that the nonindependence error Vul discusses can in fact inflate correlations (much of the argument is over how much). Furthermore, online writers, who had been writing regularly about just the type of social neuroimaging papers Vul criticizes, are the same ones who have most thoroughly praised his paper. It shows, Vul says, that the online media “is doing their job of informing the public of both the exciting developments in the field, and also the reasons that one might want to doubt certain findings.”

For Wager, the rapid pace and public forum of this debate has shown him what can go wrong when science is weighed in the open. He sees a direct connection between public opinion and science funding and support. “At the end of the day,” Wager says, “funding has to be justified to Congress. Public opinion definitely does shape which trends in science we think are worth following. A paper like this can affect funding decisions and the course of science if it makes a public impact, even before its scientific validity has thoroughly been assessed over time. Using language like ‘voodoo’ to create an emotional effect is a huge problem because of how it affects public perception, even if the scientific arguments are not well founded.”

The founding editor of Perspectives, Diener, admits that the early release of Vul’s paper and its respective coverage has been a difficult process. “As an editor for 16 years, I’ve never witnessed a paper so widely distributed before its publication,” he says. “There are some very important questions that this raises for science. Most important, how can we guarantee quality in what is sent around? The internet is full of wonderful information — but it is also full of disinformation and errors. How can readers know whether what they are reading is solid information?”

Though Diener sees these as important problems, he has faith in the scientific community’s ability to address them. Peer review, he suggests, may no longer be enough. He envisages a wider formal online-review process, in which scientists could respond to papers, with their comments weighted based on their own publication record. As for the Vul paper, Diener is keenly aware of both the conversation he has started and the criticism he has received, and he hopes that his journal’s approach to the paper — several critical commentaries will be published alongside it — will be more revealing than damaging. “The Vul paper has stirred considerable debate, even heated debate. My goal is that, ultimately, more light is generated than heat by the set of articles. This will happen if we are able to pinpoint the best statistical practices, as well as optimal descriptions of their methods, and motivate scientists to use them. If this happens, my left prefrontal lobes will light up.”

Originally published February 24, 2009

Page 2 of 2

Tags bias consensus limits medicine neuroscience research

Share this Stumbleupon Reddit Email + More


  • Ideas

    I Tried Almost Everything Else

    John Rinn, snowboarder, skateboarder, and “genomic origamist,” on why we should dumpster-dive in our genomes and the inspiration of a middle-distance runner.

  • Ideas

    Going, Going, Gone

    The second most common element in the universe is increasingly rare on Earth—except, for now, in America.

  • Ideas

    Earth-like Planets Aren’t Rare

    Renowned planetary scientist James Kasting on the odds of finding another Earth-like planet and the power of science fiction.

The Seed Salon

Video: conversations with leading scientists and thinkers on fundamental issues and ideas at the edge of science and culture.

Are We Beyond the Two Cultures?

Video: Seed revisits the questions C.P. Snow raised about science and the humanities 50 years by asking six great thinkers, Where are we now?

Saved by Science

Audio slideshow: Justine Cooper's large-format photographs of the collections behind the walls of the American Museum of Natural History.

The Universe in 2009

In 2009, we are celebrating curiosity and creativity with a dynamic look at the very best ideas that give us reason for optimism.

Revolutionary Minds
The Interpreters

In this installment of Revolutionary Minds, five people who use the new tools of science to educate, illuminate, and engage.

The Seed Design Series

Leading scientists, designers, and architects on ideas like the personal genome, brain visualization, generative architecture, and collective design.

The Seed State of Science

Seed examines the radical changes within science itself by assessing the evolving role of scientists and the shifting dimensions of scientific practice.

A Place for Science

On the trail of the haunts, homes, and posts of knowledge, from the laboratory to the field.


Witness the science. Stunning photographic portfolios from the pages of Seed magazine.

Sites by Seed Media Group: Seed Media Group | ScienceBlogs | Research Blogging | SEEDMAGAZINE.COM