Archive for July, 2011

I came across an interesting article this morning in Slate questioning recent papers on the “contagiousness” of factors ranging from obesity to divorce. The papers were published in top journals like the New England Journal of Medicine (I wrote this enthusiastic blog post about the findings back in 2008) and have generated a wide range of media attention, including the TED talk which I’ve embedded below.

As far as I know the questions surrounding these papers have been entirely statistical (as opposed to ethical) in nature. Below is the abstract of a critique published in the journal of Statistics, Politics, and Policy earlier this year which nicely outlines the problem of having a high profile paper with a poor stats section:

The chronic widespread misuse of statistics is usually inadvertent, not intentional. We find cautionary examples in a series of recent papers by Christakis and Fowler that advance statistical arguments for the transmission via social networks of various personal characteristics, including obesity, smoking cessation, happiness, and loneliness. Those papers also assert that such influence extends to three degrees of separation in social networks. We shall show that these conclusions do not follow from Christakis and Fowler’s statistical analyses. In fact, their studies even provide some evidence against the existence of such transmission. The errors that we expose arose, in part, because the assumptions behind the statistical procedures used were insufficiently examined, not only by the authors, but also by the reviewers. Our examples are instructive because the practitioners are highly reputed, their results have received enormous popular attention, and the journals that published their studies are among the most respected in the world. An educational bonus emerges from the difficulty we report in getting our critique published. We discuss the relevance of this episode to understanding statistical literacy and the role of scientific review, as well as to reforming statistics education.

I should mention that frankly this stats discussion is well over my head, and it may be that these critiques are thoroughly off base - it took the authors a long time and multiple attempts to get this article published, which could be a sign that there is little weight to the arguments, although it could also be a sign that it’s just hard to get this sort of thing published (our friend Yoni Freedhoff detailed the whole process a few weeks ago, which is where I first heard about these new issues). The point being that these papers are among the most high-profile studies published in my field of research in the past few years, and yet people are now saying things like this:

“[Christakis and Fowler’s] errors are in some places so egregious that a critique of their work cannot exist without also calling into question the rigor of review process,”

When I was reading the Slate piece this morning it got me thinking about other recent scientific findings which have been presented in “big idea” forums like TED only to have important questions raised about their veracity.

For example, earlier this year Felisa Wolfe-Simon and other NASA researchers published a paper in Science claiming to have found bacteria which could use arsenic rather than phosphorous as the backbone of its DNA. Shortly thereafter Rosie Redfield wrote a scathing review of the paper on her blog, spawning a massive backlash against the paper in the field as a whole. This backlash prompted Dr Wolfe-Simon and her co-authors to retreat from the media and argue that:

Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated.

And yet just a few months later David Dobbs reported that Dr Wolfe-Simon had not only presented her findings at TED, but had also reiterated her paper’s highly disputed conclusions. Here is what David had to say in March:

Apparently the peer-reviewed realm now includes the high-profile TED conference, where on Wednesday Wolfe-Simon talked about her paper. Neither video nor transcript is released as yet [Travis’ Note: I still haven’t seen them online, but please let me know if someone else has seen them], but accounts suggest she discussed her controversial discovery outside the realm of peer review — in fact, in the most public venue imaginable —and one anonymous source I spoke to today said she repeated the paper’s explicit and disputed claims about arsenic incorporating DNA.

And then there is the case of Marc Hauser, popular author and Harvard researcher who has been under investigation for academic misconduct for the the past year, and whose ultimate fate (as well as his guilt or innocence) remains very unclear. In fairness, he hasn’t presented at TED (although Slate called it “TED-level stuff“), but his popular book Moral Minds certainly places him into the “big idea” category of scientist.

The fact that these eminent “big idea” researchers seem to keep making questionable moral/ethical/academic misjudgments is distressing for a few reasons. First and foremost, it’s because these “big idea” scientists are really the stewards for all of us. TED talks, popular books - these are the way that many non-scientists find out about what is that we do, and why it matters. If the people doing those talks and writing those books turn out to be sketchy then it makes all of us look bad.

But it is also worries me because this is not good for science. I had to stop myself from writing this is not the way that science is done, because that just seems like a cliche in a blog-post mentioning Rosie Redfield and #arsenicDNA. But for science to be done there has to be room for genuine debate, and TED talks don’t seem to have much of that… they seem more like a monologue where you present your ideas as fact. If Dr Wolfe-Simon’s talk had been a debate between herself and one of her critics then I think it would have been far more useful to the advancement of science. And while I realize that advancing science per se is not the purpose of a TED talk, I can’t help but feel that there is something fundamentally wrong about presenting your shiny new finding as fact to a large and very influential audience when it is still eminently unclear whether the finding is legitimate or not. Not that TED talks are wrong, but that it’s dangerous to get too far ahead of the science, or present something as fact when there remain unresolved questions. I have been unable to find a video or transcript of Dr Wolfe-Simon’s talk so I could be off base here, but it certainly seems distressing on the face of it.

The final thing that I find personally distressing about these issues is that I love listening to TED talks and reading books about big ideas. And to be honest I would love to be one of those people who gives those sorts of compelling talks that so clearly demonstrate why an idea or piece or research has meaning outside of the lab. It worries me that other people who share that goal seem to be spreading a message which may not actually represent the “truth”… it makes me nervous about knowledge translation in general if those who are among the most successful are also those pushing the most questionable findings.

Travis