There has been a lot of discussion recently about the value of peer review (including this phenomenal post by Joe Pickrell of Genomes Unzipped), and whether other models might be cheaper, faster, and ultimately better than the current system.

Regardless of what these alternative models of publishing look like, I agree with Joe that social media will play an important role in identifying high quality papers. Social media would thus be acting as a form of post publication peer review (henceforth referred to as PPPR), and has actually been doing so for some time (Researchblogging.org being the best example that I can think of, although the PLoS Hubs is aimed at this as well). This is in contrast to Letters to the Editor, which up until a few years ago was the only form of PPPR available to researchers. I have recently had experience with both of these forms of PPPR, and thought it would be fun to compare and contrast the experience with each, focusing on the categories that I considered when deciding whether publish my critique in a blog post or Letter.

Speed

My experience with a Letter to the Editor came about last summer when I felt that the conclusions of this article in the International Journal of Behavioural Nutritional and Physical Activity (IJBNPA) did not match up with their data (actually, I felt that their conclusions were directly contradicted by their data). The article was published on July 29th, 2010, and my colleague Stephanie emailed it to me that same day. I read the paper in detail about a week later, and decided to write a Letter to the Editor with Stephanie and our co-supervisor.

Unfortunately IJBNPA had never published a Letter before, and it took some time for IJBNPA and their publisher (BMC) to decide whether they were willing to publish Letters in the journal, and whether or not they would charge their usual $1670 USD processing fee. Fortunately, by the end of 2010 BMC had told us that IJBNPA would begin accepting Letters, and that they would waive their processing fee.

Thus our Letter was officially submitted to the journal in January of 2011, five months after the initial article had been published. Although it was accepted quite quickly, our Letter couldn’t be published until the authors of the original paper had had a chance to respond. Thus our article was officially published on May 25, nearly ten months after the article we were critiquing. It is worth noting that the original article received BMC’s “Highly Accessed” designation, meaning that it was among the more popular articles in the journal during that time-span (during this time readers had no way to know that anyone felt there was a problem with the paper).

In contrast to a Letter, a blog post about an article can be published as soon as it is written. In this case I wrote a blog post about our critique on June 12, 2011 and published it on June 13, for a total turn-around of 1 day. Rosie Redfield’s famous #arsenicDNA blog post and Letter to the Editor showed similar time differences - her blog post was published on December 4, 2 days after the article she was critiquing. In contrast, her Letter to the Editor wasn’t published online until May 27, a full 5 months after the original article. When it comes to speed, traditional Letters can’t compete with blogs.

Winner: Blogging.

Impact

The impact of a Letter to the Editor depends on the journal in which it is published - a Letter in JAMA could arguably have more impact than an RCT in many other journals. Assuming that the original article was of interest to the journal’s readership, the Letter should be as well. However, some journals (including IJBNPA) do not alert readers when an article is the subject of a Letter, making it questionable whether readers of an article will even be aware that a Letter has been published. The one distinct advantage of a Letter is that the authors of the original article are guaranteed to be aware of your critique, which may not be the case with a blog post. In terms of total readership, as of today our Letter has been viewed just shy of 1000 times.

Like a Letter, the impact of a blog post is going to vary tremendously depending on a blog’s regular audience. A post on Pharyngula will obviously have much more reach than a post on Obesity Panacea. Similarly, if a critique is posted on a blog that typically targets lay-readers it isn’t going to have as much impact as one posted on a blog like The Tree of Life which is read by active scientists. And finally, all bets are off when a post goes viral like Redfield’s #arsenicDNA post, which I can only imagine brought in far more traffic than a usual post. My June 13 blog post has been viewed 365 times on our website, and has also been sent to our ~1,400 email/RSS subscribers (although we have no way to know how many have actually read the full post). It’s difficult to determine which has received more total views, since the post linked to the article itself, and many of our readers are active researchers… so let’s assume that the blog and the Letter were viewed by a similar number of people.

Winner: Too close to call.

Cost

In the end, our Letter to the Editor didn’t cost any actual money, although it took quite a while to convince BMC to waive their processing fee. That being said, the Letter still had to be peer reviewed which is a not-insignificant in-kind contribution.

Assuming you don’t pay hosting fees, a blog post costs nothing more than the time required to write the post. If you do pay hosting fees, you can divide the annual cost by the number of posts per year - for this blog that would work out to about $2.50 per post.

Winner: Blogging.

Career Building

Publications are the currency of research. These are what people (scholarship and grant committees, performance review committees, etc) focus on when determining your productivity, and having a few extra publications can make a huge difference for a young researcher. In this respect, blog posts can’t compare with a Letter to the Editor. The Letter is now on my CV as a first-author publication, while the blog post is not. Whether you’re applying for scholarships, grants, or faculty positions, there is no question in my mind that the value of a Letter far outweighs that of a blog post.

That being said, I should point out that this isn’t really based on the relative impact of a given Letter versus that of a blog post on the same topic -Redfield’s #arsenicDNA post almost certainly had more impact than her subsequent Letter to the Editor on the same topic. When it comes to career building, Letters just happen to be valued while blog posts are not (yet?).

Winner: Letter to the Editor.

Summary

Based on my personal experience, I would argue that our Letter probably had roughly similar impact on the field as my blog post did - the blog post was viewed by up to 1700 people, many of whom are active researchers in my field of research. The blog post was also far faster and less labour-intensive, all of which I knew before I decided to write the Letter. The Letter did result in a thoughtful response from the authors of the original article (which almost certainly wouldn’t have occurred in response to a blog post) and is far more valuable in my academic portfolio than any blog post could ever be.

Clearly the ideal situation would be some sort of hybrid - the speed of a blog post combined with the ability of a Letter to target researchers and generate respect from granting agencies and the like. I should point out that there are a few options like this coming into existence - BMJ has eLetters, while both PLoS and BMC allow comments on posts. But these hybrid models aren’t picked up by Pubmed and don’t really count as publications…. meaning that they’re not much better than blog posts. eLetters seem like the best of these new models, and the only one that is used with any regularity. But until they are valued as much as a traditional Letter, I would expect people to save their biggest critiques (and therefore the ones that should be published most rapidly) for Letters just as I did.

Travis