Science, meet World
An RCT to determine the value of blogging
When Science of Blogging first went live one of the first comments we received was from the well-known pseudonymous science blogger Drug Monkey, who said that:
One of the mission critical assignments is to figure out how to show real-world impact of blogging. Traffic numbers are insufficient to convince a traditional audience. How to make the determination of impact easier, consistent and valid?
One of the main reasons that Peter and I started Science of Blogging was because we’ve seen that it has a lot of value for us personally. It’s been a useful way to promote our research and network with others, but DM has a point – simply telling someone that your post got X number of hits doesn’t really convey the benefits of blogging. But I’m not sure that we will ever have an Impact Factor-like metric that will allow people to easily quantify just how effective an individual blog is. We could certainly create one based on some combination of comments, incoming links, and viewers per post (or google rankings, etc), but I’m skeptical that it would ever be used in performance reviews or the like. It would be terrific if it did, but I just don’t see it happening. If people don’t see value in blog traffic stats, I don’t think they’re going to value any other blog-related metric either.
Instead, since we are all researchers anyway, I think it makes sense to do the studies to see whether blogging about a topic can help achieve hard outcomes that are already valued. For example, does blogging about a journal article increase the number of downloads or citations that it receives? Does it increase the likelihood that health-care professionals will perform an evidence-based treatment, or avoid a non-evidence-based treatment? Does it help individuals to adopt healthier behaviours?
These are the things that will convince people that blogging is worth the effort. And since we’re all researchers, it really wouldn’t be that hard to actually start to measure these things.
Here’s an example of an RCT that would be tremendously useful in determining the value of blogging in terms of increasing paper downloads and citations, and would cost absolutely no money to perform. Select 30 papers from a wide range of academic disciplines, all of which are at least 5 years old and have less than 3 citations (e.g. if they aren’t cited much now, it’s unlikely that they ever will be). Randomly select 15 of these articles, and ask for volunteers from among the 1000+ active bloggers on Researchblogging.org who are willing to blog about the papers relevant to their discipline. Then, track the number of downloads and citations for the blogged and non-blogged papers over a period of several years, to see if there is a difference between the two groups.
We could even do something similar using papers in the PLoS journals as a convenience sample – are the PLoS papers that have been discussed in blogs downloaded and cited more often? This could be potentially biased (I’m assuming that the papers that get blogged about are probably more interesting or novel, which would make them more likely to get cited as well), but the data is freely available for anyone with a summer student with time to kill.
I know there are a million and one qualitative studies that could also be done in this area, and I’ve participated in a few myself. But lots of people (myself included) like to see hard numbers, and it really wouldn’t be very hard to get them. Seriously, why isn’t the science blogging community doing this? If I’m just ignorant of the research, please tell me. And if it really doesn’t exist, then why don’t we get it going?
Travis
| Print article | This entry was posted by Travis Saunders on April 18, 2011 at 10:02 am, and is filed under Knowledge Translation, Science Blogging. Follow any responses to this post through RSS 2.0. You can leave a response or trackback from your own site. |
about 2 years ago
Many science corporations that have blogs spend a lot of time asking the question: “is it worth it?” We are concerned with number of views, number of times links within the blogs or to papers discussed in a blog are visited, incoming links to the blog, Google ranks, tweets and retweets, and number of comments. All of these represent opportunities to interact with our scientist customers and build relationships, and that is what social media is about—relationship building. It’s a way to reach out, provide information and exchange ideas. Academia may not be “get” the importance of these stats yet, but the corporate world (at least some corporations) have recognized their importance.
I love the idea of a study looking at changes in citation rates on papers that are blogged about v. those not blogged about.
Michele
about 2 years ago
Thanks Michele – it’s good to hear that *some* people care
I’m hoping that mainstream academia will get there some day as well, just as many researchers and organizations already care about things like media impressions.
about 1 year ago
Just came across your post – we actually have done something similar to what you are looking for in the context of economics blogs:
http://blogs.worldbank.org/impactevaluations/the-impact-of-economic-blogs-part-i-dissemination-aka-check-out-these-cool-graphs
Not randomized, but I think still reasonably convincing – what do you think?
about 1 year ago
That’s terrific, David! I’m not familiar with those blogs, how large would you estimate their readership to be? I’m assuming that the readership of Freakonomics must be in the order of several thousand readers per day? I’m asking because I’m wondering whether those blogs are representative of all blogs, or whether they are more similar to a mainstream media outlet. Any thoughts?
Thanks for letting me know about your post!
Travis
about 1 year ago
We look at a range of blogs – from Freakonomics and marginal revolution which have 30,000 or more page views per day, to smaller blogs with 2000-4000 page views per day. Not surprisingly, larger blogs have more impact, but they also seem to have lower click-through rates than more niche research-oriented blogs. Our blog post links to a note which describes these results in more detail.
about 1 year ago
Very interesting. My guess is that even those smaller blogs are still larger than the majority of science blogs (including my own) although it’s nice to see that the relationship still holds within the articles you looked at. I’m very interested in seeing whether this holds for blogs more generally, as I think that would be a pretty strong indication that blogging does actually help disseminate research.