It sure would save a lot of money if scientific peer review worked more like social media. We looked at STM (Scientific Technical Medical) Publishing in our Creative Destruction series. Here are parts 1, 2 and 3.

We saw an industry – academia – where there is a big budget pressure yet where the cost of scientific journals keep rising faster than inflation. Prices of all digital data is crashing – but not scientific data. So, lower prices would be very, very welcome. And peer review is at the heart of the scientific publishing business.

But we might end up with science like this:

That is a spoof (hat tip to this blog highlighting the pitfalls of populist science BEFORE peer review was established.)

The fact that the Kraken research is sponsored by a vendor of Rum is perfect for today’s social media!

We don’t want Kraken style science. Nor do we want ever-rising cost of scientific journals. We look at who is trying to square this circle.

It sure would save a lot of money if scientific peer review worked more like social media. We looked at STM (Scientific Technical Medical) Publishing in our Creative Destruction series. Here are parts 1, 2 and 3.

We saw an industry – academia – where there is a big budget pressure yet where the cost of scientific journals keep rising faster than inflation. Prices of all digital data is crashing – but not scientific data. So, lower prices would be very, very welcome. And peer review is at the heart of the scientific publishing business.

But we might end up with science like this:

That is a spoof (hat tip to this blog highlighting the pitfalls of populist science BEFORE peer review was established.)

The fact that the Kraken research is sponsored by a vendor of Rum is perfect for today’s social media!

We don’t want Kraken style science. Nor do we want ever-rising cost of scientific journals. We look at who is trying to square this circle.

Why We Need Peer Review

Here is Greg Fish from World of Weird Things blog talking about the lessons behind the Kraken video:

“Sure, the video is funny and should be enjoyed for what it is, but it does strike a note if we were to look back at what it was spoofing. Back in the nineteenth century, before peer review was as widely used as it is today, the scientists and historians of the time would often make plenty of unsubstantiated claims published in journals which were read by those who were considered wealthy intellectuals, and these claims often stuck for many a decade.”

He goes on to caution about throwing the baby out with the bathwater:

“So whenever you see cranks who failed to get published in scientific journals talking about taking their ideas straight to the people and sneering at peer review, think back to the days when science worked the way they propose it should and consider all the subsequent mistakes and wild claims which the public often wouldn’t see refuted with real evidence since scientific criticisms took place in private letters and publications which a private citizen wouldn’t even know how to obtain. Even though peer review isn’t perfect and needs to become even more transparent, we shouldn’t be encouraging people to circumvent it just so they could write a book exploring a controversy or dilemma that doesn’t exist solely for the sake of selling copies while pretending to be groundbreaking experts in a field in which they have few, if any, qualifications.”

In other words, the mission is to improve peer review, not junk it.

Are They Reliable and Do Editors Care?

This post from PLoS ONE (Public Library Of Science) asks the question:

“Editorial Peer Reviewers’ Recommendations at a General Medical Journal: Are They Reliable and Do Editors Care?”

This is a daunting looking document that shouts “beware blogger, this is real academic stuff, skim at your own risk!”. And, yes, it has been peer reviewed by all these fine folks:

“Editorial peer review is universally used but little studied. We examined the relationship between external reviewers’ recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM).”

In other words, did editors pay any attention to the recommendations of reviewers? And did reviewers agree?

Their discussion raises an interesting question. Why filter at all at the pre-publication phase? By that they mean that the idea of outright rejection is no longer valid. There is no space constraint online. Why not publish everything and do the filtering later?

This idea is picked up in a FriendFeed discussion on Abundance Obsoletes Peer Review, So Drop It.

The question is only how to make the good stuff “float to the top”.

Which is exactly what all the aggregation tools in social media try to do.

So, just bung a Digg or Like button on every article?

Err, No, A Digg Or Like Button Is Not Enough

But possibly something more nuanced like FriendFeed? It is funny that a lot of academics get together on FriendFeed and ask questions that are essentially:

1. If anybody can publish, “won’t we get a lot of crap?”. Yes, just as there is on FriendFeed, but you found this good one right?

2. Why does anybody do peer reviews, it is an unpaid waste of time. Just like commenting on FriendFeed, blogging or tweeting is a waste of time. Of more likely it is a way to learn, make connections and earn respect.

So the question is really, how do we build good filters?

The 3 Tests Of A Good Filter

Good content filters are needed for complex content where important decisions will be made based on that content. For simple content such as “which song shall I listen to” or “what bar shall I go to”, the social media voting buttons are fine.

But for things like “what drug should I propose” or “what stock do I put in my kid’s college retirement fund” we need something a bit better, with these 3 attributes:

1. Authenticated Reviewers. Anonymous reviews are discounted. Maybe there are ways to see them, insight may lurk there. But one must be able to see the credentials of the reviewers and there need to be reasonable controls against spam.

2. Transparent Audit Trail. You can see all comments, conversations, ratings, etc. Very few people will look at this, but having it is a safeguard. Wikipedia does this well.

3. Reviewer rating. Something like OneThing might work. Reviewers rate each other. A high rating from somebody with a high rating gets your score higher.

This transparency also creates the motivation. Reviewers ratings within these systems will impact their academic standing. Outsiders can break in – you don’t need tenure to get rated by peers.

Many academics will resist this. All change is painful. But change is coming and it is likely to have two characteristics:

1. Dramatically reduce the cost of scientific information.

2. Leverage some social media techniques.

———————————————————————–
CONVERT BREAKS: __default__

Why We Need Peer Review

Here is Greg Fish from World of Weird Things blog talking about the lessons behind the Kraken video:

“Sure, the video is funny and should be enjoyed for what it is, but it does strike a note if we were to look back at what it was spoofing. Back in the nineteenth century, before peer review was as widely used as it is today, the scientists and historians of the time would often make plenty of unsubstantiated claims published in journals which were read by those who were considered wealthy intellectuals, and these claims often stuck for many a decade.”

He goes on to caution about throwing the baby out with the bathwater:

“So whenever you see cranks who failed to get published in scientific journals talking about taking their ideas straight to the people and sneering at peer review, think back to the days when science worked the way they propose it should and consider all the subsequent mistakes and wild claims which the public often wouldn’t see refuted with real evidence since scientific criticisms took place in private letters and publications which a private citizen wouldn’t even know how to obtain. Even though peer review isn’t perfect and needs to become even more transparent, we shouldn’t be encouraging people to circumvent it just so they could write a book exploring a controversy or dilemma that doesn’t exist solely for the sake of selling copies while pretending to be groundbreaking experts in a field in which they have few, if any, qualifications.”

In other words, the mission is to improve peer review, not junk it.

Are They Reliable and Do Editors Care?

This post from PLoS ONE (Public Library Of Science) asks the question:

“Editorial Peer Reviewers’ Recommendations at a General Medical Journal: Are They Reliable and Do Editors Care?”

This is a daunting looking document that shouts “beware blogger, this is real academic stuff, skim at your own risk!”. And, yes, it has been peer reviewed by all these fine folks:

“Editorial peer review is universally used but little studied. We examined the relationship between external reviewers’ recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM).”

In other words, did editors pay any attention to the recommendations of reviewers? And did reviewers agree?

Their discussion raises an interesting question. Why filter at all at the pre-publication phase? By that they mean that the idea of outright rejection is no longer valid. There is no space constraint online. Why not publish everything and do the filtering later?

This idea is picked up in a FriendFeed discussion on Abundance Obsoletes Peer Review, So Drop It.

The question is only how to make the good stuff “float to the top”.

Which is exactly what all the aggregation tools in social media try to do.

So, just bung a Digg or Like button on every article?

Err, No, A Digg Or Like Button Is Not Enough

But possibly something more nuanced like FriendFeed? It is funny that a lot of academics get together on FriendFeed and ask questions that are essentially:

1. If anybody can publish, “won’t we get a lot of crap?”. Yes, just as there is on FriendFeed, but you found this good one right?

2. Why does anybody do peer reviews, it is an unpaid waste of time. Just like commenting on FriendFeed, blogging or tweeting is a waste of time. Of more likely it is a way to learn, make connections and earn respect.

So the question is really, how do we build good filters?

The 3 Tests Of A Good Filter

Good content filters are needed for complex content where important decisions will be made based on that content. For simple content such as “which song shall I listen to” or “what bar shall I go to”, the social media voting buttons are fine.

But for things like “what drug should I propose” or “what stock do I put in my kid’s college retirement fund” we need something a bit better, with these 3 attributes:

1. Authenticated Reviewers. Anonymous reviews are discounted. Maybe there are ways to see them, insight may lurk there. But one must be able to see the credentials of the reviewers and there need to be reasonable controls against spam.

2. Transparent Audit Trail. You can see all comments, conversations, ratings, etc. Very few people will look at this, but having it is a safeguard. Wikipedia does this well.

3. Reviewer rating. Something like OneThing might work. Reviewers rate each other. A high rating from somebody with a high rating gets your score higher.

This transparency also creates the motivation. Reviewers ratings within these systems will impact their academic standing. Outsiders can break in – you don’t need tenure to get rated by peers.

Many academics will resist this. All change is painful. But change is coming and it is likely to have two characteristics:

1. Dramatically reduce the cost of scientific information.

2. Leverage some social media techniques.

———————————————————————–
• Don’t forget to propose your startup for our Semantic Web Impact Awards. The deadline is Sept. 15

Semantic Tech & Business Conference Returns to San Francisco

Semantic Tech & Business Conference returns to San Francisco in June! Join us from June 3-7 for complete coverage of Big Data, Linked Data, Extreme Information Management, and Semantic Web. From breakthrough approaches to solving business problems to the big data implications of fast–evolving technologies, SemTechBiz provides you with an unparalleled interactive experience and delivers tangible business value. We're offering a special early rate when you register by February 17. Sign up now!