The Peer Review Game


Peer review has been a cornerstone of modern science for nearly 300 years, but in the past decade, many in the scientific community have begun to seriously question the anonymous aspect of conventional peer review. The application of the scientific method to the questions that arise in peer review has been long in coming. Major journals such as Nature, JAMA and BMJ have applied randomized controlled trials to compare the quality of peer review produced by different peer review practices. However these trials have been unable to measure either the accuracy of peer review or social interactions between reviewers and authors.

We developed a model of peer review in the form of an on-line game based in the Amazon cloud and we recruited groups of scientific peers to play anonymous and non-anonymous versions of the game. We compare accuracy and social behavior under open and closed systems. A striking result is that an open peer review system enhances cooperative behavior in our study groups, and that this increased cooperation improves the accuracy of the review process. In addition, reviewers are rewarded for refereeing when the group knows their refereeing behavior. These results provide strong evidence for the potential benefits of increased openness and cooperation in peer review.

Data and Code

Our data and scripts are based on the R package. You can download R for free here: R project Website.A very basic intro to R, including instructions on how to install the program, is available here: The Undergraduate Guide to R. The R packages you will need are RColorBrewer, MASS, lme4, pixmap. They can be installed directly within R.