Published Peer Review: True or False Answers
Signed and published peer review improves specific areas of feedback, such as comments on methods and comments with substantiating evidence.
Correct! It is true that research has found improvements in specific areas like constructive feedback, comments on methods, length of review, and substantiating evidence to support the comments (Kowalczuk, 2013; Walsh, 2000; Bornmann, 2012; Mehmani, 2016). The open process of signed and published peer review encourages mutual respect and an openness to feedback.
While two studies by BMJ have found no difference in the overall quality between reviewer comments in conventional and open peer review systems (Van Roonen, 2001; van Rooyen, 2010), other research has found that specific areas of feedback do improve. It is true that improvement has been found in comments on methods and comments with substantiating evidence, as well as the length of reviews (Kowalczuk, 2013; Walsh, 2000; Bornmann, 2012; Mehmani, 2016).
All journals that have implemented signed and published peer review have reported a drop in reviewer acceptance rates.
Requiring signed peer review led to a 1999 BMJ study that reported reviewers were 12% more likely to decline those invitations (van Rooyen, 1999). However, this reluctance appears to have decreased since then. It is false that all journals have reported a drop in reviewer acceptance rates after implementing signed and published peer review. For example, the EMBO Journal reported stable rates through their implementation (Pulverer, 2010).
It is false that all journals have dealt with a decreased ratio of reviewer acceptances. The EMBO Journal reported stable reviewer acceptance rates when implementing signed and published peer review (Pulverer, 2010). While technically a drop in their ratio, The European Journal of Neuroscience found that only 18 of 3293 invited reviewers declined due to signed peer review (Bolam, 2017). In general, researchers seem much more accepting of open peer review than in 1999, when a BMJ study reported that reviewers were 12% more likely to decline (van Rooyen, 1999).
Citations
Bolam, P. (2017, September 14). Transparent Review at the European Journal of Neuroscience: Experiences One Year On, [Blog post].
Bornmann, L., Wolf, M. & Daniel, H.D. (2012). Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics, 91(3): 843-856.
Justice A.C., Cho M.K., Winker M.A., Berlin J.A., Rennie D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial, JAMA. 280(3):240-2.
Kowalczuk, M.K , Dudbridge, F. , Nanda, S., Harriman, S.L., & Moylan, E.C. (2013). A comparison of the quality of reviewer reports from author-suggested reviewers and editor-suggested reviewers in journals operating on open or closed peer review models. F1000Research.
Pulverer, B. (2010). A transparent black box. The EMBO Journal, 29(23):3891-3892.
Ross-Hellauer, T. (2017). What is open peer review? A systematic review [version 2; peer review: 4 approved]. F1000Research, 6:588
van Rooyen, S., Godlee, F., Evans, S., Smith, R., Black, N. (1998) Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA, 280(3):234-7.
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999, Jan 2). Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ, 318(7175):23-7.
van Rooyen, S., Delamothe, T. & Evans, S.J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ, 341:5729.
Walsh, E. Rooney, M. & Wilinson, G. (2000). Open peer review: a randomised controlled trial. British Journal of Psychiatry, 176:47-51.