IMO open review is the way to go. Having reviews public goes a long long way towards auditing how trustworthy a process is and one of the main sources of trust in Open Source software.
It can be really nerve wracking having your work ventilated in public, but you get used to it. I also think it encourages more polite reviews in some people, as their interactions are also on public display.
There are a few famous developers who used to completely brow-beat people that have publicly started talking about how they are taking feedback to heart about moderating that and I don’t think that would have happened if they had only been giving feedback in private.
Most early career people will not be willing to have their names publicized alongside their referee reports, because that will be too much risk for their career advancement. And let’s face it, most referees are early career just by the sheer amount of papers out there to review.
It’s not even just early career. The problem is that many fields in academia are small enough that researchers end up knowing most of the others. So, if your report is going to be public and the other guy, with whom you had a nice beer last year at the Annual Congress of Researchology, has produced a slightly disappointing paper (not even fraud, just mediocre science), it’s going to be psychologically harder to go full-on steam on the guy and point out all the lacking points of the paper, knowing how much work it’s going to take to just redo everything, just for the paper to be published in a less renowned journal.
Of course, you could say “it’s just part of the job, you’ll assume next time you see the guy to have a beer”. But we’re all humans, and, some exceptions apart, we want to be somewhat nice to people we like…
As you can see, I’m very torn by Open Review. I think it could easily yield a very mild, nicey-nicey reviewing among established peers of the field. Maybe the solution would be for reviews to be open, but manage a way for anonymity to be respected. Note that the Editors know it all already, and they should be the warrant of the quality of the process. The problem is, they struggle to get reviewers, so they can’t be too picky about the quality of their reviews…
IMO open review is the way to go. Having reviews public goes a long long way towards auditing how trustworthy a process is and one of the main sources of trust in Open Source software.
It can be really nerve wracking having your work ventilated in public, but you get used to it. I also think it encourages more polite reviews in some people, as their interactions are also on public display.
There are a few famous developers who used to completely brow-beat people that have publicly started talking about how they are taking feedback to heart about moderating that and I don’t think that would have happened if they had only been giving feedback in private.
Most early career people will not be willing to have their names publicized alongside their referee reports, because that will be too much risk for their career advancement. And let’s face it, most referees are early career just by the sheer amount of papers out there to review.
It’s not even just early career. The problem is that many fields in academia are small enough that researchers end up knowing most of the others. So, if your report is going to be public and the other guy, with whom you had a nice beer last year at the Annual Congress of Researchology, has produced a slightly disappointing paper (not even fraud, just mediocre science), it’s going to be psychologically harder to go full-on steam on the guy and point out all the lacking points of the paper, knowing how much work it’s going to take to just redo everything, just for the paper to be published in a less renowned journal.
Of course, you could say “it’s just part of the job, you’ll assume next time you see the guy to have a beer”. But we’re all humans, and, some exceptions apart, we want to be somewhat nice to people we like…
As you can see, I’m very torn by Open Review. I think it could easily yield a very mild, nicey-nicey reviewing among established peers of the field. Maybe the solution would be for reviews to be open, but manage a way for anonymity to be respected. Note that the Editors know it all already, and they should be the warrant of the quality of the process. The problem is, they struggle to get reviewers, so they can’t be too picky about the quality of their reviews…