mercredi 10 décembre 2014

Studies over how noisy is it to accept/reject submissions


This year, The NIPS 2014 conference did an interesting experiment: conference chairs did duplicate 10% of the submissions (170 papers) then send the exact same papers to two different groups of reviewers. The result: 25.9% of disagreement. This says that almost for every one out of four papers, the exact same paper is accepted by one group of experts while rejected from the other group. This just shows how noisy the reviewing process is. I was wondering if there were other similar experiments for other fields and what was the disagreement percentage (regardless of the venue: journal or conference)





Aucun commentaire:

Enregistrer un commentaire