- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes an LLM-assisted approach to reducing the randomness of scientific peer review. The author(s) employs historical peer reviews to construct checklists by extracting individual criticisms and clustering them into different categories. They next proposes to select the checklist’s level of granularity by maximizing expected correlated agreement, defined as the extent to which independent reviewers of the same paper agree more than reviewers of different papers. The author(s) further establishes theoretical guarantees for their selection criterion. Finally, in synthetic peer-review experiments using NeurIPS 2021 and ICLR 2022 data, the author(s) finds that their proposed checklists reduce the mean squared error of reviewer's average scores by 58% and 21%, respectively. A smaller study involving 20 human participants also finds that the checklists improve the accuracy of paper rankings.
Critical strengths
I see three critical strengths of this paper: (i) it addresses a relatively underexplored and novel problem in the literature; (ii) the author(s) provides a rigorous mathematical formulation of the problem and establish theoretical guarantees for their proposed checklist selection method; and (iii) the experimental results provide evidence of the effectiveness and practical usefulness of the proposed approach.
Critical weaknesses
As I have indicated below, this paper is not in my area. I have identified a few potential weaknesses, but some of them may arise from misunderstandings.
-
The theoretical results are interesting, but they rely on several assumptions that may be difficult to verify in practice. The authors do not appear to provide empirical evidence supporting these assumptions in Appendix A.
-
The comparison may favour the proposed approach. As described in Appendix C.6, LLM reviewers without checklists are restricted to ten specific review issues within a single category whereas checklist-assisted reviewers can examine all 100 checklist items. Thus, the improvements may come from this unequal scope of evaluation rather than the effectiveness of the proposed checklist.
-
The results from the human study are encouraging, but the evaluation involves only 20 participants and six papers, which may limit the generalisability of the findings. I understand that conducting human evaluations is time-consuming, but a larger-scale study would provide stronger evidence of the practical effectiveness of the proposed approach.
Questions for authors
Please refer to the weaknesses above.
2 comments
Thanks again for your review. We are happy to answer any questions you might have on the new information we shared before the rebuttal period closes.
We thank the reviewer for the careful reading. We appreciate the note that some questions may arise because the paper is outside the reviewer's area, and we answer them below. The replies below follow the order of questions in the Weaknesses section.
Q1. Do the theoretical assumptions hold in practice? The guarantees rely on assumptions that may be hard to verify. Does Appendix A give empirical evidence for them?
Our assumptions on monotonicity and submodularity are classic structures studied in subset selection and combinatorial optimization [1,2]. The other assumptions ensure that each signal has its own value and a unique value conditional on the others. We believe this is realistic for checklist construction, since each question should have its own focus relative to the others. But the reviewer is correct that it can be hard to verify such assumptions in practice.
Our work offers more direct empirical evidence that ECA can serve as a viable proxy for MSE reduction. Among the checklists we compare, the one with the highest ECA also has the lowest review error for every committee of three or more, on both NeurIPS and ICLR. The one with the lowest ECA has the highest error (Section 5).
[1] Nemhauser, George L., Laurence A. Wolsey, and Marshall L. Fisher. "An analysis of approximations for maximizing submodular set functions—I." Mathematical Programming 14, no. 1 (1978): 265-294.
[2] Krause, Andreas, and Daniel Golovin. "Submodular function maximization." Tractability 3, no. 71-104 (2014): 3.
Q2. Is the gain from the checklist, or from a wider scope? As described in Appendix C.6, LLM reviewers without a checklist are limited to ten issues inside one category, while checklist reviewers can examine all 100 items. Does the improvement remain when the scope of evaluation is equal?
We restricted each unaided LLM reviewer to ten issues so that the unaided reviewer simulation would more closely resemble human reviewers’ behavior as observed in our analysis of the 2021 NeurIPS review corpora. A human reviewer raises about 6% of the issues a larger pool would raise, and most reviews do not overlap (Section 3). To mimic this limitation, we gave the unaided reviewer only 10 checklist items at the k = 50,000 level, randomly sampled from different branches at k = 50. By doing so, unaided reviewers will focus on a limited, non-overlapping set of problems in the paper, similar to how human reviewers do. We will acknowledge this motivation more clearly in the paper, and mention exploring how much results depend on myopic reviewers as a worthwhile investigation for future work.
For the checklist-aided reviewers, we gave them one shared checklist at different coarsening levels as the intervention. For example, when they are assigned the k = 100 checklist, they may read all 100 items and make their decisions according to the checklist. This also simulates how the checklist will be used in real review processing, where each reviewer will be given an LLM-prefilled checklist to help them make better decisions.
A longer list is not necessarily beneficial. For example, the 2,000-item checklist has higher MSE than the 100-item checklist: 0.18 against 0.10 for a committee of four on NeurIPS, and 0.29 against 0.21 on ICLR (Section 5). In practice with human reviewers, there are likely to be cognitive limits on how many checklist items a given reviewer can engage with. In future work, we plan to model reviewers’ cognitive load, exploring the impacts of checklists when reviewers are constrained to only access or pay attention to a subset of the items, to see whether this would make a difference.
Q3. Does the human study generalize? The study has 20 participants and six papers. The reviewer grants that human studies are costly, and asks whether a larger study is needed for stronger evidence of practical effectiveness.
The human study is a proof of concept. Its error is the share of paper pairs a participant orders against the NeurIPS ranking, so 0.5 is chance. The error falls from 0.48 without the checklist to 0.22 with it, a drop of 0.26 (paired t-test, p = 0.03; 95% CI 0.03 to 0.49). The drop varies widely across participants (standard deviation 0.47), so a study of this size has 80% power only for drops of about 0.3 or more. The drop we observe is about that size. We acknowledge the study's limited scale: with 20 participants and six papers, it cannot detect smaller effects or estimate the effect size precisely.