acceptodds
Under review as a conference paper at ICLR 2027

Learning Checklists to Reduce Lottery Effects in Scientific Peer Review

Abstract

Reviewer lottery error describes error due to a small set of peer reviews attending to only some of the issues with a paper that a larger pool of eligible reviewers would raise. We show this error can be reduced by surfacing a checklist of criticisms learned from a historical review corpus. Our approach generates a tree of checklists representing coarsenings of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. We realize this approach with an LLM-assisted pipeline and test it on NeurIPS 2021 and ICLR 2022 review data. In simulated review with LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three-reviewer committee by 58% on NeurIPS and 21% on ICLR, and on both venues the checklist with the highest correlated agreement has the lowest error for committees of three or more. In a human study with 20 participants, the learned NeurIPS checklist lowers the MSE of a paper ranking, measured against the NeurIPS 2021 order, from 0.48 to 0.22.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
peer review, reviewer lottery, checklists, large language models, hierarchical coarsening, bias-variance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tDbpOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an LLM-assisted approach to reducing the randomness of scientific peer review. The author(s) employs historical peer reviews to construct checklists by extracting individual criticisms and clustering them into different categories. They next proposes to select the checklist’s level of granularity by maximizing expected correlated agreement, defined as the extent to which independent reviewers of the same paper agree more than reviewers of different papers. The author(s) further establishes theoretical guarantees for their selection criterion. Finally, in synthetic peer-review experiments using NeurIPS 2021 and ICLR 2022 data, the author(s) finds that their proposed checklists reduce the mean squared error of reviewer's average scores by 58% and 21%, respectively. A smaller study involving 20 human participants also finds that the checklists improve the accuracy of paper rankings.

Critical strengths

I see three critical strengths of this paper: (i) it addresses a relatively underexplored and novel problem in the literature; (ii) the author(s) provides a rigorous mathematical formulation of the problem and establish theoretical guarantees for their proposed checklist selection method; and (iii) the experimental results provide evidence of the effectiveness and practical usefulness of the proposed approach.

Critical weaknesses

As I have indicated below, this paper is not in my area. I have identified a few potential weaknesses, but some of them may arise from misunderstandings.

  1. The theoretical results are interesting, but they rely on several assumptions that may be difficult to verify in practice. The authors do not appear to provide empirical evidence supporting these assumptions in Appendix A.

  2. The comparison may favour the proposed approach. As described in Appendix C.6, LLM reviewers without checklists are restricted to ten specific review issues within a single category whereas checklist-assisted reviewers can examine all 100 checklist items. Thus, the improvements may come from this unequal scope of evaluation rather than the effectiveness of the proposed checklist.

  3. The results from the human study are encouraging, but the evaluation involves only 20 participants and six papers, which may limit the generalisability of the findings. I understand that conducting human evaluations is time-consuming, but a larger-scale study would provide stronger evidence of the practical effectiveness of the proposed approach.

Questions for authors

Please refer to the weaknesses above.

2 comments

We thank the reviewer for the careful reading. We appreciate the note that some questions may arise because the paper is outside the reviewer's area, and we answer them below. The replies below follow the order of questions in the Weaknesses section.

Q1. Do the theoretical assumptions hold in practice? The guarantees rely on assumptions that may be hard to verify. Does Appendix A give empirical evidence for them?

Our assumptions on monotonicity and submodularity are classic structures studied in subset selection and combinatorial optimization [1,2]. The other assumptions ensure that each signal has its own value and a unique value conditional on the others. We believe this is realistic for checklist construction, since each question should have its own focus relative to the others. But the reviewer is correct that it can be hard to verify such assumptions in practice.

Our work offers more direct empirical evidence that ECA can serve as a viable proxy for MSE reduction. Among the checklists we compare, the one with the highest ECA also has the lowest review error for every committee of three or more, on both NeurIPS and ICLR. The one with the lowest ECA has the highest error (Section 5).

[1] Nemhauser, George L., Laurence A. Wolsey, and Marshall L. Fisher. "An analysis of approximations for maximizing submodular set functions—I." Mathematical Programming 14, no. 1 (1978): 265-294.

[2] Krause, Andreas, and Daniel Golovin. "Submodular function maximization." Tractability 3, no. 71-104 (2014): 3.

Q2. Is the gain from the checklist, or from a wider scope? As described in Appendix C.6, LLM reviewers without a checklist are limited to ten issues inside one category, while checklist reviewers can examine all 100 items. Does the improvement remain when the scope of evaluation is equal?

We restricted each unaided LLM reviewer to ten issues so that the unaided reviewer simulation would more closely resemble human reviewers’ behavior as observed in our analysis of the 2021 NeurIPS review corpora. A human reviewer raises about 6% of the issues a larger pool would raise, and most reviews do not overlap (Section 3). To mimic this limitation, we gave the unaided reviewer only 10 checklist items at the k = 50,000 level, randomly sampled from different branches at k = 50. By doing so, unaided reviewers will focus on a limited, non-overlapping set of problems in the paper, similar to how human reviewers do. We will acknowledge this motivation more clearly in the paper, and mention exploring how much results depend on myopic reviewers as a worthwhile investigation for future work.

For the checklist-aided reviewers, we gave them one shared checklist at different coarsening levels as the intervention. For example, when they are assigned the k = 100 checklist, they may read all 100 items and make their decisions according to the checklist. This also simulates how the checklist will be used in real review processing, where each reviewer will be given an LLM-prefilled checklist to help them make better decisions.

A longer list is not necessarily beneficial. For example, the 2,000-item checklist has higher MSE than the 100-item checklist: 0.18 against 0.10 for a committee of four on NeurIPS, and 0.29 against 0.21 on ICLR (Section 5). In practice with human reviewers, there are likely to be cognitive limits on how many checklist items a given reviewer can engage with. In future work, we plan to model reviewers’ cognitive load, exploring the impacts of checklists when reviewers are constrained to only access or pay attention to a subset of the items, to see whether this would make a difference.

Q3. Does the human study generalize? The study has 20 participants and six papers. The reviewer grants that human studies are costly, and asks whether a larger study is needed for stronger evidence of practical effectiveness.

The human study is a proof of concept. Its error is the share of paper pairs a participant orders against the NeurIPS ranking, so 0.5 is chance. The error falls from 0.48 without the checklist to 0.22 with it, a drop of 0.26 (paired t-test, p = 0.03; 95% CI 0.03 to 0.49). The drop varies widely across participants (standard deviation 0.47), so a study of this size has 80% power only for drops of about 0.3 or more. The drop we observe is about that size. We acknowledge the study's limited scale: with 20 participants and six papers, it cannot detect smaller effects or estimate the effect size precisely.

Thanks again for your review. We are happy to answer any questions you might have on the new information we shared before the rebuttal period closes.

Reviewer_4oAaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper learns hierarchical review checklists from historical peer-review data and uses ECA to select their granularity. The authors evaluate the approach through simulated LLM reviews and a small human study, reporting reduced review error and improved agreement with historical rankings.

Critical strengths
  1. The paper addresses an interesting and practically relevant problem in scientific peer review.
  2. The proposed framework provides a systematic approach to learning and selecting review checklists from historical review data.
  3. The authors evaluate the approach through both simulated reviews and a human study.
Critical weaknesses

Weaknesses

  1. Some design choices seem rather ad hoc and are not sufficiently justified. Why did the authors choose only ICLR 2022 and NeurIPS 2021? Why not include more years of data? Also, the simulated reviewer populations have 47 and 49 reviewers, respectively. I understand that some branches related to rebuttal and revision are excluded, but the resulting numbers still seem somewhat arbitrary. More importantly, the paper does not show whether this population size is sufficient to provide a stable reference score. I would expect a sensitivity analysis showing how the results change as the size of the simulated reviewer population increases.

  2. The evaluation is limited to the same venues and years used to construct the checklists. If I understand the setup correctly, the authors learn a checklist from the historical reviews of ICLR 2022 and NeurIPS 2021, and then evaluate it on papers from those same venue-year settings. I appreciate that the authors try to avoid direct data leakage by holding out review claims when constructing the checklist tree. However, this still does not establish that the method generalizes beyond the particular review environment from which the checklist was learned. I would like to see both cross-venue and cross-year evaluations. Since review data are not publicly available for every venue, a practical starting point would be to use the checklist learned from NeurIPS to evaluate ICLR papers, and vice versa.

  3. The simulated review experiments rely on only one LLM, Composer 2.5. This seems insufficient for a method that is intended to improve scientific peer review more generally. Different models may identify different issues, follow checklist instructions differently, and respond differently to the same checklist. The authors should repeat the experiments with several other strong models, including more recent ones such as GPT-6.1 Sol. Otherwise, it is unclear whether the proposed method is genuinely robust or simply works well with the particular reviewing behavior of Composer 2.5. Given how quickly LLMs evolve, a method whose effectiveness disappears after a model upgrade would have limited long-term value.

  4. The paper lacks a direct comparison with a checklist generated by a strong LLM. I would like to see an experiment in which a strong model is simply prompted to generate a comprehensive review checklist, and that checklist is evaluated under exactly the same protocol as the proposed method. Comparing against the official NeurIPS checklist is not sufficient to answer this question. In practice, reviewers may rely on their usual reviewing habits rather than systematically consulting an existing official checklist. The important question is whether learning a checklist from historical reviews and optimizing its granularity with ECA provides any meaningful advantage over simply asking a strong LLM to produce a good checklist.

  5. The human evaluation uses a questionable reference standard, and the results do not establish improved review quality. The paper motivates its approach by arguing that peer review is subject to lottery effects, yet the human study evaluates participants by how closely their rankings match the historical NeurIPS 2021 ordering. If historical review outcomes are themselves noisy, why should this particular ordering be treated as the target? The fact that the ranking MSE decreases from 0.48 to 0.22 only shows that participants become more consistent with the historical ordering; it does not establish that their judgments are more accurate. This also raises a concern about overfitting to the reviewing conventions of the same venue and year. Consider an extreme example: if we derive a checklist from the review patterns of a particular venue and then ask reviewers to use it, they may become better at reproducing that venue's historical judgments without necessarily becoming better at identifying the actual strengths and weaknesses of a paper. Although the authors attempt to hold out review claims, the current evaluation does not fully resolve this concern.

  6. The complete checklist is not provided in the supplementary material. At a minimum, the authors should release the complete 100-item checklist selected by their method.

  7. The human study lacks an appropriate control condition. All 20 participants compare a condition with a pre-filled checklist against a condition without one. This makes it difficult to determine whether the improvement comes from the proposed learned checklist specifically, or simply from having any structured checklist available. I would prefer a blinded comparison between the proposed checklist and a plausible control checklist, such as one generated directly by a strong LLM. Participants should not know which checklist is the proposed method and which is the control. This would provide much stronger evidence that the proposed checklist itself is responsible for the observed improvement, rather than the mere presence of a checklist.

  8. I am not fully convinced that the paper's objective captures the most important purpose of peer review. Reducing reviewer disagreement is not necessarily a bad goal, but I am not sure that more concentrated scores or more consistent acceptance-related judgments should be the primary objective. Acceptance decisions are ultimately made by area chairs, who consider more than the reviewers' scores, and many useful aspects of a review may not be reflected in the final score at all. More fundamentally, I see peer review as serving an important role in helping authors identify weaknesses and improve their manuscripts, while the accept/reject decision is only one part of the process. The paper does not show that the proposed checklist leads to more useful feedback, helps authors address real problems, or improves manuscript quality. I therefore remain unconvinced that reducing score variance, by itself, is sufficient to establish the practical value of this approach.

Questions for authors

The questions and concerns I have are already detailed in the Weaknesses section. The authors may use an AI tool to help summarize them if that is useful. However, to be honest, I do not believe these fundamental issues can be adequately addressed during the rebuttal period. They require substantial additional experiments and a more careful reconsideration of the evaluation design and the paper's central objective, rather than simple clarifications or minor revisions. This is why my recommendation is a clear rejection.

4 comments

We thank the reviewer for the detailed comments. Following the reviewer’s suggestions, we have run three new experiments:

  • Cross-year evaluation: We applied the NeurIPS 2021 checklists to the NeurIPS 2022 papers under the same protocol. We found similar patterns.
  • Better LLM model as simulator: To show our pipeline and results don’t tie to Composer 2.5, we ran the same pipeline with the GPT-6.1 Sol model and found similar patterns.
  • Baseline of LLM-written Checklist: We used the most capable LLM, Fable 5.2, to generate a 110-item checklist and evaluate it with the same protocol. We found our generated k=100 checklist outperformed it by 19% at a committee size of 3.

The more detailed replies below follow the Weaknesses section in order.

Q1. Why only these two corpora? Why are the checklists learned only from ICLR 2022 and NeurIPS 2021, with no additional years?

The NeurIPS 2021 consistency experiment is a useful reference dataset, where most papers have 8 total human reviews, making it possible to observe how variance changes across a larger range of committee sizes than most datasets. This is why we used it. But as we make clear in the paper, our approach can be applied to any peer review dataset.

Therefore, we also demonstrate on ICLR 2022 as a second, more typical review corpus. Our approach can also be applied to any other review corpus. However, while the pipeline we contribute is general, the checklist it learns will be specific to the distribution of papers in the corpus used. We are not proposing a method to learn checklists that can be used across domains. In practice, learning the checklist on the previous year’s data is a likely use case; therefore, we evaluated the ICLR 2022 checklist on ICLR 2023 data. We will add these results to the paper. While a demonstration that learns a checklist for one venue and applies it to another would be interesting (and likely effective to the extent that the venues are similar in topic distribution), we do not claim that our approach can be used across different venues. We will note this as an empirical question for exploratory future work.

Q2. Is the simulated population large enough to be a reference? Where do the population sizes of 49 reviewers (NeurIPS) and 47 reviewers (ICLR) come from? After rebuttal and revision branches are dropped, is that size enough to produce a stable reference score? How do the results change as the simulated population grows?

At the k=50 cut, the checklist tree has 50 branches, and the unaided simulation places one reviewer on each branch. Reviewers therefore cover non-overlapping issues, similar to what we observe of real reviewers in our analysis of the NeurIPS 2021 data. We then drop the branches whose checklist items concern meta-review, rebuttal, and revision rather than the paper: one branch on NeurIPS and three on ICLR (Appendix C). This leaves 49 reviewers on NeurIPS and 47 on ICLR.

As the paper already reports, we verify that the population target at these sizes is stable. Its standard error is about 0.13 points on the 1–10 scale (Section 5). In Appendix D, Figure 4, the average over all 49 and all 47 reviewers lies within about ±0.17 points of the target, which is the remaining uncertainty of that target.

Q3. Is there a cross-venue or cross-year result? The checklist is evaluated on papers from the same venue and year used to build it. Holding out review claims avoids direct leakage, but does not show that the method works outside that review environment. Can the NeurIPS checklist be applied to ICLR papers, and the ICLR checklist to NeurIPS papers?

The pipeline is general: run it on a corpus, and it learns a checklist. The items are specific to that venue and domain. We do not claim that a checklist learned at one venue will transfer to another. NeurIPS papers use the NeurIPS checklist, and ICLR papers use the ICLR checklist. While it’s possible similar venues could use the same checklist, evaluating this is beyond the scope of our paper.

However, cross-year generalization is worth measuring directly, and we ran this experiment for the rebuttal. We applied the NeurIPS 2021 checklists directly to 9 NeurIPS 2022 submissions (3 strong, 3 borderline, and 3 rejected) under the same protocol. We see mostly the same pattern as in 2021: for a typical committee of three reviewers, the 2021 learned k = 100 checklist reduces review MSE by 44% (from 0.15 unaided to 0.08), and has the lowest error among both the official NeurIPS checklist (0.24) and all other checklist granularities (0.12 to 0.18).

Q4. Is the result tied to Composer 2.5? The simulation uses only this model. Does the error reduction remain after repeating the experiments with several other strong, more recent models? The reviewer names GPT-6.1 Sol as an example.

We use Composer 2.5 as the simulated reviewer, and using a weaker model is intentional: the simulation is meant to reproduce a human reviewer's limited attention, i.e., the inability to track every issue and verify every claim.

We repeated the simulation on 9 stratified NeurIPS papers (3 strong, 3 borderline, and 3 weak) with GPT-6.1-Sol under the same protocol and found similar results: at a typical committee size of three reviewers, the learned k = 100 checklist's MSE is still 49% lower than the unaided group's (0.13 vs. 0.26), and lower than both the official checklist (0.18) and all other learned granularities (0.16 to 0.18).

Q5. Is there a comparison with a checklist written by a strong LLM? Under the same protocol, does learning a checklist from historical reviews and selecting its granularity with ECA beat prompting a strong model to write a checklist directly? The official NeurIPS checklist does not answer this.

We compare our approach against the official NeurIPS checklist, which we expect to be a stronger baseline than a one-shot prompted LLM checklist. The official NeurIPS checklist was written by domain experts, so it is the harder comparison, and we have no reason to believe a prompted LLM would produce a clearly better list. However, we follow the reviewer’s suggestion and did the same simulation with a 110-item checklist generated by Fable 5.2 - High model. At every tested committee size, our best learned checklist beats the Fable-generated checklist by 19% to 28%.

Moreover, many items in the Fable-generated checklist also cannot be checked against the paper. About a quarter are instructions to the reviewer, such as checking for conflicts of interest or updating the score after the rebuttal. Some call for a high-level judgment, such as whether the contribution is significant to the community. Others can only be settled with sources outside the paper, such as whether the closest prior work is cited.

Q6. Is the human-study target review quality? Participants are scored by how closely their rankings match the historical NeurIPS 2021 ordering, and ranking MSE falls from 0.48 to 0.22. If that historical ordering is itself noisy, why is it the target? Does the drop show more accurate judgment, or only closer agreement with that venue and year?

While it would be ideal to collect many reviews per paper in our human study to create a new, highly stable population target, the amount of expert access this requires is difficult for an evaluative study. Instead, we use papers for which we observe 8 human reviewers’ judgments (a relatively large number for historical review datasets), and to reduce the chance of noise in the target ordering, we chose papers with large gaps in mean score (7.0, 6.0, 5.4 and 7.0, 6.0, 5.25; Appendix C) so that this proxy is as faithful as possible. Given that the checklist aims to reduce variance without adding too much bias relative to the population target, the relative quality of the papers is a reasonable proxy for the target in an initial, proof-of-concept human study. We will add a statement about the value of a large-scale human study as future work, which would likely rise to the level of its own stand-alone contribution given the difficulty of conducting such experiments.

Q7. Where is the selected checklist? The supplementary material does not include it. Can the authors release the full 100-item checklist selected by the method?

We will add the selected 100-item checklist to the appendix.

Q8. Does the human study have a control checklist? All 20 participants compare a pre-filled checklist with no checklist. Does the gain come from this learned checklist, or from having any structured checklist? Can the comparison be blinded against a checklist generated directly by a strong LLM, so participants do not know which list is the proposed method?

We like the idea of adding a control checklist will make the human study more convincing, and we will add one if time allows before the camera-ready. Our current human study is intended primarily to show that the pipeline-generated checklist also works with real human reviewers. If we cannot finish running the additional control in time, we will further acknowledge this limitation and emphasize that the human study is a proof-of-concept experiment in the paper.

Q9. Is lower score variance the right objective? Acceptance decisions are made by area chairs and depend on more than scores. Peer review also helps authors find weaknesses and improve the manuscript. Does the paper show that the checklist produces more useful feedback, helps authors address real problems, or improves manuscript quality?

We optimize checklists for the objective of reducing total review error. The checklists we learn do this by surfacing verifiable signals. By design, the signals that end up in the checklist are concrete, checkable problems in the paper. It is hard to imagine that fixing them would make the paper worse. How authors revise from the checklist, and how the revised manuscript changes, is beyond the scope of this paper.

I appreciate the authors' efforts and the additional experiments, and I agree to raise my score to 2. I find the proposed method itself reasonably effective. However, I still have fundamental concerns about the fairness of the experimental comparisons and, more importantly, whether the approach is sufficiently aligned with actual human reviewing practices. The current human study does not convincingly demonstrate that the learned checklist improves human reviewers' judgments beyond simply providing structured guidance. Regarding the broader value and significance of the approach, I believe this is partly a matter of perspective and will leave the final judgment to the AC. These remaining concerns prevent me from supporting a score of 3.

Reviewer_XozSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes learning review checklists from peer reviews to reduce variation across reviewers. The approach generates a tree of checklists representing coarsening of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. They use an LLM-assisted pipeline to implement this and apply it to NeurIPS and ICLR papers. In simulations with 30 papers, they found it reduced MSE by 58%. They also found in a human study with 20 participants, the NeurIPS MSE was reduced.

Critical strengths

The problem is very important in my opinion. Furthermore, the solution they propose is nice and is shown to work. They produce a lot of nice experiments, and comparisons. They check against existing checklists like the NeurIPS checklist. The experiments are set up well such as separating variance reduction from mean scores, and I also appreciate that the work seems to be quite novel. I think one of the biggest strengths is the human studies.

Critical weaknesses

I think the biggest weakness is that I'm not sure if all the confounders are accounted for. Specifically, the gains could be explained by allowing reviews to consider more issues and enforcing that they spent more time on this. So restricting reviewer resource budget as a baseline would be nice to see. In addition, the simulation does not establish improved scientific judgement. They try to recover unaided reviewer-population mean. This is not establishing a more accurate assessment of paper quality. I also think the human study is too limited: it has twenty participants on six papers across two topics. Also, it seems that the papers used were the camera-ready papers rather than the submissions the reviewers saw. I think this should somehow be addressed in some of the error-rates computed.

Questions for authors

What were the humans' levels of expertise? How are the aforementioned confounders accounted for? Were the papers the camera-ready papers? If so, how was this accounted for, since these are not the papers the reviewers saw. Does this technique generalize across venues and different conference years?

4 comments

We thank the reviewer for the thoughtful review and for the positive assessment of the problem, the experiments, and the human study. The replies below follow the order of questions in the Weaknesses section.

Q1. Are the gains from the checklist, or from more issues and more time? The gains could come from letting a review consider more issues and spend more time. A baseline that holds the reviewer resource budget fixed would address this. How are these confounders accounted for?

Thank you for the question. We restrict the number of issues that an unaided reviewer attends to to model the simulation after human reviewers’ behavior. As observed in our analysis of the 2021 NeurIPS review corpora, a human reviewer raises about 6% of the issues a larger pool would raise, and most reviews do not overlap (Section 3). To mimic this limitation, we gave the unaided reviewer only 10 checklist items at the k = 50,000 level, randomly sampled from different branches at k = 50. By doing so, unaided reviewers will focus on a limited, non-overlapping set of problems in the paper. Hence, these choices were made in the interest of making our simulation better resemble conditions for human review.

For the checklist-aided reviewers, we gave them one shared checklist at different coarsening levels as the intervention. For example, when they are assigned the k = 100 checklist, they may read all 100 items and make their decisions according to the checklist. This also simulates how the checklist will be used in real review processing, where each reviewer will be given an LLM-prefilled checklist to help them make better decisions.

A longer list is not itself the gain. For example, the 2,000-item checklist has higher MSE than the 100-item checklist: 0.18 against 0.10 for a committee of four on NeurIPS, and 0.29 against 0.21 on ICLR (Section 5). However, it is an excellent idea to systematically model the impacts of cognitive load in future work, for example by giving reviewers different amounts of time to review, or restricting checklist-aided reviewers to pay attention only to a subset of provided checklist items, and examining how the impacts of the checklist change.

We will describe our rationale for the simulation design more clearly and discuss the limitations and future work.

Q2. Does the simulation show better scientific judgment? The simulation recovers the mean of an unaided reviewer population. That is not a more accurate assessment of paper quality.

While we can understand the tendency to want a more “objective” measure of paper quality, we intentionally avoid treating paper quality as something that exists independent of human judgment. This is in line with modern metascientific and philosophy of science perspectives on what role evaluations like peer review serve in science (see e.g., Popper 2005, Longino 1990). Instead, we assume that the goal of review is to aggregate the opinions of the eligible reviewer population, and our simulation experiment models a large population of unaided reviewers, where the target is their average score. We will add further motivation for why we do not attempt to measure scientific quality independently of human assessments. We also note that our view is not in conflict with the idea that some dimensions of a paper can be measured more objectively; our checklists focus on concrete aspects of a paper that can be verified against it.

Longino, H. E. (1990). Science as social knowledge: Values and objectivity in scientific inquiry. Princeton university press.

Popper, K. (2005). The logic of scientific discovery. Routledge.

Q3. Is the human study large enough? It has twenty participants, six papers, and two topics.

We designed our study to have adequate statistical power to detect effects (in terms of error reduction) of about 0.3 or more and will report this in the paper. Error in the study is the share of paper pairs a participant orders against the NeurIPS ranking, so 0.5 is chance. The error falls from 0.48 without the checklist to 0.22 with it, a drop of 0.26 (paired t-test, p = 0.03; 95% CI 0.03 to 0.49). The drop varies widely across participants (standard deviation 0.47), so a study of this size has 80% power only for drops of about 0.3 or more. The drop we observe is close to that size. The study is a proof of concept, and we acknowledge the limited scale: with 20 participants and six papers, it cannot detect smaller effects or estimate effect size precisely. However, we felt it was important to provide some evidence that our approach could yield measurable gains with human participants.

Q4. Were the papers camera-ready? If the study used camera-ready papers rather than the submissions reviewers saw, how does that enter the error rates?

Yes, both the human study and the claim verification used the camera-ready version of PDFs. The anonymized submissions are not in the public NeurIPS 2021 release, so we cannot recover them.

However, this does not limit the result for our main simulated study. The simulated aided and unaided reviewers, and the target population, judge the same manuscript. The review MSE is measured based on the same paper.

However, this will affect the refuted-claim rate we report in our analysis, making it look higher (since some papers may have fixed the error mentioned in the original reviews) and impact the human ranking study (the NeurIPS ranking score is based on the original anonymized submissions). We will state these limitations more clearly in the paper.

Q5. What was the participants' expertise? What were the humans' levels of expertise?

The human reviewers are all CS PhD students with experience in AI/ML.

Q6. Does the method generalize? Does it generalize across venues and across conference years?

The pipeline is general: run it on a corpus, and it learns a checklist. The items are specific to the distribution of papers that comprise that corpus. If a review corpus is specific to a particular venue, we do not claim that a checklist learned at that venue will transfer to another. While it’s possible similar venues could use the same checklist, or a checklist could be learned across multiple venues, evaluating this is beyond the scope of our paper.

However, cross-year generalization is worth measuring directly, and we ran this experiment for the rebuttal. We applied the NeurIPS 2021 checklists directly to 9 NeurIPS 2022 submissions (3 strong, 3 borderline, and 3 rejected) under the same protocol. We see mostly the same pattern as in 2021: for a typical committee of three reviewers, the 2021 learned k = 100 checklist reduces review MSE by 44% (from 0.15 unaided to 0.08), and has the lowest error among both the official NeurIPS checklist (0.24) and all other checklist granularities (0.12 to 0.18).

Thanks again for your review. We are happy to answer any questions you might have on the new information we shared before the rebuttal period closes.

Thank you for the helpful response. It addresses my points and I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.