acceptodds
Withdrawn from ICLR 2027

The Predictable Core of Peer Review

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Peer reviewers often assign different scores to the same paper. Across 25,579 ICLR papers from 2020–2025, the Pearson correlation between one reviewer's score and the mean of the remaining reviewers ranges from 0.435 to 0.505. We show that collective ratings contain patterns that a small text model can learn. Our model combines word and character TF-IDF features with ridge regression over identity-filtered scientific text. On all 1,028 public NAIDv2 test papers, it reaches Spearman correlation 0.522, surpassing the dedicated NAIPv2-8B scorer's 0.439 and reducing calibrated mean-squared error by 7.03%. The comparison uses the same papers, outcomes, and calibration protocol, with broad manuscript text for the sparse model and NAIPv2's native title-and-abstract interface. Annual forward evaluations, presentation controls, and fixed-budget reading experiments trace the advantage to information beyond the opening text. Reading more of the manuscript reduces NAIPv2's added benefit for mean-score prediction by 34.8%. Human judgments supply further information after the manuscript model. Sparse learning thus reveals a predictable core in collective ratings while individual reviews contribute additional signal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
automated peer review, review score prediction, reviewer agreement, scientific document understanding, sparse text models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yC4XOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that a simple text-based TF-IDF + ridge regression model can predict reviewers’ average scores, and that giving the model access to more manuscript content substantially improves its predictive performance. It also finds that one reviewer’s score still helps predict another’s after controlling for the manuscript-based prediction.

Critical strengths

The experimental design is simple and clear, and the paper is easy to follow. The presentation-control analyses and fixed-budget reading experiments offer useful evidence that access to manuscript content beyond the title and abstract helps explain the observed performance gains.

Critical weaknesses
  • The evaluation against stronger neural/LLM-based scorers is quite limited, relying primarily on a single comparator, NAIPv2-8B. Moreover, this comparison does not isolate scoring capability from input scope: the sparse model uses broad manuscript text, whereas NAIPv2-8B receives only titles and abstracts. Since the sparse model is weaker under matched title-and-abstract inputs, the reported advantage mainly demonstrates the benefit of broader manuscript access rather than a stronger scoring method under comparable input conditions.
  • Incomplete manuscript provenance leaves potential temporal leakage unresolved. Some inputs may include revisions made after the target scores were assigned, allowing predictions to rely on text influenced by the review process.
Questions for authors
  • Could the authors compare against more LLM-based scoring baselines, ideally with matched manuscript access? Since the main advantage seems tied to broader text access, this would clarify whether the gain reflects the scoring method itself or simply the information available.
  • How well do these findings generalize beyond ICLR? The current analysis is restricted to a single machine-learning conference, and it is unclear whether the same predictable patterns would hold across other venues or scientific fields with different reviewing criteria and writing conventions.
  • To what extent does the “predictable core” reflect topic- or subfield-specific lexical correlations? Since the presentation controls do not explicitly account for topic, could the authors report topic-controlled analyses or cross-subfield generalization results to assess whether the learned signal transfers beyond these correlations?
Reviewer_EhPoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the issue of score discrepancies among different reviewers in the peer-review process of academic papers by investigating whether a collective review score can be predicted based solely on the text of a paper. The authors develop a lightweight scoring prediction model that combines word-level and character-level TF-IDF features with ridge regression, conducting experiments using historical papers and their corresponding review data from ICLR. The results demonstrate that this approach achieves a Spearman correlation coefficient of 0.522 on the NAIDv2 test set, outperforming the NAIPv2-8B model, which yields 0.439. Further analysis reveals that while the main text of a paper enhances the model's predictive capability, human-written review comments still provide additional information that the model has yet to capture.

Critical strengths
  1. Starting from the differences in reviewers' scoring, this article explores whether there are learnable collective scoring patterns in the paper text, providing a worthy research question for automated academic evaluation.

  2. The proposed method is straightforward and computationally efficient. By employing traditional TF-IDF features alongside a ridge regression model, it achieves score prediction without the need for training complex, large-scale models, offering a significant advantage in terms of implementation costs.

  3. In addition to comparing the main models, the article also conducted cross-year analysis, comparison of different text lengths, text feature control, and analysis of incremental information from manual scoring, which can explain the experimental phenomena from multiple perspectives.

Critical weaknesses
  1. The paper claims that a simple TF-IDF/Ridge regression model outperforms NAIPv2-8B; however, the input conditions for the two are fundamentally different. The former utilizes the full text of a paper (up to 64,000 characters), whereas the latter relies solely on the title and abstract (capped at 512 tokens). Consequently, the current experiments merely demonstrate that full-text content contains more predictive information; they do not prove that the proposed model itself is superior to the 8B-parameter model.

  2. If part of the full text comes from a revised version after peer review, its content may have been affected by the peer review comments, resulting in the prediction results not accurately reflecting the prediction ability of the score at the time of submission. Furthermore, both the training and test sets in the primary experiments include papers from 2025. The lack of strict temporal separation and version control for the manuscripts undermines the credibility of the core findings.

  3. The paper’s central methodology is limited to traditional TF-IDF feature extraction combined with Ridge regression. Its theoretical analysis relies on standard squared-error decomposition, offering no novel learning algorithms or mechanisms for modeling peer-review scores. As such, the work functions primarily as an empirical analysis of conventional text-regression techniques, which is insufficient to substantiate its stated focus on the fundamental mechanisms underlying the predictability of peer-review outcomes.

  4. The text matching experiment of 94 papers in 2026 in the appendix shows that the correlation coefficient of the sparse model is only 0.119, while ChatGPT reaches 0.308, indicating that the performance advantage is not stable under different data conditions. Although the authors conduct a cross-year analysis, they primarily report conditional correlations; the absence of a prospective performance comparison, makes it difficult to convincingly demonstrate the model's practical utility.

Questions for authors

Please see the weaknesses.

Reviewer_oRrjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how accurately collective peer-review scores can be predicted from manuscript text despite disagreement among individual reviewers. It proposes a TF-IDF word-and-character representation with ridge regression over identity-filtered broad manuscript text, and compares it with NAIPv2-8B. Evaluations on NAIDv2, annual forward cohorts, fixed reading budgets, and distinct-reviewer prediction assess predictive performance and complementary human signal.

Critical strengths
  1. The paper makes the input-scope comparison concrete and quantitatively convincing. In Table 2 (P4), moving the same TF-IDF/ridge family from title-and-abstract input to broad text increases Spearman correlation from 0.3427 to 0.5215 and surpasses NAIPv2-8B’s 0.4388 on the same 1,028 papers. This is the paper’s most consequential result because it demonstrates that manuscript access, rather than merely a larger language model, can be decisive in this evaluation setting.
  2. The analysis goes beyond a single aggregate benchmark result. Table 3 (P5) shows positive broad-text partial correlations after controlling for short text across annual forward cohorts, while Table 5 (P6) correctly frames human-review information as complementary rather than interchangeable with a manuscript score. These analyses make the “predictable core” framing more substantive than a one-off leaderboard comparison.
Critical weaknesses

The headline result does not convincingly establish whether peer-review outcomes can be predicted from manuscripts as they existed during the review process. My concerns are as follows:

  1. The paper's central claim relies heavily on the 1,028-paper public benchmark in Table 2. However, Appendix A (Page 9) acknowledges that 277 papers were obtained from public mirrors whose original-submission versions remain unverified. More broadly, the provenance of original submissions is incomplete, raising concerns about whether the evaluated manuscripts correspond to the versions actually reviewed.

  2. Later manuscript versions may incorporate changes made in response to reviewer feedback, rejection, or acceptance. Such revisions could introduce outcome-correlated scientific content that is not necessarily associated with author identity and therefore would not be eliminated by the proposed filtering strategy. Consequently, the observed predictive performance may partly reflect post-review information rather than signals available to reviewers at the time of evaluation.

  3. A indicates that the public partition and annual cohorts were examined during development, raising concerns about potential analysis-selection bias. Moreover, Appendix B clarifies that the reported uncertainty intervals condition on fitted models and analyzed comparisons, without accounting for model-refitting or selection uncertainty. These limitations weaken the robustness of the reported findings.

  4. TF-IDF may exploit differences in average ratings across research areas. Prior work has reported associations between vocabulary, research topics, and acceptance outcomes (Vincent-Lamarre and Larivière). The presentation controls in Section 4.2 do not explicitly control for topic, and within-topic evaluations are not reported. It therefore remains unclear how much of the predictive performance reflects differences among papers within comparable research areas.

  5. Section 4.4 shows that NAIPv2 contributes less after broad-text prediction. This is consistent with overlapping predictive signals, but may also reflect lower residual error and reduced room for improvement. The experiment establishes a change in incremental predictive value without identifying its source.

  6. Section 5 and Table 5 show that another reviewer’s score improves prediction after conditioning on the manuscript model. This establishes residual predictive association between reviewers, but does not explain whether the shared signal reflects scientific content, common criteria, or shared preferences. The paper should more clearly articulate what this analysis adds beyond existing knowledge of reviewer agreement.

  7. Direct scoring with general-purpose LLMs is examined only on small supplementary samples in Appendix C. Although some relevant methods are cited, a systematic comparison with representative alternatives under common targets and reasonably matched inputs is missing.

Questions for authors
  1. In Appendix A, can you report the complete Table 2 comparison separately for the 751 historical-archive records and the 277 public-mirror records with unverified original-submission provenance? If the broad-text advantage is comparable in both strata, it would substantially increase my confidence that the result is not driven by version provenance.

  2. For the annual forward evaluations in Section 4.1, what fraction of the broad-text manuscripts can be verified to be the version available during the relevant review period? An analysis limited to those records would clarify whether the annual results support deployment-time prediction rather than retrospective association.

  3. Appendix A notes that the public partition was examined during development, and Appendix B notes that reported intervals exclude selection uncertainty. Which decisions among character budget, feature limits, regularization, cleaning/routing rules, and the broad-text view were finalized before inspecting public-test outcomes? This determines how much confidence to place in the precise 7.03% MSE-reduction estimate.

  4. Could the authors provide representative specialized-model and general-purpose LLM baselines with reasonably matched manuscript access? How do the positive broad-text results relate to NAIPv2’s existing input-expansion experiments?

  5. How much predictive value remains after controlling for research topic or evaluating within topic groups?

  6. How can Section 4.4 distinguish information overlap from reduced improvement headroom? What additional insight does Section 5 establish beyond residual association between reviewer scores?

  7. Could the authors resolve the eight candidate train–test title overlaps and report results after excluding overlapping document families and restricting evaluation to verified submission versions? Please also clarify how test results informed development and report MSE using calibration fitted on an independent reserve set.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.