acceptodds
Under review as a conference paper at ICLR 2027

When Can an AI Reviewer Replace a Human? Human-Equivalence Selective Review

Abstract

As AI systems spread through the research pipeline, peer review faces pressure to scale without losing human judgment; generating a plausible critique is different from reproducing how human reviewers decide. We study 454 ICLR submissions from 2024–2026 with 1,810 public human reviews and evaluations from four foundation models. The sharpest reliability gap occurs in the half-level region around the human recommendation boundary: moving outside this region improves AI–human agreement, although rare extreme scores are not monotonically safer. This observation yields Human-Equivalence Selective Review (HESR), a fixed assignment rule that lets one AI review occupy a review slot when its ordinal score lies at least half a level from the human-aligned boundary and otherwise routes the paper to a person. HESR requires no training, model-specific parameters, batch ranking, or multi-model inference. Across 99 deduplicated papers and 396 model–paper judgments, HESR retains 54.8% of reviews and raises agreement with individual human recommendations from 67.6% to 73.6% (paper-cluster bootstrap 95% CI for the gain, 2.4–9.8 percentage points). Historical decision accuracy increases from 73.5% to 87.1%; all four base models and all six evaluation blocks improve, with 109 accept-side and 108 reject-side assignments. Two frozen tests reproduce the effect on 40 unseen papers. On a quality-controlled cross-year test, the gain remains positive after adversarially removing one human review from every paper. These results show that a single AI reviewer can match the variability of human recommendations on an identifiable subset of papers while preserving human review for the least stable cases.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI peer review, large language models, human–AI alignment, selective prediction, reviewer allocation, scientific evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3AvsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks when an AI reviewer can reliably replace a human reviewer in peer review. The authors observe that AI–human disagreement is concentrated near the accept/reject decision boundary, while judgments farther from the boundary are generally more aligned with human recommendations. Based on this observation, they propose HESR, a selective review rule that lets AI occupy a reviewer slot only for judgments sufficiently far from the boundary and defers ambiguous cases to humans.

Critical strengths

The paper highlights an interesting empirical pattern in the studied setting: AI–human recommendation agreement is lower near the accept/reject boundary than in adjacent score ranges.

Critical weaknesses
  • The evaluation is quite narrow, focusing primarily on recommendation agreement while leaving other important aspects of review quality unassessed. It remains unclear whether the selected AI reviews provide technically sound and well-supported critiques, reliably identify substantive flaws, or offer genuinely useful feedback to authors.
  • The experimental protocol lacks sufficient transparency. Key details about the relationships among evaluation stages and the comparison procedures remain unclear, making it difficult to assess comparison fairness and the robustness of HESR’s reported transfer.
  • The evidence for HESR’s generalizability remains insufficient. The fixed margin is an empirically chosen heuristic, and the overall evidence covers only 99 papers, including just 40 frozen-test papers, providing limited support for its reliability across broader submission populations and different review settings.
Questions for authors
  • Please provide: (i) how each data block was sampled and used in method development and evaluation, any overlap between blocks, and the LOO procedure; (ii) implementation details, comparison settings, and per-block results for the alternatives in Sec. 4.1; and (iii) reviewer prompts and inference settings.
  • Could the authors assess HESR’s sensitivity to the margin choice and validate the fixed rule on a larger, more diverse independent sample beyond the 99 papers studied, including the 40 frozen-test papers?
  • For the subset selected by HESR, could the authors assess review quality beyond recommendation agreement and provide representative case studies of both successful and unsuccessful AI reviews, comparing them with human reviews in terms of factual correctness, technical soundness, and identification of substantive flaws?
Reviewer_rKfJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when an AI-generated recommendation can occupy a human reviewer slot. It proposes Human-Equivalence Selective Review (HESR), a fixed assignment rule based on the distance between an AI’s score and a shared recommendation boundary. On the harmonized 0–5 scale, AI recommendations with scores at or below 2.5 or at or above 3.5 are retained, while the remaining cases are deferred to human reviewers. The rule requires only one review from a single model. The study draws on 454 ICLR submissions and 1,810 human reviews. Its pooled HESR evaluation contains 99 unique papers and 396 judgments from four independently evaluated models. HESR retains 54.8% of judgments and increases agreement with individual human accept/reject recommendations from 67.6% to 73.6%. The paper also reports historical decision accuracy, human–human agreement, two frozen tests, and several robustness analyses.

Critical strengths
  1. The paper addresses a practically relevant allocation problem with a simple, transparent rule. Deciding which AI recommendations can be used and which should receive human attention is an important operational question. HESR is inexpensive to apply and does not require model ensembles, repeated sampling, or batch-dependent ranking.

  2. The empirical evaluation includes several useful controls. The authors evaluate four models separately, conduct two frozen tests, compare against human–human agreement, and examine sensitivity to changes in the human reference. These analyses provide more informative evidence than reporting agreement with historical decisions alone.

Critical weaknesses
  1. The retained score region is not sufficiently justified by the reported reliability pattern.

Figure 2 shows that AI–human agreement is 74.8% at distances of 0.5–<1.0 from the boundary and 75.7% at distances of 1.0–<1.5, but falls to 59.6% for distances of at least 1.5. Human–human agreement in this extreme-score group is 73.5%. Nevertheless, HESR retains all judgments at distances of at least 0.5, including this relatively unreliable group. The authors explicitly acknowledge that reliability is non-monotonic, but do not adequately establish why an unbounded retained region is preferable to a bounded interval.

  1. The methodological contribution is limited and consists mainly of an empirical statistical observation.

The central method identifies an association between score regions and AI–human recommendation agreement, then converts this association into a fixed threshold rule. The assignment mechanism itself is straightforward, and the paper provides limited additional methodological innovation. An empirical finding can be valuable, but its contribution would be stronger with deeper explanations of why boundary judgments disagree with humans and why extreme judgments become unreliable.

  1. Differences in model scoring tendencies and the transferability of the fixed threshold are insufficiently characterized.

The paper compares four models and considers provider-specific score mappings, but does not fully establish that the same absolute score distance indicates comparable reliability across models. Models may differ in scoring strictness, dispersion, and use of intermediate scores despite identical semantic anchors. Pooled coverage ranges from 46.5% for GLM to 69.7% for GPT, with substantial variation in agreement gains. Qwen’s gain on the cross-year frozen test is below 0.1 percentage points. These results warrant closer analysis of model-specific score distributions, systematic biases, and agreement by boundary distance.

For example, two models may recommend acceptance yet assign 3.2 and 3.8 to the same paper, producing different assignment decisions due to scoring conventions. Stronger evidence is needed to distinguish these conventions from actual reliability differences. All four models also participated in method development. Validation on a model excluded from development would strengthen evidence that the fixed threshold transfers across models.

Questions for authors

See Critical Weaknesses.

Reviewer_M1QGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The manuscript argues that humans can only spend their effort for borderline papers, rather than focusing on clear-accept or clear-reject ones. The manuscript presents an heuristic rule (i.e., +-0.5 the borderline) to decide when to avoid AI-based reviewing tools.

Critical strengths

This paper has been generated entirely (as also written in AI USE STATEMENT at page 8), and it has many flaws, both on the formulation of the scientific questions and the methodology.

Critical weaknesses

The novelty of the paper rises in setting an heuristic threshold, but it is not clear why exactly this rule. The paper introduces Human-Equivalence Selective Review without explaining what it is. The problem setting is formalized and never really used, the key concept of the paper are not explained. Table 2 is not even explained, but these metrics are used heavily in experiments. While I can read that they are informally described, I can not guess what they actually measure. Table 3 is not explained, all results are so obscure that I honestly do not have understood the scope of the paper.

In general, informally-speaking everything is inside the paper. But this is not an informal place, it is ICLR, a scientific venue, and a manuscript should rigorously assess all its components. Also, the code does not allow to reproduce the work, but just computing the plots without interactions with local or remote LLMs (which is not useful).

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.