acceptodds
Under review as a conference paper at ICLR 2027

Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories

Abstract

AI reviewers are increasingly used to assess scientific manuscripts, yet it remains unclear whether their judgments reflect the improvements authors make through revision. We investigate this question using **PaperTrajectory-2026**, the 2026 release of **PaperTrajectory**, a dataset of 6,959 paper trajectories across five AI conferences, with automatic annotations of changes along 14 dimensions and separate judgments of overall quality. The annotations show that major improvements emphasize empirical evidence and framing, while later revisions tend toward incremental improvement. Using independent assessments of manuscript versions from eight AI reviewers, we examine whether annotated major improvements receive higher scores, which revision dimensions are associated with score changes, and whether user opinions can outweigh observed revision gains. Every reviewer leaves scores unchanged on most major improvements, and agreement on the direction of score changes is weak. Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers. Even when a revision earns a higher neutral score, a brief negative user opinion can erase or reverse that advantage, with substantial variation across reviewers. These findings show that substantive revisions are not consistently reflected in review scores and that observed scoring gains can be vulnerable to user framing, motivating assessments that connect manuscript evidence to overall judgments and remain stable when no scientific evidence changes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Reviewers, Scientific Peer Review, Sycophancy, Self-Preference Bias, Manuscript Revision, Large Language Models, Evaluation Bias

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VYfgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces PaperTrajectory-2026, built from the arXiv version histories of 6,959 papers accepted at ICLR, ICML, ACL, CVPR and ECCV 2026. Versions more than 30 days apart are chained into trajectories and adjacent versions are paired (9,092 pairs, 9,066 with completed annotations). GPT-6 Astra reads both full manuscripts and labels the magnitude of change on 14 dimensions with before-and-after evidence, plus a separate overall quality judgment (major or minor improvement, no meaningful change, deterioration, uncertain). Five graduate-level evaluators checked 150 pairs: 98.6% of dimension descriptions were rated accurate, and 74 of 75 pairs labeled as major improvements were judged improved to some degree. Descriptive analyses show that major improvements tend to revise evidence and framing together, and that later revisions are narrower than earlier ones.

The main study reviews both versions of 1,484 major-improvement pairs independently with eight LLM reviewers, using venue-specific prompts and rating scales. Every reviewer keeps the same overall score on most pairs (55% to 78%), and only 21% to 41% of pairs receive a higher score (Fig. 3, Tab. 9). Agreement across reviewers on the direction of change is close to chance (Fleiss' kappa 0.063). Revision-dimension indicators are associated with score increases in reviewer-specific ways and give held-out AUCs between 0.507 and 0.641. When the user message expresses disappointment with the revised version, many of the observed neutral gains turn into ties or reversals, from 24.7% of gains for GPT-6 Astra to 98.5% for Claude Haiku 4.5 (Tab. 21).

Overall, the paper investigates whether AI reviewers can recognize improvements introduced through revisions to accepted papers. However, the study still lacks sufficient rigor in establishing ground truth, accounting for heterogeneity among AI judges (Only GPT-6 Astra as an annotator), and providing clearer and more generalizable empirical evidence to support its findings.

Critical strengths

One critical strength for the paper to be accepted is as below:

The paper raises a timely question with real revision data and a good experimental attempt. With AI assistance, submission numbers to any ai conferences now are growing explosively, and human expert reviewing cannot grow fast enough to keep up, so whether AI reviewers' scores reflect the real improvements authors make is a timely question. The dataset is built from revisions that authors actually made, and the dataset and the experimental design are a good attempt at studying this question.

Critical weaknesses

Two critical weakness that may lead to support rejection for the paper as below:

  1. The evidence is not solid enough: the annotation relies on a single model, and there is no ground truth. All change descriptions and quality labels, including the "major improvement" label that defines the review sample, come from one model, GPT-6 Astra (Sec. 3.3). If that model is biased, every downstream result inherits the bias; the labels should be produced or checked with multiple LLMs or with human experts. There is also no ground truth for each paper against which the AI reviewers' score changes, or the improvement labels, can be matched or verified.

  2. Finding 3 may have issues. The claim that a brief negative user opinion can erase or reverse a revised paper's score advantage (Sec. 7) does not feel solid to me. As one of the paper's three headline findings, it needs stronger support than the current analysis provides.

Questions for authors

Clarification requests for below three questions, if solved properly, that may change my assessment:

  1. How stable are the labels if a subset is re-annotated by other LLMs or by human experts?

  2. Is there a ground truth for each paper,since the papers are accepted to the conference already, u still cannot obtain the real scores for each paper? Or for example expert judgments of the same versions, against which the AI reviewers' score changes and the improvement labels could be verified?

  3. What further evidence supports Finding 3?

7 comments
AuthorsOpenReview ↗

Response to Reviewer VYfg

Response to Weakness 1 and Questions 1–2

“The evidence is not solid enough: the annotation relies on a single model, and there is no ground truth.”

“How stable are the labels if a subset is re-annotated by other LLMs or by human experts?”

Thank you for your valuable comment. We re-annotated 225 revision pairs using four additional models, with 75 pairs from each original improvement category. Each model received the complete manuscripts and the same rubric, without Astra’s labels or explanations.

Table 1. Cross-model consistency of improvement annotations. Overall agreement is calculated across 225 pairs. Category-specific retention rates use the 75 pairs originally assigned to each category by Astra.

Independent annotatorExact agreement with AstraMajor improvement retainedMinor improvement retainedNo meaningful change retained
GPT-5.6 Sol80.9% (182/225)98.7% (74/75)56.0% (42/75)88.0% (66/75)
Gemini 3.8 Flash86.2% (194/225)86.7% (65/75)73.3% (55/75)98.7% (74/75)
Claude Sonnet 581.3% (183/225)74.7% (56/75)73.3% (55/75)96.0% (72/75)
GLM-5.3 Flash84.9% (191/225)77.3% (58/75)78.7% (59/75)98.7% (74/75)

Exact agreement with Astra is 80.9–86.2%, and major-improvement retention is 74.7–98.7%, supporting label consistency across model judges.

Conference acceptance is a paper-level outcome; our analysis requires judgments of specific manuscript versions. We sampled 30 revision pairs and had three human reviewers independently score both versions under the corresponding conference criteria, with version order, Astra labels, and AI scores hidden. Majority vote determined score-change direction. The same reviewers independently assessed improvement magnitude, providing a human reference for the labels.

Table 2. Human reference judgments and AI reviewers’ score-change directions. Score-change directions compare the revised and earlier versions. Agreement measures whether an AI reviewer’s direction matches the human majority judgment for the same pair.

ReviewerHigher score (%)Unchanged score (%)Lower score (%)Agreement with human (%)
Human reviewers—majority vote36.760.03.3—
GPT-5.6 Sol40.053.36.756.7
GPT-5.546.746.76.753.3
GPT-6 Astra33.363.33.383.3
Gemini 3.5 Flash17.279.33.451.7
Gemini 3.8 Flash37.958.63.431.0
Claude Sonnet 536.763.30.040.0
Claude Haiku 4.530.056.713.346.7
GLM-5.3 Flash27.669.03.455.2

Human majority judgments assign unchanged scores to 60.0% of pairs, while agreement with this reference varies across AI reviewers. These checks support label stability and provide a version-specific human reference for evaluating score changes beyond Astra’s judgments.

Response to Weakness 2 and Question 3

“The claim that a brief negative user opinion can erase or reverse a revised paper's score advantage (Sec. 7) does not feel solid to me.”

“What further evidence supports Finding 3?”

Thank you for your valuable comment. For each reviewer, we fixed pairs with an initially positive neutral score gap and re-evaluated the revised manuscripts under neutral and negative-opinion conditions. Manuscript content, review criteria, and earlier-version scores remained fixed. We measured loss of the advantage as a tie or strict reversal.

Table 3. Loss of revision score advantages under neutral repetition and negative opinions. Entries report tied / strictly reversed gaps (%). Gain-loss combines these outcomes; the final column reports the difference between conditions in percentage points, with 95% confidence intervals.

ReviewerNeutral repeat: tied / reversed (%)Negative opinion: tied / reversed (%)Difference in gain-loss rate (pp; 95% CI)
GPT-5.6 Sol5.9 / 0.058.1 / 0.0+52.2 [38.8, 66.1]
Gemini 3.8 Flash9.9 / 2.021.9 / 9.9+20.0 [8.1, 32.1]
Claude Sonnet 515.9 / 0.058.4 / 15.8+58.3 [44.3, 70.8]

Negative opinions increase the gain-loss rate by 20.0–58.3 percentage points relative to neutral repetition, with all reported confidence intervals excluding zero. Sol exhibits erasure without strict reversal, while Gemini and Sonnet exhibit both outcomes. These controls support Finding 3: unsupported negative opinions destabilize revision score advantages beyond the losses observed under neutral repetition.

Reviewer_VYfgOpenReview ↗

Response to the above 2 sections

Thanks to the authors for the new experiments. They respond directly to the three points I raised, and I have a few follow-up comments.

On the single annotator (Q1), the re-annotation is what I was hoping to see. With three balanced categories, 81–86% exact agreement across four other models is well above chance, and it makes me much less worried that the "major improvement" sample is an artifact of GPT-6 Astra. I would still like to know where Astra's major improvements go when another model disagrees. Claude Sonnet 5 keeps only 74.7% of them, and it matters whether the rest become minor improvements or no meaningful change. Please also say that only the overall label was re-annotated, not the 14 dimension descriptions, and note that GPT-5.6 Sol comes from the same provider as Astra. This table should go into the paper.

On ground truth (Q2), I appreciate the human study, although 30 pairs is small: a single pair moves each percentage by 3.3 points. Two things in Table 2 stood out to me. First, the human majority keeps the score unchanged on 60% of the pairs. If these pairs come from the major-improvement sample, then human reviewers also often leave the score where it was, and an unchanged AI score is not by itself evidence that the AI missed the improvement. I think Finding 1 and the title question should be framed with this in mind. Second, most AI reviewers agree with the human direction only 31–57% of the time. A reviewer that simply gave the same mix of higher, unchanged and lower scores as the humans would agree about 49% of the time by chance, so most reviewers are close to chance on individual pairs. Gemini 3.8 Flash has almost the same mix as the humans but agrees on only 31%. Astra is the clear exception at 83%, and since Astra also selects the sample, this deserves some discussion. Please also report how the 30 pairs were chosen, how often the three human reviewers agreed with each other, and the human judgments of improvement size, which the response mentions but does not show.

On Finding 3 (Q3), the neutral-repeat control is the right one, and it is the check the paper itself said was needed in Sec. 7.3. A gap of 20–58 points over repeated neutral reviews, with intervals excluding zero, convinces me that the negative opinion has a real effect beyond run-to-run noise and regression to the mean. I have two questions. Why only three of the eight reviewers? And why are the negative-opinion rates here much lower than in Table 21 of the paper? Sol goes from 80.9% to 58.1%, Sonnet from 94.2% to 74.2%, and Gemini 3.8 Flash from 60.1% to 31.8%. If this is a new sample or a fresh run, the single-run numbers in Fig. 4 and Table 21 seem to vary quite a bit, and the paper should report the new estimates, with sample sizes, next to the old ones. The opposing-opinion setting in Sec. 7.2 still has no matching control, so I would soften the claims there.

Overall, the new experiments go a long way toward my concerns, and the human reference and the neutral-repeat control make the paper more informative. I have raised my Informativeness score from 2 to 3. Because the human reference is still small and the new negative-opinion rates do not match the ones in the paper, I am keeping my overall rating for now.

Reviewer_6eMJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PaperTrajectory-2026, a dataset of 6,959 paper revision trajectories from five AI conferences, with annotations across 14 revision dimensions and separate judgments of overall quality changes. Using eight AI reviewers, the authors examine whether substantial improvements lead to higher scores, which revision types are associated with score increases, and whether user opinions affect these gains. They find that most improvements leave scores unchanged, scoring responses vary across reviewers, and negative user opinions can erase or reverse observed score gains.

Critical strengths
  1. The dataset is a valuable resource, especially given the limited accessibility of complete paper revision histories on OpenReview. By reconstructing these trajectories from arXiv and providing detailed revision annotations, it enables systematic research on how papers improve and how reviewers respond to those improvements.

  2. The study examines both whether AI reviewers reward substantive revisions and whether those rewards remain stable under user opinions, providing a broader assessment of their reliability across eight models.

Critical weaknesses
  1. The independent-review setting does not fully reflect how reviewers assess revisions in practice. During revision or rebuttal, reviewers typically consider their previous concerns, the authors’ responses, and the changes made when deciding whether to raise their scores. Evaluating each version independently therefore offers a limited view of whether AI reviewers can recognize and reward improvements in this context. Adding a revision-aware setting would strengthen the paper’s practical relevance.

  2. Using score increases as a measure of responsiveness to revision has limitations. Since review scores are discrete, meaningful improvements may not warrant a higher rating, particularly when other weaknesses remain. An unchanged score therefore does not necessarily mean that the reviewer failed to recognize the improvements.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6eMJ

Response to Weakness 1

“During revision or rebuttal, reviewers typically consider their previous concerns, the authors’ responses, and the changes made when deciding whether to raise their scores.”

“Adding a revision-aware setting would strengthen the paper’s practical relevance.”

Thank you for your valuable comment. We compared independent reviewing with a condition providing each model with its complete earlier review, including its score, strengths, and weaknesses. Revision pairs, model versions, scoring criteria, and generation settings were held fixed.

Table 1. Score responses with and without access to the previous review. Each cell reports higher / unchanged / lower scores (%) relative to the earlier-version score.

ModelIndependent reviewWith previous review
GPT-5.6 Sol36% / 56% / 8%54% / 46% / 0%
Gemini 3.8 Flash42% / 50% / 8%38% / 62% / 0%
Claude Haiku 4.544% / 40% / 16%16% / 84% / 0%

With previous reviews, unchanged scores remain the majority outcome for Gemini 3.8 Flash and Claude Haiku 4.5 and account for 46% of GPT-5.6 Sol’s assessments. Previous-review context has model-dependent effects, while the broader pattern of substantial improvements frequently receiving unchanged scores persists.

Response to Weakness 2

“Since review scores are discrete, meaningful improvements may not warrant a higher rating, particularly when other weaknesses remain.”

“An unchanged score therefore does not necessarily mean that the reviewer failed to recognize the improvements.”

Thank you for your valuable comment. Finding 1 measures how often major overall improvements lead to higher rating categories; recognition is assessed through the review text. A revision can strengthen a manuscript while remaining within the same category.

We used DeepSeek-V4.1-Flash to audit whether Astra’s later reviews acknowledged the corresponding new evidence or revisions, with scores hidden.

Table 2. Astra’s independent score responses across validation subsets. Percentages are calculated within each subset. In the final row, 42/50 denotes acknowledged revisions among 50 audited pairs; score-change percentages use the 42 acknowledged pairs.

Evaluation subsetNumber of pairsHigher score (%)Unchanged score (%)
Astra-labeled major improvements1,48421.0277.63
Human-confirmed improvement subset6036.763.3
Pairs with revisions explicitly acknowledged in Astra’s later review42/5031.069.0

The audit identified acknowledgment in 42/50 pairs (84.0%). Among these 42 pairs, 13 received higher scores, 29 retained the same score, and none received lower scores. Acknowledged revisions therefore still receive unchanged scores in most cases (69.0%). This supports separating recognition from rating changes; an unchanged score alone does not establish a recognition failure. We will make this interpretation explicit in the revised presentation.

Thank you for your responses. My major concerns have been addressed and I will maintain my positive rating.

Reviewer_vn6aOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether AI reviewers recognize improvements made during paper revisions. The authors construct PaperTrajectory-2026, containing 6,959 revision trajectories and 9,092 version pairs from papers accepted to five major conferences in 2026. GPT-6 Astra annotates changes across 14 dimensions and overall quality.

Eight AI reviewers independently evaluate 1,484 pairs labeled as major improvements. Despite these improvements, 55–78% of scores remain unchanged, while only 21–41% increase (Finding 1). The relationship between revision dimensions and score changes varies across reviewers, with low predictive performance (AUC 0.51–0.64, Finding 2). Furthermore, a brief negative user comment can eliminate or reverse the revised version's scoring advantage (Finding 3).

The authors conclude that reliable AI reviewing requires both sensitivity to substantive improvements and resistance to user-induced bias.

Critical strengths
  • Valuable dataset. The large collection of real revision trajectories, with traceable arXiv versions, is a useful resource. The 14-dimensional rubric is well documented, and the authors appropriately assess quality changes rather than assuming later versions are better.
  • Compelling evidence of susceptibility to bias. Finding 3 demonstrates that an unsupported negative comment can substantially alter review scores. The effect varies considerably across models: strict reversal rates are 54.3% for GPT-5.5 and 56.9% for Claude Haiku 4.5, but 0.0% for GPT-6 Astra (Table 21).
  • Careful statistical methodology. The paper employs paper-level bootstrapping, clustered standard errors, Firth regression, GroupKFold, and common-sample robustness checks.
  • Transparency. Annotation and review prompts are fully disclosed. The authors also acknowledge several limitations, including regression to the mean and exploratory feature selection.
Critical weaknesses
  • The central "major improvement" labels come from a single GPT-6 Astra judgment. Yet Astra itself assigns unchanged review scores to 78% of these pairs (Table 9). This discrepancy raises a fundamental question: does it indicate reviewer failure or unreliable improvement labels? The current design cannot distinguish between them.

  • The five human evaluators are coauthors, lack explicit domain-expertise matching, and evaluate versions side by side. They only confirm whether some improvement exists, counting even minor improvements as positive. Agreement on improvement magnitude is modest (α = 0.521). More importantly, an improvement identified by Astra may not be meaningful from another reviewer's perspective. Finding 1 may therefore measure agreement with Astra rather than reviewers' ability to recognize improvements.

  • The study lacks comparisons with "no meaningful change" and minor-improvement pairs, as well as repeated reviews of identical versions. Without these baselines, the reported rates of unchanged scores cannot be interpreted reliably. In fact, score increases greatly outnumber decreases (40.5% vs. 4.6% for GPT-5.5; 21.0% vs. 1.4% for Astra), suggesting that reviewers do detect improvements to some extent.

  • The trajectories include papers dating back several years, but the study does not report model knowledge cutoffs or test whether reviewers recognize the manuscripts. Removing author and publication identifiers does not guarantee anonymity. If models have previously encountered these papers, memorized assessments could explain why scores remain unchanged across versions.

Questions for authors
  • In what percentage of pairs does the revised-version review acknowledge that weaknesses identified in the earlier review have been resolved? How often do scores increase in these cases?

  • Among the 75 sampled major-improvement pairs, how many were independently judged as major improvements by human evaluators? How was domain expertise matched?

  • What are the score increase, unchanged, and decrease rates for no-change pairs, minor-improvement pairs, and repeated reviews of identical versions?

  • What are the reviewers' knowledge cutoffs? Can the models identify anonymized manuscripts, their venues, or acceptance status?

5 comments
AuthorsOpenReview ↗

Response to Reviewer vn6a

Response to Weakness 1 and Question 1

“This discrepancy raises a fundamental question: does it indicate reviewer failure or unreliable improvement labels? The current design cannot distinguish between them.”

“In what percentage of pairs does the revised-version review acknowledge that weaknesses identified in the earlier review have been resolved? How often do scores increase in these cases?”

Thank you for your valuable comment. Improvement annotation compares both versions directly, whereas independent reviewing scores one version at a time. For example, added ablations can substantially improve a manuscript while both versions remain within the same rating category.

We re-annotated 225 pairs using four additional models, with 75 pairs from each original improvement category. Each model received the complete manuscripts and the same rubric, without Astra’s labels or explanations.

Table 1. Cross-model consistency of improvement annotations. Overall agreement is calculated across 225 pairs; major-improvement retention is calculated among the 75 Astra-major pairs.

AnnotatorAgreement with AstraAstra-major pairs also judged as major improvements
GPT-5.6 Sol80.9% (182/225)98.7% (74/75)
Gemini 3.8 Flash86.2% (194/225)86.7% (65/75)
Claude Sonnet 581.3% (183/225)74.7% (56/75)
GLM-5.3 Flash84.9% (191/225)77.3% (58/75)

The additional models retained 74.7–98.7% of Astra’s major-improvement labels, supporting consistency beyond a single model’s judgment.

We also examined Astra’s scores for human-confirmed improvements and revisions explicitly acknowledged in its later reviews.

Table 2. Astra’s independent score responses across validation subsets. Percentages are calculated within each subset. In the final row, 42/50 denotes acknowledged revisions among 50 audited pairs; score-change percentages use the 42 acknowledged pairs.

Evaluation subsetNumber of pairsHigher score (%)Unchanged score (%)
Astra-labeled major improvements1,48421.0277.63
Human-confirmed improvement subset6036.763.3
Pairs with revisions explicitly acknowledged in Astra’s later review42/5031.069.0

DeepSeek-V4.1-Flash audited the later reviews with scores hidden and identified explicit acknowledgment in 42/50 pairs (84.0%). Among these 42 pairs, 13 received higher scores, 29 retained the same score, and none received lower scores. Unchanged scores therefore remain common when improvement is human-confirmed or explicitly acknowledged.

Response to Weakness 2 and Question 2

“The five human evaluators are coauthors, lack explicit domain-expertise matching, and evaluate versions side by side. They only confirm whether some improvement exists, counting even minor improvements as positive.”

“Among the 75 sampled major-improvement pairs, how many were independently judged as major improvements by human evaluators? How was domain expertise matched?”

Thank you for your valuable comment. The five coauthor evaluators each had reviewing experience at conferences represented in our dataset. They compared versions using common criteria, with sampling and assignment balanced across conferences and automatic improvement categories. We did not match each paper to a specialist in its precise subfield. We can also allow AI tools to clarify unfamiliar concepts and technical details, with evaluators retaining responsibility for their judgments.

Table 1 shows that four additional models retained 74.7–98.7% of Astra’s major-improvement labels, supporting the significance of these revisions beyond Astra’s perspective.

We further applied a stricter human-confirmation criterion requiring every assigned evaluator to judge the improvement as major.

Table 3. Score responses under broad and strict human-confirmation criteria. Both criteria apply to the 75 sampled Astra-major pairs. The strict criterion requires unanimous major-improvement judgments from the assigned evaluators. Ranges summarize results across AI reviewer models.

Human-confirmation criterionConfirmed pairs among the 75 sampled Astra-major pairsPairs used per AI reviewerHigher score (%)Unchanged score (%)
Major or minor improvement74/7559–6028.3–45.053.3–68.3
Major improvement only, agreed by every assigned evaluator49/7537–3828.9–44.752.6–71.1

The strict criterion retains 49/75 pairs, and unchanged scores remain the majority outcome for every AI reviewer (52.6–71.1%). Together with Table 1, this supports Finding 1 beyond Astra’s labels and the broader criterion accepting minor improvements.

Response to Weakness 3 and Question 3

“The study lacks comparisons with ‘no meaningful change’ and minor-improvement pairs, as well as repeated reviews of identical versions.”

“What are the score increase, unchanged, and decrease rates for no-change pairs, minor-improvement pairs, and repeated reviews of identical versions?”

Thank you for your valuable comment. We compared major improvements, minor improvements, no meaningful change, and identical-version repeats under matched model settings and the same neutral review protocol.

Table 4. Score-change distributions across revision categories and identical-version repeats. Each cell reports higher / unchanged / lower scores (%). Identical-version repeats compare independent reviews of exactly the same manuscript version.

ReviewerMajor improvement (%)Minor improvement (%)No meaningful change (%)Identical-version repeat (%)
GPT-5.6 Sol48 / 44 / 836 / 58 / 66 / 76 / 1822 / 72 / 6
GPT-5.546 / 52 / 226 / 66 / 86 / 84 / 1014 / 78 / 8
Gemini 3.8 Flash38 / 60 / 226 / 68 / 610 / 84 / 616 / 78 / 6
Claude Haiku 4.524 / 64 / 1212 / 78 / 1012 / 84 / 412 / 84 / 4
GLM-5.3 Flash52 / 48 / 030 / 60 / 108 / 80 / 1214 / 66 / 20

All five models assign higher scores more frequently to major improvements than to any comparison condition, supporting partial responsiveness to improvement. Identical-version repeats also produce score changes, providing a baseline for ordinary scoring variability. These comparisons place the unchanged-score rates in the context of both improvement magnitude and review variability.

Comments on the paper

2 comments

Dear reviewers,

Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.

Many thanks for your time and effort.

AuthorsOpenReview ↗

Gerneral response

We thank the reviewers for their time and feedback. We have strengthened the validation and clarified the scope and interpretation of our findings to address the main concerns raised in the discussion:

(1) We added re-annotation by four additional models, stricter human confirmation of major improvements, and an expanded human scoring reference to address concerns about label reliability and dependence on a single annotator (Reviewers vn6a, VYfg).

(2) We added minor-improvement, no-change, and identical-version controls to contextualize score changes, and probed publication-outcome recovery while clarifying the limits of inferences about prior model exposure (Reviewer vn6a).

(3) We added evaluations with previous reviews provided and audits of revision acknowledgments, addressing practical relevance while clarifying that unchanged scores alone do not establish a failure to recognize improvements (Reviewers vn6a, 6eMJ, VYfg).

(4) We added matched neutral-repeat controls across seven reviewers, finding additional loss of observed score gains under negative opinions and addressing concerns about scoring noise and regression to the mean (Reviewer VYfg).

Together, these additions strengthen our contribution to evaluating AI reviewers through real author revision trajectories. PaperTrajectory provides revision histories for 6,959 papers accepted at five AI conferences, with 14-dimensional change annotations and separate quality judgments. Our findings show that annotated improvements do not consistently translate into higher scores and that unsupported user opinions can undermine observed gains. This identifies complementary requirements for reliable AI review: responsiveness to substantive revisions and stability when user framing changes without new scientific evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.