- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces PaperTrajectory-2026, built from the arXiv version histories of 6,959 papers accepted at ICLR, ICML, ACL, CVPR and ECCV 2026. Versions more than 30 days apart are chained into trajectories and adjacent versions are paired (9,092 pairs, 9,066 with completed annotations). GPT-6 Astra reads both full manuscripts and labels the magnitude of change on 14 dimensions with before-and-after evidence, plus a separate overall quality judgment (major or minor improvement, no meaningful change, deterioration, uncertain). Five graduate-level evaluators checked 150 pairs: 98.6% of dimension descriptions were rated accurate, and 74 of 75 pairs labeled as major improvements were judged improved to some degree. Descriptive analyses show that major improvements tend to revise evidence and framing together, and that later revisions are narrower than earlier ones.
The main study reviews both versions of 1,484 major-improvement pairs independently with eight LLM reviewers, using venue-specific prompts and rating scales. Every reviewer keeps the same overall score on most pairs (55% to 78%), and only 21% to 41% of pairs receive a higher score (Fig. 3, Tab. 9). Agreement across reviewers on the direction of change is close to chance (Fleiss' kappa 0.063). Revision-dimension indicators are associated with score increases in reviewer-specific ways and give held-out AUCs between 0.507 and 0.641. When the user message expresses disappointment with the revised version, many of the observed neutral gains turn into ties or reversals, from 24.7% of gains for GPT-6 Astra to 98.5% for Claude Haiku 4.5 (Tab. 21).
Overall, the paper investigates whether AI reviewers can recognize improvements introduced through revisions to accepted papers. However, the study still lacks sufficient rigor in establishing ground truth, accounting for heterogeneity among AI judges (Only GPT-6 Astra as an annotator), and providing clearer and more generalizable empirical evidence to support its findings.
Critical strengths
One critical strength for the paper to be accepted is as below:
The paper raises a timely question with real revision data and a good experimental attempt. With AI assistance, submission numbers to any ai conferences now are growing explosively, and human expert reviewing cannot grow fast enough to keep up, so whether AI reviewers' scores reflect the real improvements authors make is a timely question. The dataset is built from revisions that authors actually made, and the dataset and the experimental design are a good attempt at studying this question.
Critical weaknesses
Two critical weakness that may lead to support rejection for the paper as below:
-
The evidence is not solid enough: the annotation relies on a single model, and there is no ground truth. All change descriptions and quality labels, including the "major improvement" label that defines the review sample, come from one model, GPT-6 Astra (Sec. 3.3). If that model is biased, every downstream result inherits the bias; the labels should be produced or checked with multiple LLMs or with human experts. There is also no ground truth for each paper against which the AI reviewers' score changes, or the improvement labels, can be matched or verified.
-
Finding 3 may have issues. The claim that a brief negative user opinion can erase or reverse a revised paper's score advantage (Sec. 7) does not feel solid to me. As one of the paper's three headline findings, it needs stronger support than the current analysis provides.
Questions for authors
Clarification requests for below three questions, if solved properly, that may change my assessment:
-
How stable are the labels if a subset is re-annotated by other LLMs or by human experts?
-
Is there a ground truth for each paper,since the papers are accepted to the conference already, u still cannot obtain the real scores for each paper? Or for example expert judgments of the same versions, against which the AI reviewers' score changes and the improvement labels could be verified?
-
What further evidence supports Finding 3?
7 comments
Response to Weakness 2 and Question 3
“The claim that a brief negative user opinion can erase or reverse a revised paper's score advantage (Sec. 7) does not feel solid to me.”
“What further evidence supports Finding 3?”
Thank you for your valuable comment. For each reviewer, we fixed pairs with an initially positive neutral score gap and re-evaluated the revised manuscripts under neutral and negative-opinion conditions. Manuscript content, review criteria, and earlier-version scores remained fixed. We measured loss of the advantage as a tie or strict reversal.
Table 3. Loss of revision score advantages under neutral repetition and negative opinions. Entries report tied / strictly reversed gaps (%). Gain-loss combines these outcomes; the final column reports the difference between conditions in percentage points, with 95% confidence intervals.
| Reviewer | Neutral repeat: tied / reversed (%) | Negative opinion: tied / reversed (%) | Difference in gain-loss rate (pp; 95% CI) |
|---|---|---|---|
| GPT-5.6 Sol | 5.9 / 0.0 | 58.1 / 0.0 | +52.2 [38.8, 66.1] |
| Gemini 3.8 Flash | 9.9 / 2.0 | 21.9 / 9.9 | +20.0 [8.1, 32.1] |
| Claude Sonnet 5 | 15.9 / 0.0 | 58.4 / 15.8 | +58.3 [44.3, 70.8] |
Negative opinions increase the gain-loss rate by 20.0–58.3 percentage points relative to neutral repetition, with all reported confidence intervals excluding zero. Sol exhibits erasure without strict reversal, while Gemini and Sonnet exhibit both outcomes. These controls support Finding 3: unsupported negative opinions destabilize revision score advantages beyond the losses observed under neutral repetition.
Response to the above 2 sections
Thanks to the authors for the new experiments. They respond directly to the three points I raised, and I have a few follow-up comments.
On the single annotator (Q1), the re-annotation is what I was hoping to see. With three balanced categories, 81–86% exact agreement across four other models is well above chance, and it makes me much less worried that the "major improvement" sample is an artifact of GPT-6 Astra. I would still like to know where Astra's major improvements go when another model disagrees. Claude Sonnet 5 keeps only 74.7% of them, and it matters whether the rest become minor improvements or no meaningful change. Please also say that only the overall label was re-annotated, not the 14 dimension descriptions, and note that GPT-5.6 Sol comes from the same provider as Astra. This table should go into the paper.
On ground truth (Q2), I appreciate the human study, although 30 pairs is small: a single pair moves each percentage by 3.3 points. Two things in Table 2 stood out to me. First, the human majority keeps the score unchanged on 60% of the pairs. If these pairs come from the major-improvement sample, then human reviewers also often leave the score where it was, and an unchanged AI score is not by itself evidence that the AI missed the improvement. I think Finding 1 and the title question should be framed with this in mind. Second, most AI reviewers agree with the human direction only 31–57% of the time. A reviewer that simply gave the same mix of higher, unchanged and lower scores as the humans would agree about 49% of the time by chance, so most reviewers are close to chance on individual pairs. Gemini 3.8 Flash has almost the same mix as the humans but agrees on only 31%. Astra is the clear exception at 83%, and since Astra also selects the sample, this deserves some discussion. Please also report how the 30 pairs were chosen, how often the three human reviewers agreed with each other, and the human judgments of improvement size, which the response mentions but does not show.
On Finding 3 (Q3), the neutral-repeat control is the right one, and it is the check the paper itself said was needed in Sec. 7.3. A gap of 20–58 points over repeated neutral reviews, with intervals excluding zero, convinces me that the negative opinion has a real effect beyond run-to-run noise and regression to the mean. I have two questions. Why only three of the eight reviewers? And why are the negative-opinion rates here much lower than in Table 21 of the paper? Sol goes from 80.9% to 58.1%, Sonnet from 94.2% to 74.2%, and Gemini 3.8 Flash from 60.1% to 31.8%. If this is a new sample or a fresh run, the single-run numbers in Fig. 4 and Table 21 seem to vary quite a bit, and the paper should report the new estimates, with sample sizes, next to the old ones. The opposing-opinion setting in Sec. 7.2 still has no matching control, so I would soften the claims there.
Overall, the new experiments go a long way toward my concerns, and the human reference and the neutral-repeat control make the paper more informative. I have raised my Informativeness score from 2 to 3. Because the human reference is still small and the new negative-opinion rates do not match the ones in the paper, I am keeping my overall rating for now.
Response to Reviewer VYfg
Response to Weakness 1 and Questions 1–2
Thank you for your valuable comment. We re-annotated 225 revision pairs using four additional models, with 75 pairs from each original improvement category. Each model received the complete manuscripts and the same rubric, without Astra’s labels or explanations.
Table 1. Cross-model consistency of improvement annotations. Overall agreement is calculated across 225 pairs. Category-specific retention rates use the 75 pairs originally assigned to each category by Astra.
Exact agreement with Astra is 80.9–86.2%, and major-improvement retention is 74.7–98.7%, supporting label consistency across model judges.
Conference acceptance is a paper-level outcome; our analysis requires judgments of specific manuscript versions. We sampled 30 revision pairs and had three human reviewers independently score both versions under the corresponding conference criteria, with version order, Astra labels, and AI scores hidden. Majority vote determined score-change direction. The same reviewers independently assessed improvement magnitude, providing a human reference for the labels.
Table 2. Human reference judgments and AI reviewers’ score-change directions. Score-change directions compare the revised and earlier versions. Agreement measures whether an AI reviewer’s direction matches the human majority judgment for the same pair.
Human majority judgments assign unchanged scores to 60.0% of pairs, while agreement with this reference varies across AI reviewers. These checks support label stability and provide a version-specific human reference for evaluating score changes beyond Astra’s judgments.