acceptodds
Under review as a conference paper at ICLR 2027

AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification

Abstract

Recent advances in large language models (LLMs) have enabled AI systems to assist scientific research and peer review. However, an essential capability for reliable AI-assisted scientific workflows remains underexplored: verifying whether reviewer feedback leads to meaningful and evidence-supported manuscript improvements. We introduce AutoSpervision, which evaluates whether scientific manuscript revisions genuinely address reviewer concerns through grounded evidence. AutoSpervision leverages transparent peer-review records as a natural source of supervision, where reviewer comments specify scientific concerns, author responses describe claimed resolutions, and revised manuscripts provide evidence of changes. Given reviewer comments, author responses, and revised manuscripts, models must characterize reviewer concerns, determine whether concerns have been addressed, and identify supporting manuscript evidence. We construct 8,790 episode-level concerns from Nature Communications articles and corresponding review records. Then we conducted experiments on LLMs, an ablation study, and a case study. Our results show that while LLMs perform well in characterizing reviewer concerns, with GPT-5.5 achieving a score of 0.754, evidence-based verification remains the primary bottleneck, with the best-performing model reaching only 0.501.

Primary area:
datasets and benchmarks
Keywords:
LLMs, Auto Research, Peer Review, Benchmarking, Scientific revision verification, Evidence grounding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MxvEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AutoSupervision is a benchmark for Auto-Supervision that evaluates concern-level, evidence-grounded verification of scientific manuscript revisions. It is built from transparent peer-review records of Nature Communications and contains 8,790 concern-level instances. The authors evaluate proprietary and open-source LLMs, together with an agentic baseline and a supervised fine-tuned model. Current models perform reasonably well in understanding reviewer concerns, but are much weaker at deciding whether the concerns have actually been resolved.

Critical strengths
  1. The scientific revision process is meaningful and AutoSupervision asks whether the reviewer concern has actually been resolved and requires the judgment to be supported by evidence. This makes the task relevant to closed-loop AI-assisted scientific workflows.
  2. AutoSupervision evaluates multiple LLMs, an agentic inference pipeline, and task-specific supervised fine-tuning. Across these settings, verification remains consistently more difficult than concern characterization.
Critical weaknesses
  1. AutoSupervision relies heavily on LLM-assisted concern extraction and structured annotation, while the agreement study shows uneven agreement across different fields. Since these annotations are used as benchmark targets, human validation on a representative subset is needed to establish their reliability.
  2. This benchmark contains very few unsuccessful revisions. Of the 8,790 episode-level instances, 8,265 are resolved, while only 434 are partially resolved and 91 are unresolved. Although macro-F1 reduces the effect of label imbalance on the metric, the small number of unresolved cases makes it difficult to assess model performance on failed or incomplete revisions.
Questions for authors
  1. When reconstructing an earlier manuscript version, does GPT-5.5 use information from later review rounds? If so, how is its influence controlled?
  2. Were the benchmark annotations manually checked by human annotators? If yes, could you report the agreement on resolution labels and evidence blocks?
  3. Could you report performance separately on partially resolved and unresolved cases? How many of the 91 unresolved concerns are correctly identified?
Reviewer_hTEGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Autosupervision, a benchmark that asks whether a LLM model can characterize the reviewer concern, decide whether it was resolved, and ground that decision in manuscript evidence. The data come from published peer-review files of Nature Communications, annotated by GPT-5.5 into 8,790 concern instances across 847 revision episodes. Across fifteen systems, the main claim is that Characterization is largely handled while Verification is the bottleneck.

Critical strengths

S1. The problem is well chosen. Requiring manuscript evidence rather than trusting the response letter targets a real gap, and the split into Characterization, Verification, and Grounding is clean in my view.

S2. The ablation in Table 2 is useful: going from Concern-only to Full context shows that response letters alone do not suffice, which is the paper's core argument. Leakage handling (Section 4.1) also looks careful to me.

Critical weaknesses

W1. The revisions are not real version pairs. GPT-5.5 reconstructs the round-specific manuscript states from the published article plus the full review record, author responses included (Section 4.1), and no faithfulness check is reported, so I'm not sure the benchmark can capture its own motivating case of a response that overstates the revision. This also limits the novelty, since ARIES already links reviewer comments to real manuscript edits, and real version pairs exist in Kuznetsov et al. [1] and Ruan et al. [2].

W2. Gold labels come from GPT-5.5 with no human validation reported. The only check (Appendix B.2) is agreement with Claude Opus 4.8.

W3. The resolution labels are highly skewed. Table 5 has 8,265 resolved, 434 partially resolved, and only 91 unresolved instances, with no per-class scores or majority-class baseline, so I'm not sure the "verification is the bottleneck" headline reflects verification ability rather than performance on a few minority cases.

[1] Kuznetsov, Buchmann, Eichler, Gurevych, 2022, "Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review", Computational Linguistics

[2] Ruan, Kuznetsov, Gurevych, 2024, "Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision", ACL 2024

Questions for authors

Q1. Section 2 states that ARIES connects reviewer comments to manuscript edits. Could you clarify the novelty of AUTOSUPERVISION relative to ARIES beyond the resolution label?

Q2. Section 4.1 states that GPT-5.5 reconstructs manuscript states from a record that includes author responses. Could you analyze how closely the reconstructed changes match the original submitted versions on a human-audited sample?

Q3. Table 5 lists 91 unresolved labels out of 8,790. Could you report per-class resolution label F1 and the Verification score of a majority-class baseline?

Reviewer_Y56LOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper propsoed autosupervison, a benchmark designed for test whether AI agent can verify if a manuscript revision can address reviewer concerns using grounded evidences. The paper's goal is to close the feedback loop by auditing the actual revisions of papers. The benchmark is built by sourcing 8790 revisions from nature communications peer-review records. The experiments show that frontier LLMs are excellent and understanding a review but consistently fails at fixing the issues in the text.

Critical strengths
  • The source data from nature is of high quality, this can be a valuable asset for the community.
  • The paper covers a very comprehensive evaluation.
Critical weaknesses
  • The data is sourced from nature communications, therefore it would be heavily biased towards published and accepted papers, meaning that the dataset inherently overrepresents well addressed revisions.
  • Additionally these data are of a single domain, therefore has the risk of overfitting to one domain and that journals editorial style.
Questions for authors
  • The paper can be stronger if considering papers sourced from other sources like openreview.
  • I think there exists an related work that haven't been discussed in this paper: https://arxiv.org/abs/2603.03142

Comments on the paper

1 comment
Area_Chair_Yv5rOpenReview ↗

Please take a look at the response!

Dear reviewers,

As the rebuttal period is about to end, please take a look at the authors' responses.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.