- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AutoSupervision is a benchmark for Auto-Supervision that evaluates concern-level, evidence-grounded verification of scientific manuscript revisions. It is built from transparent peer-review records of Nature Communications and contains 8,790 concern-level instances. The authors evaluate proprietary and open-source LLMs, together with an agentic baseline and a supervised fine-tuned model. Current models perform reasonably well in understanding reviewer concerns, but are much weaker at deciding whether the concerns have actually been resolved.
Critical strengths
- The scientific revision process is meaningful and AutoSupervision asks whether the reviewer concern has actually been resolved and requires the judgment to be supported by evidence. This makes the task relevant to closed-loop AI-assisted scientific workflows.
- AutoSupervision evaluates multiple LLMs, an agentic inference pipeline, and task-specific supervised fine-tuning. Across these settings, verification remains consistently more difficult than concern characterization.
Critical weaknesses
- AutoSupervision relies heavily on LLM-assisted concern extraction and structured annotation, while the agreement study shows uneven agreement across different fields. Since these annotations are used as benchmark targets, human validation on a representative subset is needed to establish their reliability.
- This benchmark contains very few unsuccessful revisions. Of the 8,790 episode-level instances, 8,265 are resolved, while only 434 are partially resolved and 91 are unresolved. Although macro-F1 reduces the effect of label imbalance on the metric, the small number of unresolved cases makes it difficult to assess model performance on failed or incomplete revisions.
Questions for authors
- When reconstructing an earlier manuscript version, does GPT-5.5 use information from later review rounds? If so, how is its influence controlled?
- Were the benchmark annotations manually checked by human annotators? If yes, could you report the agreement on resolution labels and evidence blocks?
- Could you report performance separately on partially resolved and unresolved cases? How many of the 91 unresolved concerns are correctly identified?
Please take a look at the response!
Dear reviewers,
As the rebuttal period is about to end, please take a look at the authors' responses.
Best,
AC