acceptodds
Under review as a conference paper at ICLR 2027

RevTrack: Evaluating Whether Scientific Assistants Know When a Review Concern Has Been Fixed

Abstract

Scientific review assistance requires temporal judgment: after authors revise a paper, a useful assistant must decide whether an old reviewer concern is still valid, partially addressed, or obsolete. Existing review-assistance evaluations mostly score static critiques and therefore miss this revision-aware capability. We introduce RevTrack, an issue-level benchmark that aligns a review concern with author response and revision evidence, then asks whether the concern is fixed, partially fixed, unresolved, or regressed. On a hardened ICLR 2024 benchmark, a structured revision-evidence model reaches 0.704 macro-F1, compared with 0.389 for the strongest semantic encoder baseline. On ICLR 2025, a 21-row stress set and an 80-row standard-labeled active frontier show that cross-year transfer remains brittle: TF-IDF collapses to zero fixed-case F1, and the best expanded-frontier macro-F1 is 0.469. A user-confirmed 80-row NeurIPS 2024 active frontier adds a second venue stress axis: the best transferred semantic encoder reaches 0.348 macro-F1, while prompted LLMs and vote ensembles remain near majority on this frontier. These transfer slices are stress-oriented diagnostics, not venue-wide prevalence estimates. RevTrack reframes scientific review assistance as longitudinal evidence tracking and ships with auditable construction artifacts for blind validation, leakage control, label evidence, and claim readiness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
scientific review assistance, revision-status tracking, peer review, benchmark, evaluation methodology, human-AI interaction, large language models, evidence tracking, model calibration, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yj17OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a benchmark, RevTrack, that evaluates the ability of LLMs to judge whether the revision solves the reviewers' concerns. Specifically, the status of revised paper is classified into fixed, partially fixed, unresolved, or regressed. The authors build the benchmark based on publicly available discussions from NeurIPS and ICLR, and manually verified some of the labels for quality control. Experiments on several baselines, including traditional NLP classification, prompted LLM and their voting variants, show that existing methods perform poorly on the benchmark.

Critical strengths

The paper considers a practically relevant evaluation task, determining whether reviewers' concerns have been addressed in manuscript revisions, which could be useful for developing scientific assistants.

Critical weaknesses
  1. Insufficient transparency and questionable reliability of benchmark construction

    • The benchmark construction is not sufficiently documented. It is unclear how reviewer concerns were extracted and aligned with author responses and how the labels were generated with LLM assistance. The relevant prompts and detailed procedures should be provided.

    • I doubt the reliability of the labels. The paper does not provide information about the annotators' backgrounds or the research domains of the selected papers. Line 189-192 reports "no version mapping is available; these statistics therefore describe the archived sample, not reliability of the current evaluation labels". While I could not understand what this claim means, it raises concerns about the credibility of the benchmark. Paper also acknowledges issues without resolving them, such as unresolved annotation-version discrepancies, potential data leakage, and missing experimental records. Just to list a few, L189/199/245...

  2. I have concerns about the overall scientific rigor and the extent to which the experimental findings have been carefully verified.

    • Beyond the label-version issue above, several results appear not to have been checked before submission. E.g., the bootstrap interval for the constant majority predictor on ICLR24 is [0.184, 0.184] in Table 9 but [0.167, 0.198] in Table 13, and the authors state that the two protocols “have not been independently reconciled.”.

    • The paper repeatedly qualifies its results as "exploratory," "retrospective," or ""insufficient" to establish particular conclusions (e.g., L310/314...), making it difficult to identify which findings are actually supported by reliable evidence. This overly defensive writing style, together with how the authors being aware of these problems but have not resolved them, also raises concerns about whether the authors have adequately reviewed and verified the extent of AI-generated content.

Questions for authors
  1. Could the authors explain the discrepancies between the archived labels and the current evaluation labels and how the reliability of the final labels is established?

  2. Could the authors provide more details about the benchmark construction procedure (e.g., prompts, research domains of the selected papers, annotation protocols)?

Reviewer_PukGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RevTrack, a benchmark for determining whether reviewer concerns have been resolved after manuscript revisions. It defines four revision-status labels and evaluates semantic encoders, structured models, and LLMs across ICLR and NeurIPS datasets. The results highlight poor generalization and difficulties in distinguishing resolved from unresolved concerns.

Critical strengths
  1. Relevant problem setting. Revision-aware review assistance addresses a practical limitation of existing static review benchmarks.
  2. Useful evaluation perspective. The four-way status classification and cross-year evaluations provide insights into potential failures of scientific assistants.
Critical weaknesses
  1. Questionable benchmark validity. Labels primarily rely on author responses and revision summaries rather than independently verified manuscript changes. The manuscript audit reports only 60/84 agreement, raising concerns about whether the benchmark measures actual concern resolution or merely textual evidence of resolution.
  2. Insufficiently reliable annotations and experimental evidence. The independent annotation audit does not validate the current label version, with 59/96 archived labels differing from the evaluation manifest. Furthermore, the reported 0.704 macro-F1 structured-model result has unresolved feature-lineage concerns, while transfer evaluations rely on small, heavily imbalanced subsets. These issues substantially weaken the reliability of the main findings.
Questions for authors
  1. How can the authors ensure that the labels reflect actual manuscript revisions rather than potentially incomplete or misleading author responses?
  2. Can the authors reconcile the annotation-version discrepancies and provide independent agreement measurements on the current evaluation labels?
  3. Can the main results be reproduced under strictly leakage-controlled evaluation on a larger, independently annotated dataset with sufficient coverage of all four classes?
Reviewer_cMQ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Given a paper, reviewer concern, author response, and revision evidence ReTtrack predicts the status of the concern, ie if it is fixed, partially fixed, unresolved, or regressed. The system is evaluated on ICLR and NeurIPS papers, and finds that using LLM's for evaluation results in errors of promoting unresolved concerns to fixed or partially fixed.

Critical strengths
  1. RevTrack checks if an author responded to a concern.

  2. RevTrack checks if the author response provides evidence that a concern is addressed.

  3. RevTrack check if the paper implements the change.

This may be useful in the review process both for the rebuttal phase in which authors respond to reviewer concerns, and for manuscript decision and evaluation upon revise and resubmit

Critical weaknesses
  1. Overall, AI writes code and works on research papers. In this work AI evaluates AI changes, ie AI is generating the evidence and evaluating the evidence. The LLM may fabricate evidence too. A detailed consideration of the level of human oversight is missing.

  2. Missing is a discussion and application of best practices using LLM's, ie cross model evaluation, using models of different families for generation and evaluation.

Minor:

  1. The notation p,c,a,r is never used later on.

  2. The second part of the abstract is not well written, reads like a mechanical summary of numerical results.

Questions for authors

How does the RevTrack architecture compare with a pure LLM prompt baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.