- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a benchmark, RevTrack, that evaluates the ability of LLMs to judge whether the revision solves the reviewers' concerns. Specifically, the status of revised paper is classified into fixed, partially fixed, unresolved, or regressed. The authors build the benchmark based on publicly available discussions from NeurIPS and ICLR, and manually verified some of the labels for quality control. Experiments on several baselines, including traditional NLP classification, prompted LLM and their voting variants, show that existing methods perform poorly on the benchmark.
Critical strengths
The paper considers a practically relevant evaluation task, determining whether reviewers' concerns have been addressed in manuscript revisions, which could be useful for developing scientific assistants.
Critical weaknesses
-
Insufficient transparency and questionable reliability of benchmark construction
-
The benchmark construction is not sufficiently documented. It is unclear how reviewer concerns were extracted and aligned with author responses and how the labels were generated with LLM assistance. The relevant prompts and detailed procedures should be provided.
-
I doubt the reliability of the labels. The paper does not provide information about the annotators' backgrounds or the research domains of the selected papers. Line 189-192 reports "no version mapping is available; these statistics therefore describe the archived sample, not reliability of the current evaluation labels". While I could not understand what this claim means, it raises concerns about the credibility of the benchmark. Paper also acknowledges issues without resolving them, such as unresolved annotation-version discrepancies, potential data leakage, and missing experimental records. Just to list a few, L189/199/245...
-
-
I have concerns about the overall scientific rigor and the extent to which the experimental findings have been carefully verified.
-
Beyond the label-version issue above, several results appear not to have been checked before submission. E.g., the bootstrap interval for the constant majority predictor on ICLR24 is [0.184, 0.184] in Table 9 but [0.167, 0.198] in Table 13, and the authors state that the two protocols “have not been independently reconciled.”.
-
The paper repeatedly qualifies its results as "exploratory," "retrospective," or ""insufficient" to establish particular conclusions (e.g., L310/314...), making it difficult to identify which findings are actually supported by reliable evidence. This overly defensive writing style, together with how the authors being aware of these problems but have not resolved them, also raises concerns about whether the authors have adequately reviewed and verified the extent of AI-generated content.
-
Questions for authors
-
Could the authors explain the discrepancies between the archived labels and the current evaluation labels and how the reliability of the final labels is established?
-
Could the authors provide more details about the benchmark construction procedure (e.g., prompts, research domains of the selected papers, annotation protocols)?