- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
- This paper studies novelty audit, providing prior work to prevent research proposals from overclaiming an established mechanism as a new contribution.
- The proposed RINI compares contribution claims against prior evidence, checks whether any remaining distinction is also covered by the same prior, and performs local revisions while preserving the research question and technical method.
- On the same 240 proposals judged to require revision, successful repair rates are 11.7% for Self-Revision, 39.1% for Retrieve-and-Revise, and 72.2% for RINI.
Critical strengths
- The paper identifies an interesting failure mode of overclaiming novelty in literature-grounded research proposal generation.
- It applies multiple revision methods to the same source proposals and evaluates revision success with human judgments.
- The evaluation considers both correction of contribution claims and preservation of the research question and technical method.
Critical weaknesses
- The repair evaluation only considers cases where the model is explicitly told that the prior work already contains the relevant contribution (false positive). An opposite error of false negative shouldbe measured.
- As RINI adds several changes over direct revision, including structured analysis, local editing, and validation, it is unclear which component drives the improvement without careful ablations.
- The main human evaluation uses one annotator per item, while judgments of whether the same original proposal requires revision vary substantially across evaluations. The reliability of the results is therefore insufficiently established.
- Readability of figures can be improved (crowded and AI-like)
Questions for authors
- Can RINI distinguish individual contributions when multiple prior papers establish different parts of a proposed contribution?
- False negative rates of RINI?
- How does RINI compare with a simple baseline given the same information but instructed to modify only the problematic spans, instead of complet rewrites?
- Ablations of its individual components?
3 comments
Follow-up on Q3/Q4 and W3: span-only baseline, ablations, and annotation reliability
We have completed the promised V1/V2 human annotation and manual verification, as well as the independent human re-annotation. All repair comparisons below use the paper's fixed 240 common-flaw originals, with equal weighting across 30 research tasks; the V1/V2 methods were run on all 360 frozen originals before this matched analysis.
For Q3, V2 is the requested simple span-only baseline: it receives the same evidence and RINI repair instructions and can make at most four local edits, but has no Contribution Audit or Remainder Check. Its successful-repair rate is 52.7%, compared with 72.2% for RINI. For Q4, V1 removes the same audit stages while permitting a full-text revision; it reaches 69.7%. The original Retrieve-and-Revise baseline is 39.1%. Thus, constrained local editing alone falls 19.5 percentage points below RINI, whereas instruction-matched full revision comes within 2.5 points. These comparisons support the value of testing the full workflow, while also showing that instructions and editing mode are important; they cannot separately attribute the result to Contribution Audit versus Remainder Check.
For W3, an independent human annotator re-labeled the 60 stored RR/RINI revisions from 30 randomly sampled originals. Agreement with the original labels was 91.6%. We will report this reliability check while also acknowledging the original-need discrepancies across method contexts noted in our first reply. Keeping the same 240-original common-flaw subset ensures the main methods are compared on a fixed set of sources.
Clarification acknowledged, but lean to maintain score.
Thank you for the review and the concrete questions. We answer them in order.
W1 / Q2. False negatives. We ran two settings on the same 360 originals with the same model. In the first, the true D is shown, but the prompt no longer says that D establishes X. In the second, a topical hard negative N, which does not contain X, replaces D.
RINI's audit missed D's coverage of X in 8.6% of proposals when the relation was not stated. It reported that N covers X in 10.0% when N replaced D. The final texts under the wrong prior:
An LLM read each output and recorded what it credits; these checks have not been validated against human labels. A changed X claim under N is not always an error, because X is in fact established elsewhere and a model may correct the claim on its own.
Q1. Multiple priors. We showed D and N together, in random order under neutral labels P1 and P2, for the 240 sources used in the success analysis. Neither method credited N for X. RR credited D and not N in 236 of 240 proposals; RINI did so in 143 of 240. The gap comes from RINI's edit gate: when the gate rejects the proposed edits, RINI returns the original text unchanged by design, which happened for 96 of 240 proposals. Where the edits were applied, RINI credited D and not N in 134 of 144. These are the same automated checks as above.
Q3. Span-only baseline. This is our V2: the same information and instructions, at most four local edits, and no audit. It has been run on the same 360 originals, and we will post its human labels in this thread as they come in (general response).
Q4. Component ablations. V2 removes the audit, including the remainder check. V1 keeps RINI's repair instructions in a full rewrite without the audit. Both have been run, and we will post their human labels in this thread as they come in (general response).
W3. One annotator per item, and varying original-need labels. You are right on both points. In the submission, the field "original needs revision" was recorded beside each revision. Each original therefore received three judgments, one per method context, and they disagree for 104 of 360 originals. The success endpoint uses the 240 originals judged as needing revision in all three contexts, so all methods share one denominator. The disagreement still points to a reliability problem. The re-annotation therefore collects new labels for 60 stored revisions from 30 originals. Each original appears twice, once with each revision, so we can report agreement with the original labels and how often the same annotator gives the same original-need judgment in both contexts. We will report both in this thread once the annotation is complete.
W4. Figures. We will redraw all figures with less text, larger labels and one consistent style.