acceptodds
Under review as a conference paper at ICLR 2027

RINI: Seeing the Prior Is Not Enough

Abstract

A research proposal can describe an established mechanism correctly while claiming to introduce it. We study whether providing the earlier paper corrects such contribution claims. Three controlled experiments compare proposals generated with a contribution-bearing prior and a same-topic control. Providing the prior yields no clear aggregate reduction in unsupported novelty. Human analysis of 175 interpretable exposed proposals finds that 137 recognize the prior's relevance, but 61 correctly attribute the established contribution. Of 71 proposed remaining distinctions, 37 are covered by the same prior. We introduce Research Idea Novelty Inspection (RINI), which audits contribution claims against evidence, checks the remaining distinction, and applies local revisions. Five human annotators evaluate 1,080 original-revision pairs across three methods. On the same 240 originals judged to require correction, successful repair is 11.7% for Self-Revision, 39.1% for Retrieve-and-Revise, and 72.2% for RINI, with research tasks weighted equally. The improvement over same-evidence direct revision is 33.0 percentage points. The revised proposals retain their research questions and technical methods. These results motivate explicit contribution attribution when using literature to generate and revise research proposals.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scientific novelty, scientific ideation, literature retrieval, novelty assessment, evidence-grounded review, research proposal evaluation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LHEeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • This paper studies novelty audit, providing prior work to prevent research proposals from overclaiming an established mechanism as a new contribution.
  • The proposed RINI compares contribution claims against prior evidence, checks whether any remaining distinction is also covered by the same prior, and performs local revisions while preserving the research question and technical method.
  • On the same 240 proposals judged to require revision, successful repair rates are 11.7% for Self-Revision, 39.1% for Retrieve-and-Revise, and 72.2% for RINI.
Critical strengths
  • The paper identifies an interesting failure mode of overclaiming novelty in literature-grounded research proposal generation.
  • It applies multiple revision methods to the same source proposals and evaluates revision success with human judgments.
  • The evaluation considers both correction of contribution claims and preservation of the research question and technical method.
Critical weaknesses
  • The repair evaluation only considers cases where the model is explicitly told that the prior work already contains the relevant contribution (false positive). An opposite error of false negative shouldbe measured.
  • As RINI adds several changes over direct revision, including structured analysis, local editing, and validation, it is unclear which component drives the improvement without careful ablations.
  • The main human evaluation uses one annotator per item, while judgments of whether the same original proposal requires revision vary substantially across evaluations. The reliability of the results is therefore insufficiently established.
  • Readability of figures can be improved (crowded and AI-like)
Questions for authors
  • Can RINI distinguish individual contributions when multiple prior papers establish different parts of a proposed contribution?
  • False negative rates of RINI?
  • How does RINI compare with a simple baseline given the same information but instructed to modify only the problematic spans, instead of complet rewrites?
  • Ablations of its individual components?
3 comments

Thank you for the review and the concrete questions. We answer them in order.

W1 / Q2. False negatives. We ran two settings on the same 360 originals with the same model. In the first, the true D is shown, but the prompt no longer says that D establishes X. In the second, a topical hard negative N, which does not contain X, replaces D.

RINI's audit missed D's coverage of X in 8.6% of proposals when the relation was not stated. It reported that N covers X in 10.0% when N replaced D. The final texts under the wrong prior:

Wrong prior N in place of DRRRINI
Credits N for X0.3%1.7%
Misreads N0.0%3.3%
Changes the X claim67.2%25.3%

An LLM read each output and recorded what it credits; these checks have not been validated against human labels. A changed X claim under N is not always an error, because X is in fact established elsewhere and a model may correct the claim on its own.

Q1. Multiple priors. We showed D and N together, in random order under neutral labels P1 and P2, for the 240 sources used in the success analysis. Neither method credited N for X. RR credited D and not N in 236 of 240 proposals; RINI did so in 143 of 240. The gap comes from RINI's edit gate: when the gate rejects the proposed edits, RINI returns the original text unchanged by design, which happened for 96 of 240 proposals. Where the edits were applied, RINI credited D and not N in 134 of 144. These are the same automated checks as above.

Q3. Span-only baseline. This is our V2: the same information and instructions, at most four local edits, and no audit. It has been run on the same 360 originals, and we will post its human labels in this thread as they come in (general response).

Q4. Component ablations. V2 removes the audit, including the remainder check. V1 keeps RINI's repair instructions in a full rewrite without the audit. Both have been run, and we will post their human labels in this thread as they come in (general response).

W3. One annotator per item, and varying original-need labels. You are right on both points. In the submission, the field "original needs revision" was recorded beside each revision. Each original therefore received three judgments, one per method context, and they disagree for 104 of 360 originals. The success endpoint uses the 240 originals judged as needing revision in all three contexts, so all methods share one denominator. The disagreement still points to a reliability problem. The re-annotation therefore collects new labels for 60 stored revisions from 30 originals. Each original appears twice, once with each revision, so we can report agreement with the original labels and how often the same annotator gives the same original-need judgment in both contexts. We will report both in this thread once the annotation is complete.

W4. Figures. We will redraw all figures with less text, larger labels and one consistent style.

AuthorsOpenReview ↗

Follow-up on Q3/Q4 and W3: span-only baseline, ablations, and annotation reliability

We have completed the promised V1/V2 human annotation and manual verification, as well as the independent human re-annotation. All repair comparisons below use the paper's fixed 240 common-flaw originals, with equal weighting across 30 research tasks; the V1/V2 methods were run on all 360 frozen originals before this matched analysis.

For Q3, V2 is the requested simple span-only baseline: it receives the same evidence and RINI repair instructions and can make at most four local edits, but has no Contribution Audit or Remainder Check. Its successful-repair rate is 52.7%, compared with 72.2% for RINI. For Q4, V1 removes the same audit stages while permitting a full-text revision; it reaches 69.7%. The original Retrieve-and-Revise baseline is 39.1%. Thus, constrained local editing alone falls 19.5 percentage points below RINI, whereas instruction-matched full revision comes within 2.5 points. These comparisons support the value of testing the full workflow, while also showing that instructions and editing mode are important; they cannot separately attribute the result to Contribution Audit versus Remainder Check.

For W3, an independent human annotator re-labeled the 60 stored RR/RINI revisions from 30 randomly sampled originals. Agreement with the original labels was 91.6%. We will report this reliability check while also acknowledging the original-need discrepancies across method contexts noted in our first reply. Keeping the same 240-original common-flaw subset ensures the main methods are compared on a fixed set of sources.

Clarification acknowledged, but lean to maintain score.

Reviewer_6PiXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a problem in LLM-generated research proposals: even when large language models (LLMs) are provided with evidence from relevant prior work, they may still misrepresent existing technical contributions as their own innovations. The authors conduct three controlled studies comparing proposals generated with access to prior work establishing a target contribution against those generated with access to topically relevant literature that does not contain that contribution. The results show that providing the relevant prior work does not lead to a clear overall reduction in unsupported novelty claims. A manual analysis of 175 interpretable proposals further reveals that, although 137 identify the prior work as relevant to the target contribution, only 61 correctly attribute the existing contribution, suggesting that recognizing literature relevance does not necessarily translate into correct contribution attribution.

Motivated by this observation, the authors propose RINI (Research Idea Novelty Inspection), a workflow comprising Contribution Audit, Remainder Check, and Local Repair. It aims to correct novelty claims based on literature evidence while preserving the original research question and technical method as much as possible. The authors evaluate three revision methods—Self-Revision, direct revision with the same evidence provided (Retrieve-and-Revise), and RINI—with five annotators. Among the 240 original proposals judged to require correction, the respective successful repair rates are 11.7%, 39.1%, and 72.2%. RINI therefore outperforms direct revision by 33.0 percentage points. Overall, through controlled experiments, manual behavioral analysis, and a revision evaluation, the paper investigates the gap between literature visibility and correct contribution attribution and demonstrates that RINI improves contribution correction in the evaluated settings.

Critical strengths
  1. Empirical evidence of the gap between recognizing relevant prior work and correctly attributing existing contributions. Through three controlled studies and a manual analysis, the paper finds that even when LLMs recognize the relevance of prior work to a target contribution, they do not necessarily attribute that contribution correctly. This distinction provides concrete empirical evidence for understanding novelty assessment failures in LLM-generated research proposals.
  2. RINI substantially outperforms direct revision based on the same literature evidence in the evaluated setting. Among the 240 original proposals judged to require correction, RINI achieves a successful repair rate of 72.2%, compared with 39.1% for Retrieve-and-Revise, an improvement of 33.0 percentage points. This result suggests that integrating contribution auditing, remainder checking, and local repair into a single workflow may be more effective than directly asking a model to revise a proposal based on literature evidence.
Critical weaknesses
  1. Insufficient overall significance and originality. The paper does not sufficiently establish the significance of the problem it addresses or the broader impact of its proposed solution. Although the authors identify a failure mode in LLM-generated research proposals, they do not clearly demonstrate why this problem warrants a distinct methodological approach or how addressing it would substantially advance existing methods for research novelty assessment and LLM-assisted scientific ideation. Overall, the significance of the problem, the originality of the proposed solution, and the practical impact of the findings remain insufficiently established, weakening the case for acceptance.
  2. Limited originality of the first claimed contribution, viewed through the paper’s own contribution-attribution framework. A central claim of this paper is that LLMs may fail to correctly attribute existing contributions to prior work, even when provided with relevant literature, and may instead present those contributions as novel. However, prior work has already examined the overlap between LLM-generated research ideas and existing studies, including the failure to adequately account for existing contributions. Viewed through the paper’s own framework of contribution attribution, the authors should distinguish what has already been established by prior work from what genuinely remains as a new contribution of this paper.
  3. Insufficient isolation of the factors responsible for RINI’s reported improvements. RINI combines Contribution Audit, Remainder Check, and Local Repair into a unified workflow. Although the evaluation reports a substantial improvement over Retrieve-and-Revise, the experimental design does not isolate the effects of these individual components. In particular, it remains unclear whether the gains arise from contribution auditing, remainder checking, the local repair strategy, or simply their combination in a structured workflow. Without ablation studies or more targeted controlled comparisons, the results provide limited evidence that RINI’s specific methodological design, rather than the general use of a structured revision process, is responsible for the observed improvements. This limits the strength of the paper’s claims regarding RINI’s methodological novelty and effectiveness.
Questions for authors
  1. Clarification on the source of RINI’s performance gains: RINI consists of three components: Contribution Audit, Remainder Check, and Local Repair. Could the authors provide additional experimental results that isolate the contribution of each component to RINI’s performance improvement over Retrieve-and-Revise? In particular, would the Local Repair strategy alone, without Contribution Audit and Remainder Check, achieve comparable results? Addressing this question would help clarify whether RINI represents a substantive methodological advance or primarily improves performance by combining existing techniques into a structured workflow.

  2. Clarification on cross-domain generalizability: The experiments appear to focus primarily on research ideas in computer science. Have the authors evaluated RINI in other research domains, such as biomedicine, physics, or the social sciences? Given that disciplines may differ in their research methodologies, forms of contribution, and criteria for assessing novelty, to what extent can the current findings be generalized beyond the domains evaluated? If cross-domain evaluations have not been conducted, how do the authors characterize the scope and generalizability of their approach?

2 comments

Thank you for the review. We address significance and originality first, then component isolation and scope.

W1–W2. Significance and originality of the first contribution. We agree that prior work has examined overlap between LLM-generated ideas and existing studies. Gupta & Pruthi (2025) found that experts could trace a substantial share of LLM-generated research documents to existing work that the documents did not credit. Beel et al. (2025) found that an automated novelty check rated established techniques as novel. These studies examine outputs after the fact, and they leave open whether supplying the relevant prior fixes the problem.

Our first contribution tests that question under controlled exposure: the same seed is generated with the decisive prior D or with a topical hard negative N. To our knowledge, three results are not established by prior work:

  1. Supplying D does not reliably reduce unsupported novelty claims.
  2. Recognizing D is not crediting it. In the manual analysis, 137 of 175 proposals identify D as relevant to the target contribution, but only 61 credit it correctly.
  3. A withdrawn novelty claim can move to another component that D already covers.

This matters for literature-grounded systems such as SciMON, ResearchAgent and The AI Scientist, which retrieve related papers to ground ideas or to check their novelty. Our results show that a visible prior is not enough; attribution has to be checked contribution by contribution. We will add this positioning, including Si et al. (2024), to the introduction and related work.

W3 / Q1. Which component drives RINI's gain? We agree this was not isolated. Two baselines have been run on the same originals with the same model, and we will post their human labels in this thread as they come in (general response):

  • V2 is Local Repair alone: at most four local edits under RINI's instructions, with no Contribution Audit and no Remainder Check. It answers your question directly.
  • V1 is direct revision with RINI's repair instructions but no audit. It separates instruction wording from the structured workflow.

RINI also builds on existing ideas; for example, RARR (Gao et al., 2023) revises text to fix attribution while preserving the rest. We will state RINI's contribution narrowly, as a contribution-level audit against a supplied prior combined with a deterministic edit gate, and match the claim to what the ablation supports.

Q2. Cross-domain generalizability. All experiments are in computer science: the corpus is arXiv cs.CL, cs.LG and cs.CV (2018–2023), and Study 3's priors are computer-science papers. We will state this scope in the abstract and limitations. Fields differ in how contributions are framed; in biomedicine, for example, a contribution is often a finding rather than a mechanism. We therefore do not claim the rates transfer. The protocol itself, controlled exposure plus contribution-level attribution, needs only field-specific seeds and priors.

AuthorsOpenReview ↗

Follow-up on W3/Q1: completed V1/V2 ablations

We have completed the two ablations promised in our previous response and manually checked the new labels. Both use the same model and 360 frozen original proposals as RINI; the primary comparison retains the same 240 common-flaw originals and equally weights the 30 research tasks. The successful-repair rates are 39.1% for Retrieve-and-Revise, 52.7% for V2, 69.7% for V1, and 72.2% for RINI.

V2 directly addresses your Local Repair alone question. It retains RINI's repair instructions and the limit of four local edits, but removes Contribution Audit and Remainder Check. Its 52.7% repair rate is 19.5 percentage points below full RINI. Thus, local repair under the same instructions does not by itself reproduce RINI's result.

V1 tests the role of the instructions. It uses RINI's repair instructions without either audit stage, while allowing a full-text revision. At 69.7%, it comes close to RINI's 72.2%, a difference of only 2.5 percentage points. This shows that explicit repair instructions account for a substantial share of the improvement over the original direct-revision baseline. We therefore do not claim that the audit alone explains the original gain, or that these joint removals identify the independent effect of each audit stage. We will narrow the methodological claim to the explicit contribution-level audit, remainder checking, and bounded edit procedure, and report both instruction-matched baselines alongside RINI.

Reviewer_eZZJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Language models asked to propose research ideas sometimes present a known method as new. The authors gave the model a short excerpt (about 180 words) of the paper that introduced the method. The proposals usually noticed it but often still did not credit it, and some moved their claim of novelty to another detail the same paper already covers. Their fix, RINI, has a model check each "this is new" claim against the excerpt and make at most four small edits. Volunteers, judging each revision once, found that RINI fixed the problem more often than a model simply told to rewrite with the same excerpt.

Critical strengths

The paper names a specific failure of literature-grounded idea generation: a model can see, and even cite, the paper that introduced its method and still claim the method. Separating recognition, credit and narrowing (sec. 4.1) is a distinction that novel/not-novel checkers do not make, and the finding that abandoned claims often move to a component the same paper covers is the most reusable result. The design isolates 'exposure' well, and the outcomes are comprehensively reported.

The macro-structure of the paper is quite clear: introduction is mostly easy to follow and the related work is logically divided into sections

Critical weaknesses

The headline effect cannot be separated from how it was measured. Each item was judged once and so there isn't any measure of agreement; various biases of LLM judging need to be accounted for including self-affinity, verbosity bias and so on. To my understanding, RINI is a framework of essentially two prompts, but the advantage cannot be directly isolated to this design, because it is given the correct paper and uses only one revision model, potentially with instructions rather close to the success rubric. A more rigorous analysis would include several ablations (such as providing the wrong paper or different revision methods).

Some aspects of data collection are not very clearly justified: in Sec. 3.1 what is the corpus and why are 376 candidates generated (not more or fewer)? How were they chosen? Similarly the introduction would benefit from some reference to motivating evidence or examples, as not all of the assertions made there are necessarily novel/original.

On presentation: The paper gets bogged down at times trying to coin new terminology, meanwhile forgetting to mention some important points that may not be obvious to all readers: i.e. that the proposals and revisions are written by LLMs. Some of the vocabulary makes the proposed method sound more general than it really is. The term "cite" is used frequently but it is not always clear when it refers to formal academic citation or simply mentioning documents and passages where concepts appear. Figure 1 is not particularly clear on this point and is rather text-heavy; it's also unclear whether the numerical results should be included in what is otherwise a schematic. Figure 2 inherits the same issues: the particular example is ambiguous because the "prior" is just a sentence of unspecified provenance. The Equation 1 seems to be more decorative than instructive. Perhaps a minor point but the tables are incorrectly formatted, with structure within cells, numbers erroneously centrally aligned, headers that are too wide and spurious precision in percentages, so they do not do a good job of supporting the central thesis.

No code was provided.

Questions for authors

Was any item labelled by more than one person, and if so, what was the agreement?

On what data were RINI's prompts, the validator and the rubric developed?

Where does the corpus come from and how were the corpus papers chosen from which we get 376 candidate "seeds"?

3 comments

Thank you for the detailed review, and for singling out the relocation finding. We take the measurement concerns as the central issue and answer them first.

Q1. Was any item labelled by more than one person? No. In the submission each of the 1,080 revisions received one label, so we could not report agreement. We are now collecting a second, independent label for 60 stored revisions, the RR and RINI outputs for 30 randomly sampled originals, from an annotator who did not take part in the original study. Comparing these with the original labels gives agreement. This annotation is under way; we will post Cohen's κ and the re-estimated comparison here as soon as it is complete.

Instructions close to the rubric. We agree this is a real risk. RINI's instructions and the rubric share one idea: an uncertain distinction framed as a testable possibility, rather than as established novelty, can be acceptable. RINI's prompts were frozen before any revision was labelled, and the rubric was finalized afterwards on the pilot. To test whether shared wording explains the effect, V1 gives the direct-revision baseline RINI's repair instructions without the audit; V1 has been run on the same 360 originals, and we will post its human labels in this thread as they come in (general response).

LLM-judging biases. Because the revisers are LLMs, an LLM judge would face exactly the self-affinity and verbosity biases you list. We therefore compare repair success on human labels only, and report automated checks only as labelled diagnostics. In the exposure studies, the diagnostic findings come from human annotation of the Study 3 proposals: 137 of 175 recognize D, but only 61 credit it, and 37 of 71 proposed remaining distinctions are covered by D. The aggregate exposure contrast uses an automated score; in Study 3 its judge (DeepSeek) comes from a different model family than the three generators (Kimi, GLM and Qwen). We will label that score as automated in the main text.

Only the correct paper is given. We tested a wrong paper. With a topical hard negative N in place of D (N does not contain X), RINI credited N for X in 1.7% of proposals (RR: 0.3%) and changed the X claim in 25.3% (RR: 67.2%). These are automated attribution checks, not yet validated by humans, and a change under N can also be a correct self-correction. Details are in our response to LHEe.

One revision model. We agree. A second revision model would need human labels that do not fit in this discussion period, so we will state the single-model limitation in the abstract and limitations.

Q2. On what data were the prompts, validator and rubric developed?

  • Prompts and edit gate. Two development versions preceded the frozen version used for all results. They were run on two proposals, both from one seed (VICReg); the retained records show parsing and validation failures in those runs, which the frozen parser and edit gate now handle. That seed is one of the 30 evaluation seeds, so we will also report all Study 3 results without it (12 of 360 sources).
  • Rubric. As stated in Appendix J, it was finalized after a 90-item pilot (30 revisions per method) labelled by a colleague outside the formal annotation. We did not keep a dated record of the pilot discussion. The re-annotation therefore uses an annotator who did not see the pilot, under a rubric frozen before any new label.

Q3. Where does the corpus come from, and how were the 376 candidates chosen?

  • Corpus. arXiv cs.CL, cs.LG and cs.CV papers from 2018–2023: 50,938 records, 50,932 after removing six duplicate titles.
  • Candidates. gpt-5.4-mini generated research seeds per domain, 47 in each of eight domains, with no paper text as input. The run targeted 360 seeds in eight equal domain cells; the number was not set by a power calculation. Deduplication (SPECTER2 similarity ≥ 0.85) left 315 seeds, and human review retained 314 (Appendix B.3). Appendix B.3.1 says each candidate was "attached to a paper in the corpus"; this is inaccurate, because papers were attached later, during literature curation. We will correct it.
  • Study 3 uses a separate set of 30 seeds. The authors chose each decisive prior D and hard negative N; a program located the evidence spans and an LLM checked them, without independent human vetting (Appendix B). We will state this selection procedure, and its limits, in the main text.

Presentation. The revised PDF will:

  • state in the abstract and introduction that all proposals and revisions are LLM-generated;
  • cut coined terms, use "credit" or "attribute" for ownership statements, and reserve "cite" for bibliographic citations;
  • redraw Figure 1 as a schematic without results, and replace Figure 2's example with a real case that names D and quotes its excerpt;
  • remove Equation 1;
  • reformat the tables: no structure inside cells, decimal-aligned numbers, one decimal place for percentages, narrower headers;
  • narrow the vocabulary so the method reads as specific to contribution attribution against a supplied prior;
  • support the introduction's claims with motivating evidence and citations (Gupta & Pruthi, 2025; Beel et al., 2025).
AuthorsOpenReview ↗

Follow-up: independent human re-annotation and instruction-matched ablations

We have completed the independent human re-annotation promised in our earlier response. A new annotator re-labeled 60 stored RR/RINI revisions from 30 randomly sampled originals, and agreement with the corresponding original labels was 91.6%. This supplies a direct reliability check for the original human evaluation; we will also retain the limitation that the full 1,080-item evaluation originally had one label per revision.

We have also completed and manually checked the additional V1/V2 annotations. Both baselines were run on the same 360 frozen originals with the same revision model. For the fixed set of 240 originals used in the paper's main comparison, with equal weighting across the 30 research tasks, successful-repair rates are 39.1% for Retrieve-and-Revise, 69.7% for V1, 52.7% for V2, and 72.2% for RINI. As defined in our earlier response, V1 applies RINI's repair instructions to a full-text revision without the audit, while V2 applies the same instructions in at most four local edits, also without the audit.

The result qualifies the original interpretation. V1 closes most of the gap between the original direct-revision baseline and RINI, showing that the repair instructions themselves matter. RINI is 2.5 percentage points above the instruction-matched V1 and 19.5 points above the local-edit V2. We do not treat the small RINI–V1 difference as statistically established without a paired uncertainty estimate, nor do these two ablations isolate Contribution Audit from Remainder Check. We will make these qualifications explicit and avoid attributing the entire original gain to the audit.

Comments on the paper

1 comment

We thank the reviewers for careful and constructive reviews. Three concerns are shared: whether the human repair labels are reliable (eZZJ, LHEe), which part of RINI matters (all three reviewers), and how the diagnostic findings relate to prior work (6PiX). This note summarizes what we are doing; the individual responses give details.

Label reliability (annotation in progress). In the submission, each revision received a single label. We are running an independent re-annotation by an annotator who did not take part in the original study.

  • 60 stored revisions: the RR and RINI outputs for 30 originals sampled at random from the 240-original comparison set. Comparing each new label with the original label of the same revision gives agreement (Cohen's κ per field), and the new labels give a paired re-estimate of the RR–RINI difference.
  • Each original appears twice, once with each revision, and the annotator judges each time whether the original needed revision. This shows how stable that judgment is across method contexts, the issue LHEe raised.

The rubric is the one in Appendix J, fixed before any new label was collected. Method names are hidden, and the two revisions of an original are placed at least six items apart.

Which part of RINI matters. Two baselines have been run on the same 360 originals with the same model.

  • V1 gives Retrieve-and-Revise RINI's repair instructions, but no audit. It tests whether wording close to the rubric explains the gain (eZZJ).
  • V2 makes at most four local edits under the same instructions, but without the audit. It is the span-only baseline (LHEe Q3) and "Local Repair alone" (6PiX Q1).

Repair success is compared on human labels only, so each baseline needs its own labels: 720 for the two. At about ten minutes per label under the Appendix J rubric, this is roughly 120 annotator-hours, two thirds of the 1,080 labels in the original study. The 240 originals in the success comparison are labelled first, in random order, so any completed part is a random sample of them. We will post results in this thread as they come in, and the final version will include all of them.

Positioning (6PiX). We will add a comparison with work that examines overlap between LLM-generated ideas and existing papers, and state what controlled exposure adds.

Other additions. Automated attribution checks for false negatives, a wrong prior and two priors (LHEe, eZZJ). Provenance of the corpus, seeds, prompts and rubric (eZZJ). A computer-science-only scope statement (6PiX). Presentation fixes (LHEe).

The re-annotation is under way. We will post its results in this thread as soon as it is complete, together with a revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.