acceptodds
Under review as a conference paper at ICLR 2027

CONSTRUCTING A PROVENANCE-ATTRIBUTION BENCHMARK OVER THE SCIENTIFIC LITERATURE: A SUPPLY CENSUS AND A SHORTCUT GATE

Abstract

We set out to test whether a reported operating-regime boundary in provenance-constrained generation (attribution improves when a claim could plausibly come from several sources and collapses when it could not) transfers to the scientific literature. Building the benchmark produced a different result: across three successive constructions, a rule consuming no evidence could answer it. Gold was identifiable from question surface features in 335 of 335 items; the headline metric scored 0.873 in the treatment family against 0.250 in the control at zero capability, because its denominator was the manipulation; and publication year, reachable only through the provenance map under test, identified gold in 131 of 135 items, only in the arm the map is supplied to. A suite of 639 checks passed throughout. We contribute a shortcut gate that decides which such artifacts are fatal by conditioning each candidate rule on the information it consumes: question-only rules must not beat chance, evidence-reading rules may, with their maximum published as a ceiling, and provenance-reading rules must not beat the null of their own candidate set. Applied to our construction, the gate eliminated all 27 question-level rule–scope failures and left the evidence-level ceiling at 0.994 and 0.996, which we read as a diagnosis of this design rather than an impossibility. We also report a supply census and a decision pilot of 888 calls ($10.15) that failed its prespecified validity criteria; each triggered criterion was itself misspecified, so we did not proceed to the full experimental grid.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, benchmark construction, benchmark validity, LLM evaluation, benchmark shortcuts, dataset artifacts, provenance attribution, scientific literature

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_7CJDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates construction artifacts in a provenance-attribution benchmark built from biomedical literature. The authors identify several shortcuts that can undermine the intended evaluation, including question-only answer leakage, biased evaluation baselines, and metadata-based shortcuts. They propose a three-tier Shortcut Gate to systematically detect these issues and report a supply census and an unsuccessful pilot study. The main contribution is a methodological framework for auditing benchmark validity rather than a completed attribution benchmark.

Critical strengths
  • Important and well-supported observations. The paper identifies several non-trivial construction artifacts with clear quantitative evidence. These findings highlight practical issues that can substantially affect the validity of attribution benchmarks.

  • Systematic auditing framework. The proposed Shortcut Gate offers a structured way to distinguish legitimate evidence use from potentially problematic shortcuts. I also appreciate the authors' transparency in reporting unsuccessful experiments and remaining limitations.

Critical weaknesses
  • Limited validation and methodological novelty. The proposed framework is mainly evaluated on the authors' own benchmark. Since several components build on existing auditing techniques, it remains unclear how much additional value the framework provides in more general settings.

  • Unclear construct validity. The identification of original sources relies on assumptions such as numerical uniqueness and citation absence, which do not necessarily establish true provenance. Moreover, the distinction between metadata leakage and legitimate provenance information could be better justified.

Questions for authors
  • Have the authors manually verified a sample of gold provenance labels to assess the reliability of the current construction criteria?

  • How does the proposed framework distinguish harmful metadata shortcuts from legitimate provenance evidence, such as publication chronology?

  • Can the authors further clarify what advantages the proposed build-level gate offers over existing shortcut auditing approaches, beyond the current case study?

2 comments

(Part 1/2) Thank you for the review. Your questions led to three changes: a catalogue-matched item filter on the same pool (Appendix E.2), which shows that build-level blocking is not needed to reach a passing subset and measures what the tier distinction changes; a stratified gold-label audit of 100 items (Appendix K); and an explicit tier criterion, applied to chronology (Section 3.3). The stated contribution is narrowed accordingly, and a pre-registered one-model evaluation is added and reported as found; its result is inconclusive (Section 6.5).

R2.1 (validation and novelty).\textbf{R2.1 (validation and novelty).} Agreed: all evidence comes from one construction on one corpus, and we claim no transfer (Section 7.2). What we could test is how much the framework adds over the techniques it builds on, so the revision runs a catalogue-matched item filter on the same pool (Appendix E.2): same 63-rule catalogue, same per-rule nulls, evidence reading permitted, items dropped only where a crossing rule implicates them. It also reaches a passing subset\textbf{It also reaches a passing subset} (119 items in-sample; 165 to 183 cross-fitted, 11 of 11 seeds passing), so build-level blocking is not needed; we withdraw that claim. The tier distinction changes the outcome:\textbf{The tier distinction changes the outcome:} with evidence reading treated as fatal, the filter removes all 348 items and a flat gate fails on every set. The filter’s outcome depends on its implication criterion\textbf{The filter's outcome depends on its implication criterion} (unique hits: fails at 310 items; any hit: empty); a build-level verdict does not. The contribution is narrowed to that combination (Sections 1, 2, 4.1; Table 11). We also registered, before calling the model, a one-model comparison of the 179 items the passing subset keeps against the 169 it removes (Section 6.5, Appendix L). Its result is inconclusive (D = −0.061, 95% interval [−0.122, 0.001], margin 0.10), and we draw no conclusion from it: the point estimate has the sign a shortcut would produce, but the interval includes zero, so the sign is not evidence of shortcut use; the two sets also differ in composition, with an effect of unestablished direction. The map-channel difference-in-differences is null. The evaluation does not establish that gate-passing items resist shortcuts better or worse.

R2.2 (construct validity).\textbf{R2.2 (construct validity).} We agree that numerical uniqueness and citation absence are operational criteria and do not establish true provenance; the revision measures how often they fail. A stratified audit of 100 items (Appendix K; method in R2.3) finds, design-weighted over the shipped build, 7.6% [3.3, 14.7] of gold values wrong and 17.4% [10.6, 26.3] reported by the gold document from another source, which numerical uniqueness cannot see; Section 3.2 (condition 5) and Section 4.1 (“What a verdict certifies”) now say so. Removing the flagged items changes neither gate verdict and moves the text-only ceilings by at most 0.005. On metadata versus provenance, the criterion is now explicit (Section 3.3, Table 2): a cue's tier is set by the information each arm supplies, not by whether it would be legitimate evidence in the world. The provenance map is the manipulation under test, so a cue reachable only through it (T2) would produce the very map benefit the design is meant to detect, and a T2 rule beating its null fails the build. A cue in the passage text, available to both arms, is evidence reading (T1): it may beat its null, and its maximum is published as a ceiling. R2.4 applies this to chronology.

(Part 2/2) R2.3 (gold-label verification).\textbf{R2.3 (gold-label verification).} An author read the labels of only 8 of the 100 audited items, shown as the suggested answer (not blind); 2 were first disputed and kept after an explanation, one of them a defective item already excluded. The revision states this as a limitation. We ran a stratified audit with two language-model annotators (Appendix K): Sampling:\textbf{Sampling:} 100 of 348 items, fixed seed, design hashed before the draw; 12 strata (179-item subset or not ×\times family ×\times gold-passage section), 50 items per subset; rates design-weighted with Korn–Graubard 95% intervals. Labels:\textbf{Labels:} numeric extraction (exact / minor format / wrong) and origination (own finding / another source / cannot tell); an item is excluded when its value is wrong or from another source. Annotators:\textbf{Annotators:} claude-opus-5-5 and gpt-6.1-sol (run through Codex), both at effort xhigh and from different vendors, the second blinded to the design fields and to the first's labels. The 53 disagreements were adjudicated by claude-opus-5-5, the first annotator's model, so no label carries an independent check. Agreement\textbf{Agreement} on the judgements that enter a rate: κ\kappa = 0.85 (value wrong) and 0.95 (another source). Shipped build:\textbf{Shipped build:} 7.6% [3.3, 14.7] wrong, 17.4% [10.6, 26.3] from another source. Removing them changes neither gate verdict and moves the ceilings by at most 0.005. With 100 of 348 items audited this is a partial correction, and the gate certifies no label.

R2.4 (shortcuts vs legitimate evidence).\textbf{R2.4 (shortcuts vs legitimate evidence).} By the information each arm supplies, not by whether the cue would be legitimate evidence in the world. Rules reading only the question (T0) or the provenance map and document metadata (T2) fail the build if they exceed the null of their own candidate set at z > 3; rules reading the passage text (T1) may, and their maximum is published as a ceiling (Section 4.1, Table 2). Chronology shows why. When publication year appeared only on the document cards reached through the map, the older document's passage was gold in 131 of 135 items, because a review cites older work. Chronology is real provenance evidence, but there it existed in one arm only and would have produced exactly the map benefit under test, so it was T2. On the shipped build year is not rendered: the chronology rules abstain on every item and are reported as not exercisable, not as passing. A build that wants chronology as evidence must render it in the passage text of both arms, where the text-only ceiling counts it (Section 3.3).

R2.5 (advantages of the gate).\textbf{R2.5 (advantages of the gate).} Fewer than the submission claimed. On our pool a catalogue-matched item filter also reaches a passing subset of comparable size (119 in-sample, 165 to 183 cross-fitted, against 179), so build-level blocking is not needed and we withdraw that claim (Appendix E.2). What the comparison supports: The tier distinction changes what ships:\textbf{The tier distinction changes what ships:} treating evidence reading as fatal, as comparable procedures that count any solvability as a defect do, empties the pool under the matched filter, and a flat gate fails on every set. A verdict on the build as shipped\textbf{A verdict on the build as shipped}, with its mechanism: the 348 items fail on two named metadata rules. No dependence on an implication criterion\textbf{No dependence on an implication criterion}, on which the item filter's outcome turns. Beyond this case study we claim no advantage; the evidence is one construction on one corpus (Section 7.2). Table 11 compares the method with published audits and item filters on simultaneity, restricted access, null calibration and blocking granularity.

Reviewer_DzKBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper, which could only be understood in the context of a particular experimentatl paradigm or tradition, which is not described, examines cases of straightforward rules being able to reproduce gold-standard labels.

Critical strengths

If I had enough context to understand what is being evaluated, the suggested methodological approaches might be valuable to know about.

Critical weaknesses

The paper would be useful as an internal report on a subtask in a research project, but cannot be read as an independent paper. This is immediately clear on reading the abstract, and this view does not change on reading it in its entirety,

Questions for authors

Was this submitted by accident, instead of the paper to which its results were intended to contribute?

1 comment

Thank you for the review. We agree that the submitted abstract and introduction assumed context the paper did not supply: they put the hypothesis we set out to test before the task, and used terms such as the provenance map without defining them. The revision rewrites both, so that the task, the two item families, the contribution and what we do not claim come first. We do not agree that the paper can only be read as an internal report on a subtask: its contribution (a shortcut gate, a supply census, and a negative result with its diagnosis) was listed in the submitted introduction and reported in Sections 4 to 6 (R1.2).

R1.1 (missing context).\textbf{R1.1 (missing context).} Agreed for the abstract and introduction, and the revision supplies that context in the paper itself. The abstract and introduction now open with the task: a provenance-attribution benchmark asks which of several candidate passages a stated quantity came from, and the rules that assemble its items can leave the answer in the question's wording, a metadata field or the mix of surviving items, so that a score reads as attribution when it measures construction. They define the two item families (a treatment item's quantity comes from one of eight cited studies, a control item's from one source) and what the design was built to measure, state the contribution, and add a paragraph “What we do not claim”. The body of the submission defined the task and the two families (Section 3.1), and Section 4.1 and Table 2 defined when a rule reproducing gold labels is fatal (it reads only the question or provenance metadata) and when it is legitimate (it reads the evidence text, and its maximum is published as a ceiling). The revision also defines the provenance map in Section 1 and the pilot's three prompt variants at the start of Section 6.

R1.2 (internal report).\textbf{R1.2 (internal report).} We agree that the submitted abstract and introduction put the study we set out to run, and context the paper did not give, before this paper's task and contribution. We do not agree that it can be read only as an internal report on a subtask. The submitted introduction listed the contribution: a shortcut gate, a supply census, and a negative result with its diagnosis. After Section 3 defined the task and the two families, Sections 4 to 6 reported the gate and what it found, the ceiling and its diagnosis, the census, and the pilot. The revision opens with the task and then states the contribution, a validity audit of one provenance-attribution benchmark: a shortcut test\textbf{shortcut test} conditioned on information tier, in which every rule of an enumerated, model-free catalogue of 63 is scored against the null of the information it consumes; its verdict on our build\textbf{verdict on our build}: the shipped 348 items fail on two metadata rules, and a 179-item subset passes; a stratified audit of the gold labels\textbf{audit of the gold labels} of 100 items (new in the revision); a census\textbf{census} of what the construction rules retain from the corpus. Every result the revision relies on is reported in it. The design hypothesis is stated in Section 1, the pilot is read against it (Section 6.4), and Section 7.2 states that the contrast the design was built for cannot be measured on this build, because the text alone identifies gold (text-only ceiling 0.994 and 0.996).

R1.3 (intended submission).\textbf{R1.3 (intended submission).} No. This is the paper we intended to submit. Its contribution is the benchmark-validity audit itself, as the submitted introduction stated (see our response to R1.2). The revision adds the gold-label audit and replaces the motivation with the task and the contribution.

Comments on the paper

4 comments

We thank the reviewers. The revision answers three questions with new analyses.

1. Does the paper stand alone?\textbf{1. Does the paper stand alone?} The abstract and introduction now define the task, the item families, the contribution and what we do not claim (Section 1).

2. What does build-level gating add?\textbf{2. What does build-level gating add?} A catalogue-matched item filter also reaches a passing subset, so we withdraw the claim that build-level blocking is needed; the tier distinction is what changes the outcome (Appendix E.2).

3. Are the gold labels right?\textbf{3. Are the gold labels right?} A stratified audit of 100 items by two language-model annotators finds 7.6% [3.3, 14.7] of gold values wrong and 17.4% [10.6, 26.3] reported from another source; removing them changes neither gate verdict, and an author has read the labels of only 8 of the 100 items, not blind (Appendix K).

4. One-model evaluation.\textbf{4. One-model evaluation.} A pre-registered one-model comparison of the 179 items the passing subset keeps against the 169 it removes is inconclusive, and we draw no conclusion from it (Section 6.5, Appendix L).

5. Table 9 counts.\textbf{5. Table 9 counts.} Four counts in Table 9 (Appendix D) came from a one-off check whose code was not kept; a released script now computes them. Two do not reproduce, and the other two match the script under looser definitions. The largest change: 98 (23.2%) rather than 34 (8.1%) answers have a negation or contrast word near the gold numeral. The table's conclusion, that no contrast is read off the strict path, holds even if all 98 were wrong (details under R3.2).

6. Supplementary package.\textbf{6. Supplementary package.} The revised supplementary package adds the code for the new audit, evaluation and item filter with the stored readings, and restores three prompt blocks and a probe_kind string that the submitted package had shortened.

(Part 1/2) Thank you for the review. We ran the matched comparison you proposed: the catalogue-matched item filter also reaches a passing subset, so we withdraw the claim that build-level blocking is needed (Appendix E.2, Table 10). We also ran a stratified audit of extraction and origination that includes the repaired subset (Appendix K); it is not independent in the sense you ask, and we say so.

R3.1 (build-level gating vs item filtering).\textbf{R3.1 (build-level gating vs item filtering).} Agreed, and the comparison changes our conclusion. Policy M (Appendix E.2) runs the full gate (63 rules, three scopes, T1 permitted), selects every T0/T2 rule–scope pair crossing z > 3 against its per-item null, drops the items those pairs implicate, iterates to a fixed point and re-scores every output set with fresh nulls. It was fixed in a design note before the run, makes no model call, and is also cross-fitted across random halves. M-E\textbf{M-E} (implicated: the rule's set contains gold and is smaller than the item's conditioning set; primary) passes: 119 items in-sample, 165 cross-fitted (163 inside the repaired 179), 165 to 183 over 11 seeds, all passing. M-U\textbf{M-U} (unique hit, Table 10's criterion) fails at 310 / 315: fewest passages persists (z = 11.776) but implicates no remaining item. M-A\textbf{M-A} (any hit) and M-E with T1 fatal\textbf{M-E with T1 fatal} remove all 348; a flat gate with T1 fatal fails on every set, the 179 included. So build-level blocking is not needed to reach a passing set; we withdraw that claim and the Appendix E.1 argument, which excludes only the any-hit union. Supply is comparable, not an advantage. With catalogue, coverage and calibration fixed, granularity does not decide whether a passing set exists; the treatment of T1 does.

R3.2 (label validity).\textbf{R3.2 (label validity).} Agreed: passing the gate and a high text-only score do not establish correctly labelled origination. The audit (Appendix K; design in R3.4) treats the repaired subset as its own stratum: 10.8% [3.8, 22.8] of its gold values are wrong and 10.8% [3.8, 22.8] come from another source, against 4.2% [0.6, 14.0] and 24.5% [13.4, 38.7] over the other 169 items (shipped build: 7.6% [3.3, 14.7] and 17.4% [10.6, 26.3]). Post hoc, repaired minus other 169 is 6.6 points [−2.4, 15.7] for wrong values and −13.7 [−24.7, −2.7] for another source (not registered, not corrected for multiplicity). Removing the flagged items leaves the shortcut counts unchanged and moves the ceilings by at most 0.005 (R3.4), but the labels are not certified, and Sections 3.2, 4.1 and 7.2 now say so. Three corrections: the Appendix D sample of 30 was read individually by a language-model reader, not by hand, as were the 27 conflict candidates and the 41 excluded erratum pairs of Section 5; the code review that found construction defects (Section 3.4) was by a second language model, not independent; and four Table 9 counts came from a one-off check whose code was not kept. Two do not reproduce: a released script gives 98 and 322 where the submission had 34 and 388; the largest change is 98 (23.2%) rather than 34 (8.1%) answers with a negation or contrast marker near the gold numeral. The other two (422, 353) match the script under looser definitions (gold value as a substring; citation on the secondary-path record); the revision reports them under stricter ones (420, 349). Even if all 98 were wrong, 322 rejected answers carry the gold value with no marker nearby, so the conclusion that no contrast is read off the strict path holds; the flag is a wide net (seven of the seeded 30 fall in it, and each gives the gold value). The revision states all three.

R3.3 (matched comparison).\textbf{R3.3 (matched comparison).} Yes (R3.1, Appendix E.2): under matched conditions build-level gating shows no advantage in reaching a passing set (119 in-sample, 165 to 183 cross-fitted, against 179), and we withdraw the necessity claim. Three narrower claims remain: The tier distinction\textbf{The tier distinction} decides whether a passing set exists (R3.1). A build-level verdict\textbf{A build-level verdict} reports the build as shipped (four pairs from two rules: fewest passages, z = 19.321 pooled; retraction status, z = 3.479 control) and needs no implication criterion; the filter passes only under the excess criterion. A pass is a property of a set:\textbf{A pass is a property of a set:} the one-pass M-E set (167 items) fails on t2_not_retracted (z = 3.577), and the 297 items the single-probe T0 filter keeps carry two crossing T0 rules (z up to 3.575), though none crosses on the 348. Every filtered set must be re-scored in full, as Policy M does. Section 2, Section 4.1 and Table 11 now state the contribution without build granularity as necessary.

(Part 2/2) R3.4 (audit independence).\textbf{R3.4 (audit independence).} Not independent in the full sense: two language-model annotators, adjudicated by the first annotator's model, with the labels of only 8 of the 100 items read by an author, not blind, which we state as a limitation (Section 7.2). It covers both relations and the repaired subset (Appendix K). Sampling:\textbf{Sampling:} 100 of 348 items, seed 20261110, design hashed before the draw; 12 strata (repaired 179 or other 169 ×\times family ×\times gold-passage section), 50 per subset, background sections weighted 3; inverse-inclusion weighting, Korn–Graubard 95% intervals. Criteria:\textbf{Criteria:} extraction E exact / minor format / wrong; origination O own finding / another source / cannot tell; excluded if E is wrong or O another source. Two post-draw clarifications are disclosed and define strict and lenient variants. Adjudication:\textbf{Adjudication:} annotators claude-opus-5-5 and gpt-6.1-sol, both at xhigh (the second via Codex, blinded to design fields and to annotator 1's labels); 53 disagreements adjudicated by claude-opus-5-5 (46 for annotator 1, 7 for annotator 2), each with a text-checked quote; κ\kappa = 0.85 (E wrong) and 0.95 (O another source). Errors:\textbf{Errors:} shipped build 7.6% [3.3, 14.7] wrong, 17.4% [10.6, 26.3] another source; repaired subset 10.8% [3.8, 22.8] each. Effect\textbf{Effect}, under all seven label variants: the shipped build fails on the same four pairs (adjudicated z = 18.403 / 16.855 / 7.387 and 3.274); the repaired subset passes with membership fixed (163) or re-derived (162), largest z 2.53; ceilings move by at most 0.005. So the gate and ceiling results do not depend on the flagged labels, but the labels carry errors at these rates, which the gate does not certify and the audit corrects only partially.

Dear reviewers, The authors' rebuttal has been posted. Can you take a look at the authors' responses to see whether they addressed your concerns? Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.