- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates construction artifacts in a provenance-attribution benchmark built from biomedical literature. The authors identify several shortcuts that can undermine the intended evaluation, including question-only answer leakage, biased evaluation baselines, and metadata-based shortcuts. They propose a three-tier Shortcut Gate to systematically detect these issues and report a supply census and an unsuccessful pilot study. The main contribution is a methodological framework for auditing benchmark validity rather than a completed attribution benchmark.
Critical strengths
-
Important and well-supported observations. The paper identifies several non-trivial construction artifacts with clear quantitative evidence. These findings highlight practical issues that can substantially affect the validity of attribution benchmarks.
-
Systematic auditing framework. The proposed Shortcut Gate offers a structured way to distinguish legitimate evidence use from potentially problematic shortcuts. I also appreciate the authors' transparency in reporting unsuccessful experiments and remaining limitations.
Critical weaknesses
-
Limited validation and methodological novelty. The proposed framework is mainly evaluated on the authors' own benchmark. Since several components build on existing auditing techniques, it remains unclear how much additional value the framework provides in more general settings.
-
Unclear construct validity. The identification of original sources relies on assumptions such as numerical uniqueness and citation absence, which do not necessarily establish true provenance. Moreover, the distinction between metadata leakage and legitimate provenance information could be better justified.
Questions for authors
-
Have the authors manually verified a sample of gold provenance labels to assess the reliability of the current construction criteria?
-
How does the proposed framework distinguish harmful metadata shortcuts from legitimate provenance evidence, such as publication chronology?
-
Can the authors further clarify what advantages the proposed build-level gate offers over existing shortcut auditing approaches, beyond the current case study?
2 comments
(Part 2/2) An author read the labels of only 8 of the 100 audited items, shown as the suggested answer (not blind); 2 were first disputed and kept after an explanation, one of them a defective item already excluded. The revision states this as a limitation. We ran a stratified audit with two language-model annotators (Appendix K): 100 of 348 items, fixed seed, design hashed before the draw; 12 strata (179-item subset or not family gold-passage section), 50 items per subset; rates design-weighted with Korn–Graubard 95% intervals. numeric extraction (exact / minor format / wrong) and origination (own finding / another source / cannot tell); an item is excluded when its value is wrong or from another source. claude-opus-5-5 and gpt-6.1-sol (run through Codex), both at effort xhigh and from different vendors, the second blinded to the design fields and to the first's labels. The 53 disagreements were adjudicated by claude-opus-5-5, the first annotator's model, so no label carries an independent check. on the judgements that enter a rate: = 0.85 (value wrong) and 0.95 (another source). 7.6% [3.3, 14.7] wrong, 17.4% [10.6, 26.3] from another source. Removing them changes neither gate verdict and moves the ceilings by at most 0.005. With 100 of 348 items audited this is a partial correction, and the gate certifies no label.
By the information each arm supplies, not by whether the cue would be legitimate evidence in the world. Rules reading only the question (T0) or the provenance map and document metadata (T2) fail the build if they exceed the null of their own candidate set at z > 3; rules reading the passage text (T1) may, and their maximum is published as a ceiling (Section 4.1, Table 2). Chronology shows why. When publication year appeared only on the document cards reached through the map, the older document's passage was gold in 131 of 135 items, because a review cites older work. Chronology is real provenance evidence, but there it existed in one arm only and would have produced exactly the map benefit under test, so it was T2. On the shipped build year is not rendered: the chronology rules abstain on every item and are reported as not exercisable, not as passing. A build that wants chronology as evidence must render it in the passage text of both arms, where the text-only ceiling counts it (Section 3.3).
Fewer than the submission claimed. On our pool a catalogue-matched item filter also reaches a passing subset of comparable size (119 in-sample, 165 to 183 cross-fitted, against 179), so build-level blocking is not needed and we withdraw that claim (Appendix E.2). What the comparison supports: treating evidence reading as fatal, as comparable procedures that count any solvability as a defect do, empties the pool under the matched filter, and a flat gate fails on every set. , with its mechanism: the 348 items fail on two named metadata rules. , on which the item filter's outcome turns. Beyond this case study we claim no advantage; the evidence is one construction on one corpus (Section 7.2). Table 11 compares the method with published audits and item filters on simultaneity, restricted access, null calibration and blocking granularity.
(Part 1/2) Thank you for the review. Your questions led to three changes: a catalogue-matched item filter on the same pool (Appendix E.2), which shows that build-level blocking is not needed to reach a passing subset and measures what the tier distinction changes; a stratified gold-label audit of 100 items (Appendix K); and an explicit tier criterion, applied to chronology (Section 3.3). The stated contribution is narrowed accordingly, and a pre-registered one-model evaluation is added and reported as found; its result is inconclusive (Section 6.5).
R2.1 (validation and novelty). Agreed: all evidence comes from one construction on one corpus, and we claim no transfer (Section 7.2). What we could test is how much the framework adds over the techniques it builds on, so the revision runs a catalogue-matched item filter on the same pool (Appendix E.2): same 63-rule catalogue, same per-rule nulls, evidence reading permitted, items dropped only where a crossing rule implicates them. It also reaches a passing subset (119 items in-sample; 165 to 183 cross-fitted, 11 of 11 seeds passing), so build-level blocking is not needed; we withdraw that claim. The tier distinction changes the outcome: with evidence reading treated as fatal, the filter removes all 348 items and a flat gate fails on every set. The filter’s outcome depends on its implication criterion (unique hits: fails at 310 items; any hit: empty); a build-level verdict does not. The contribution is narrowed to that combination (Sections 1, 2, 4.1; Table 11). We also registered, before calling the model, a one-model comparison of the 179 items the passing subset keeps against the 169 it removes (Section 6.5, Appendix L). Its result is inconclusive (D = −0.061, 95% interval [−0.122, 0.001], margin 0.10), and we draw no conclusion from it: the point estimate has the sign a shortcut would produce, but the interval includes zero, so the sign is not evidence of shortcut use; the two sets also differ in composition, with an effect of unestablished direction. The map-channel difference-in-differences is null. The evaluation does not establish that gate-passing items resist shortcuts better or worse.
R2.2 (construct validity). We agree that numerical uniqueness and citation absence are operational criteria and do not establish true provenance; the revision measures how often they fail. A stratified audit of 100 items (Appendix K; method in R2.3) finds, design-weighted over the shipped build, 7.6% [3.3, 14.7] of gold values wrong and 17.4% [10.6, 26.3] reported by the gold document from another source, which numerical uniqueness cannot see; Section 3.2 (condition 5) and Section 4.1 (“What a verdict certifies”) now say so. Removing the flagged items changes neither gate verdict and moves the text-only ceilings by at most 0.005. On metadata versus provenance, the criterion is now explicit (Section 3.3, Table 2): a cue's tier is set by the information each arm supplies, not by whether it would be legitimate evidence in the world. The provenance map is the manipulation under test, so a cue reachable only through it (T2) would produce the very map benefit the design is meant to detect, and a T2 rule beating its null fails the build. A cue in the passage text, available to both arms, is evidence reading (T1): it may beat its null, and its maximum is published as a ceiling. R2.4 applies this to chronology.