- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Crossed-Witness Grounding (CWG), a protocol for evaluating whether a document reader can condition its answer-or-abstain decision on the current question when only partial OCR evidence from a known page is available. The protocol pairs two questions with distinct direct-witness blocks on the same page and removes each witness in turn. On each identical altered packet, the reader must abstain on the question whose witness was removed while producing an exact answer with a correct citation for the question whose witness remains. The four-decision joint endpoint requires all decisions under both deletions to be correct, ruling out answer-or-abstain policies based solely on packet-level damage.The authors use paired answer-or-abstain supervision for LoRA adaptation and compare matched repeated-intact continuation, random answer-preserving deletion, and partner-witness deletion. On 77 source-disjoint held-out question pairs, the 8B reader improves from 3 successful pairs when frozen to 26 after paired adaptation and 33 after the full training sequence. Random deletion adds four successful pairs over matched intact continuation, while partner-witness deletion provides no further improvement. The 2B experiments reproduce the paired-adaptation gain but do not clearly establish an additional benefit from deletion augmentation over matched continuation. The empirical scope is restricted to selected direct-span questions and controlled OCR-block removal.
Critical strengths
S1. The crossed construction directly tests question-conditioned evidence use. The protocol requires opposite answer-or-abstain behavior for two questions on the same altered packet, with the decisions reversing when the other witness is removed. Equations 3–4 clearly explain why a policy based solely on packet-level state cannot satisfy all four decisions. This is a meaningful diagnostic design that can expose behavior not adequately characterized by reporting answer accuracy or abstention under evidence removal separately. (§3.1, Figure 1) S2. The continuation controls are carefully designed, and the interpretation is appropriately restrained. The three continuation arms share the paired initialization and match training rows, target balance, and optimization schedule, helping distinguish additional training from answer-preserving deletion and deliberate partner-witness selection. The authors explicitly report that partner deletion does not outperform random deletion and acknowledge that the deletion-specific increment is unclear at 2B. They do not attribute all improvements to the specific crossed augmentation, which strengthens the credibility of the experimental interpretation. (§3.2; §4.3–4.4; Tables 2–3; Appendix A) S3. Paired answer-or-abstain supervision produces substantial gains within the defined task. For the 8B reader, original-task joint success increases from 78/308 to 177/308, while crossed success improves from 3/77 to 26/77. Paired training also raises the 2B reader’s crossed success from 0/77 to 17/77. These results support the effectiveness of explicit answer-or-abstain supervision in this setting. Jointly requiring correct answers, correct citations, and explicit refusals is also more informative than measuring abstention propensity alone. (Tables 1–3)
Critical weaknesses
W1. The additional empirical value of the audit over existing evidence-faithfulness approaches is not sufficiently established. The paper acknowledges prior work using relevant/irrelevant evidence interventions, and its training objective is standard supervised sequence learning. The clearest new contribution therefore lies in the two-question crossed audit on shared packets. However, the experiments primarily compare training variants within one model family. They do not sufficiently demonstrate which important failures of existing evidence-faithfulness approaches the audit reveals, or which model-comparison conclusions are not clearly captured by conventional per-question three-view analyses. The tie between partner and random deletion further concentrates the contribution in the audit itself. The construction is reasonable, but the additional knowledge it provides needs stronger empirical support. (§2; §3.2–3.3; Tables 2–3) W2. The semantic validity of the crossed labels is insufficiently validated. Eligibility checks primarily rely on answer-string occurrence and retention of the mapped witness block. Retaining the answer-bearing block does not necessarily preserve the year, unit, header, or other context needed to interpret that answer. String presence alone therefore does not guarantee that the current question retains sufficient direct support after the partner block is deleted. The PDF does not report independent semantic validation of these cases. In addition, packets are constructed around annotated witnesses, and sensitivity to this window-construction procedure is not sufficiently assessed. The authors appropriately restrict the task to direct witnesses, but the generated labels still need validation within that contract. (Equations 1–2; Appendix A) W3. The incremental deletion-augmentation gains are small, and their stability and answer–abstention trade-off remain unclear. Relative to matched intact continuation, random deletion yields a net gain of 4/77 successful pairs at 8B and only 1/77 at 2B, with the latter confidence interval including zero. Source-clustered intervals capture uncertainty over test sources but do not account for variation across independent training runs, and multiple-seed results are not reported for the continuation arms. Meanwhile, random-deletion continuation reduces the 8B reader’s correct intact answers on the original task from 226/308 at the paired starting point to 216/308. The evidence is therefore insufficient to establish whether the additional intervention consistently improves the overall answer–abstention trade-off, beyond increasing abstention propensity and the particular joint endpoint. (§4.3–4.4; Tables 2–4)
Questions for authors
1.Can you use the existing predictions to demonstrate important failure modes or model-comparison differences that conventional per-question intact/own-deletion/other-deletion metrics miss, but the shared-packet four-decision audit identifies? Are results available for existing evidence-faithfulness approaches under this protocol that would further establish its independent diagnostic value? 2.Beyond answer-string checks, how do you verify that deleting the partner-witness block preserves the headers, units, and context required to answer the current question? Have the crossed test labels undergone independent semantic validation? If so, how many inconsistent cases were identified, and would correcting them change the main results? 3.Does the four-pair net gain over matched continuation at 8B persist across training seeds? Can you provide the pair-level success/failure transitions between these conditions and analyze the reduction in correct intact answers, to determine whether the gain is stable and reflects an improved answer–abstention trade-off?