acceptodds
Withdrawn from ICLR 2027

Crossed-Witness Grounding for Evidence-State-Guided Visual Document QA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Partial evidence makes document question answering a question-relative decision: the same incomplete page can retain a direct witness for one question while losing it for another. Marginal scores on separately altered packets do not establish whether the same incomplete packet elicits opposite question-conditioned decisions. We introduce Crossed-Witness Grounding (CWG), a source-bound OCR intervention, reader-adaptation, and reciprocal-audit protocol. CWG pairs two questions with distinct witnesses on one page and removes either witness in turn. Each altered packet requires one exact answer with a citation to a visible block and one explicit refusal; the two deletions reverse these decisions. At deployment the adapted reader sees one question and its visible packet. On 77 source-disjoint held-out pairs, a frozen Qwen3-VL-8B reader passes 3 (3.9%) four-decision tests, paired adaptation passes 26, and the full training sequence passes 33 (42.9%). On 308 original questions, paired adaptation improves joint intact-answer and own-witness-removal behavior by 32.1 percentage points. A 2B reader shows the paired-training gain on the same protocol. Answer-preserving deletion exposure further improves the 8B four-decision score over matched intact continuation. CWG provides a strict reader-level test of question-conditioned evidence use on incomplete known-page packets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual document question answering, evidence grounding, selective prediction, counterfactual evaluation, source attribution, missing evidence

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mK6aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Crossed-Witness Grounding (CWG), a protocol for evaluating whether a document reader can condition its answer-or-abstain decision on the current question when only partial OCR evidence from a known page is available. The protocol pairs two questions with distinct direct-witness blocks on the same page and removes each witness in turn. On each identical altered packet, the reader must abstain on the question whose witness was removed while producing an exact answer with a correct citation for the question whose witness remains. The four-decision joint endpoint requires all decisions under both deletions to be correct, ruling out answer-or-abstain policies based solely on packet-level damage.The authors use paired answer-or-abstain supervision for LoRA adaptation and compare matched repeated-intact continuation, random answer-preserving deletion, and partner-witness deletion. On 77 source-disjoint held-out question pairs, the 8B reader improves from 3 successful pairs when frozen to 26 after paired adaptation and 33 after the full training sequence. Random deletion adds four successful pairs over matched intact continuation, while partner-witness deletion provides no further improvement. The 2B experiments reproduce the paired-adaptation gain but do not clearly establish an additional benefit from deletion augmentation over matched continuation. The empirical scope is restricted to selected direct-span questions and controlled OCR-block removal.

Critical strengths

S1. The crossed construction directly tests question-conditioned evidence use. The protocol requires opposite answer-or-abstain behavior for two questions on the same altered packet, with the decisions reversing when the other witness is removed. Equations 3–4 clearly explain why a policy based solely on packet-level state cannot satisfy all four decisions. This is a meaningful diagnostic design that can expose behavior not adequately characterized by reporting answer accuracy or abstention under evidence removal separately. (§3.1, Figure 1) S2. The continuation controls are carefully designed, and the interpretation is appropriately restrained. The three continuation arms share the paired initialization and match training rows, target balance, and optimization schedule, helping distinguish additional training from answer-preserving deletion and deliberate partner-witness selection. The authors explicitly report that partner deletion does not outperform random deletion and acknowledge that the deletion-specific increment is unclear at 2B. They do not attribute all improvements to the specific crossed augmentation, which strengthens the credibility of the experimental interpretation. (§3.2; §4.3–4.4; Tables 2–3; Appendix A) S3. Paired answer-or-abstain supervision produces substantial gains within the defined task. For the 8B reader, original-task joint success increases from 78/308 to 177/308, while crossed success improves from 3/77 to 26/77. Paired training also raises the 2B reader’s crossed success from 0/77 to 17/77. These results support the effectiveness of explicit answer-or-abstain supervision in this setting. Jointly requiring correct answers, correct citations, and explicit refusals is also more informative than measuring abstention propensity alone. (Tables 1–3)

Critical weaknesses

W1. The additional empirical value of the audit over existing evidence-faithfulness approaches is not sufficiently established. The paper acknowledges prior work using relevant/irrelevant evidence interventions, and its training objective is standard supervised sequence learning. The clearest new contribution therefore lies in the two-question crossed audit on shared packets. However, the experiments primarily compare training variants within one model family. They do not sufficiently demonstrate which important failures of existing evidence-faithfulness approaches the audit reveals, or which model-comparison conclusions are not clearly captured by conventional per-question three-view analyses. The tie between partner and random deletion further concentrates the contribution in the audit itself. The construction is reasonable, but the additional knowledge it provides needs stronger empirical support. (§2; §3.2–3.3; Tables 2–3) W2. The semantic validity of the crossed labels is insufficiently validated. Eligibility checks primarily rely on answer-string occurrence and retention of the mapped witness block. Retaining the answer-bearing block does not necessarily preserve the year, unit, header, or other context needed to interpret that answer. String presence alone therefore does not guarantee that the current question retains sufficient direct support after the partner block is deleted. The PDF does not report independent semantic validation of these cases. In addition, packets are constructed around annotated witnesses, and sensitivity to this window-construction procedure is not sufficiently assessed. The authors appropriately restrict the task to direct witnesses, but the generated labels still need validation within that contract. (Equations 1–2; Appendix A) W3. The incremental deletion-augmentation gains are small, and their stability and answer–abstention trade-off remain unclear. Relative to matched intact continuation, random deletion yields a net gain of 4/77 successful pairs at 8B and only 1/77 at 2B, with the latter confidence interval including zero. Source-clustered intervals capture uncertainty over test sources but do not account for variation across independent training runs, and multiple-seed results are not reported for the continuation arms. Meanwhile, random-deletion continuation reduces the 8B reader’s correct intact answers on the original task from 226/308 at the paired starting point to 216/308. The evidence is therefore insufficient to establish whether the additional intervention consistently improves the overall answer–abstention trade-off, beyond increasing abstention propensity and the particular joint endpoint. (§4.3–4.4; Tables 2–4)

Questions for authors

1.Can you use the existing predictions to demonstrate important failure modes or model-comparison differences that conventional per-question intact/own-deletion/other-deletion metrics miss, but the shared-packet four-decision audit identifies? Are results available for existing evidence-faithfulness approaches under this protocol that would further establish its independent diagnostic value? 2.Beyond answer-string checks, how do you verify that deleting the partner-witness block preserves the headers, units, and context required to answer the current question? Have the crossed test labels undergone independent semantic validation? If so, how many inconsistent cases were identified, and would correcting them change the main results? 3.Does the four-pair net gain over matched continuation at 8B persist across training seeds? Can you provide the pair-level success/failure transitions between these conditions and analyze the reduction in correct intact answers, to determine whether the gain is stable and reflects an improved answer–abstention trade-off?

Reviewer_yK5tOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper holds an interesting setting of a document with partial data loss, and proposes crossed-witness grounding, a method to evaluate whether a reader can correctly decide whether to answer a question or decline to answer, based on whether the evidence supporting its answer remains intact. The approach pairs two questions that rely on distinct evidence blocks within a single page. Then it alternately removes the evidence for each question and presents both questions to the model. To pass, the model must decline to answer the question for which the evidence was removed, while providing an answer for the question for which the evidence remains. The training process consists of two stages. first, applying LoRA to teach the model both how to answer and how to decline. second, further training the model to ensure it continues to provide answers even when other, irrelevant blocks are removed. Tested on 77 pairs derived from the OCR text of TAT-DQA, the success rate significantly improved from almost zero to around 40 %. Notably, the strategy of specifically removing the evidence associated with a paired question performed no differently than randomly removing document blocks.

Critical strengths
  1. The observation that a model might be able to answer some questions but not others is insightful. The authors also make a persuasive point that existing evaluation frameworks, which score original and evidence-ablated versions separately, can inadvertently reward models that simply "decline to answer" whenever any part of the document is missing.
  2. The method of pairing two questions into a single "packet" and systematically removing the evidence for each in turn appears to be a unique contribution, at least within the context of the related literature cited by the authors.
  3. Complex concepts are formalized mathematically, which aids in understanding.
  4. The authors honestly report instances where their proposed method fails to yield improvements.
Critical weaknesses
  1. The evaluation is restricted to a single dataset (TAT-DQA) and Qwen3 model family (2B/8B). Although MMDocIR is mentioned as a dataset, no results for it are provided.
  2. The baselines consist solely of frozen (untrained) models. Since these frozen models must adhere to specific output requirements without the benefit of training, a significant portion of the observed performance gains may simply reflect the model learning the required output format. Comparisons against prompted or larger models are necessary.
  3. The study only includes questions where the ground truth answer appears exactly once in the document. Consequently, numerical reasoning questions are excluded while mentioning finance application in the beginning. 4 The authors created their own training and test splits from the official TAT-DQA training set rather than using the official setting, making direct comparison with other research on TAT-DQA impossible.
  4. The experiment scale is very small and insufficient to validate the claim. The primary evaluation is based on only 77 pairs, resulting in wide confidence intervals for the main findings.
  5. Although the title mentions "Visual" Document QA, the actual input consists of OCR text, and it is not specified whether layout or coordinate information is utilized. Furthermore, while the term "page" is used, the actual input is merely a short "packet" (with a median of 11 blocks) cropped around two pieces of evidence.
  6. While the title and abstract claim to address "evidence use" in a broad sense, the metric actually measures only whether the model responds when a single block containing the correct answer string remains present.
  7. While the method section is readable, some symbols and terms remain undefined. For instance, idx\mathrm{idx} in Eq. (2), iqi_q in Eq. (5), and the term "evidence state" (mentioned in the title) are never defined. Furthermore, non-standard terms such as "packet," "witness," and "publisher" are used without explanation, starting as early as the abstract, which makes the abstract particularly difficult to follow.
Questions for authors
  1. Can the authors generalize their method by reporting results on additional datasets and other model families?
  2. How much of the performance gain comes from learning the output format rather than from improved answer-or-withhold decisions? How do prompted models or larger models perform on the same test?
  3. How would the approach extend to questions whose answers require numerical reasoning, which are central to financial document QA?
  4. Why did the authors construct their own splits instead of using the official TAT-DQA splits? Can results on the official setting be provided?
  5. Given the small evaluation set (77 pairs), how stable are the main results across different training seeds?
  6. Would the conclusions still hold with image inputs, full pages, or multi-page documents instead of short OCR packets?
Reviewer_8EPrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Crossed-Witness Grounding (CWG), a framework for evaluating and improving question-conditioned evidence use in document QA. By selectively removing OCR evidence blocks, CWG constructs paired answer-or-withhold tasks and introduces a four-decision evaluation protocol. Experiments on TAT-DQA-derived data show that the four-way success rate improves from 3.9% for the frozen 8B reader to 42.9% after adaptation.

Critical strengths
  1. Meaningful problem formulation. The paper addresses an important aspect of evidence-grounded document QA: distinguishing missing evidence from incomplete documents that still contain sufficient evidence for a particular question. The crossed-witness evaluation provides a useful way to detect indiscriminate abstention.

  2. Carefully controlled experiments. The paper compares positive-only, paired, and answer-preserving deletion training, with matched continuation controls and source-disjoint evaluation. The substantial improvement over the frozen reader demonstrates the value of evidence-aware adaptation within the proposed setting.

Critical weaknesses
  1. Limited evidence for the distinctive training mechanism. The partner-witness and random-preserving deletion strategies achieve identical four-way accuracy (42.9%) on the 8B reader. Moreover, most of the improvement over the frozen model comes from paired answer-or-withhold training. While the crossed-witness evaluation is valuable, the experiments do not establish a clear advantage of the proposed partner-witness training strategy over generic evidence-preserving deletion.

  2. Restricted evaluation setting. The experiments use controlled OCR-block deletion, exact answer-string matching, and known-page packets from a single document dataset. The main four-way evaluation contains only 77 held-out pairs. Furthermore, the reader operates on OCR text rather than document images. These constraints limit the extent to which the findings generalize to realistic document QA involving noisy OCR, implicit evidence, and multi-page reasoning.

Questions for authors
  1. Since partner-witness deletion performs similarly to random evidence-preserving deletion, what additional benefit does the crossed-witness training strategy provide beyond the proposed evaluation protocol?

  2. How would the approach handle questions requiring multiple evidence blocks or implicit reasoning rather than exact answer spans? Can the authors provide evidence that the observed improvements extend beyond the current direct-witness setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.