acceptodds
Under review as a conference paper at ICLR 2027

CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

Abstract

Multimodal Large Language Models (MLLMs) have significantly advanced document understanding, yet current Doc-VQA evaluations score only the final answer and leave the supporting evidence unchecked. This answer-only approach masks a critical failure mode: a model can land on the correct answer while grounding it in the wrong passage—a critical risk in high-stakes domains like law, finance, and medicine, where every conclusion must be traceable to a specific source region. To address this, we introduce CiteVQA, a benchmark that requires models to return element-level bounding-box citations alongside each answer, evaluating both jointly. CiteVQA comprises 1,897 questions across 711 PDFs spanning seven domains and two languages, averaging 40.6 pages per document. To ensure fidelity and scalability, the ground-truth citations are generated by an automated pipeline—which identifies crucial evidence via masking ablation and enforces multi-stage quality control. At the core of our evaluation is Strict Attributed Accuracy (SAA), which credits a prediction only when the answer and the cited region are both correct. Auditing 20 MLLMs reveals a pervasive Attribution Hallucination: models frequently produce the right answer while citing the wrong region. The strongest system (Gemini-3.1-Pro-Preview) achieves an SAA of only 76.0, and the strongest open-source MLLM reaches just 22.5. Ultimately, towards trustworthy document intelligence, CiteVQA exposes a reliability gap that answer-only evaluations overlook, providing the instrumentation needed to close it. Our repository is available at https://anonymous.4open.science/r/CiteVQA.

Primary area:
datasets and benchmarks
Keywords:
MLLM, DocVQA, Evidence Attribution, Document Intelligence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_not6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CiteVQA evaluates document question answering together with element-level bounding-box citations. It contains 1,897 questions over 711 PDFs across seven domains and two languages. An automated pipeline constructs questions and evidence annotations, using masking ablations to identify crucial evidence. The paper introduces Strict Attributed Accuracy (SAA), evaluates 20 multimodal models, and finds that correct answers often come with incorrect citations. Additional analyses include human audits, input-resolution ablations, and training-based validation of the annotation pipeline.

Critical strengths
  1. The task addresses a useful gap in document evaluation. Requiring a model to identify the specific passage, table, or figure supporting its answer makes failures visible that answer-only evaluation misses. The inclusion of long documents and multi-document questions makes this relevant to practical document assistants.
  2. The paper makes a substantial effort to validate the resource. Beyond evaluating 20 models, the authors provide a 200-instance annotation audit and an auxiliary training comparison against manually annotated data (Appendices B.3–B.4). These results offer useful evidence that the automated pipeline produces meaningful supervision.
Critical weaknesses
  1. SAA does not reliably enforce the attribution requirement claimed by the paper. In Section 4.1, an answer passes whenever Ans. ≥ 4 and Rec. ≥ 0.6, even if its relevance score is low. For example, citing two of three crucial elements would suffice despite omitting evidence explicitly labeled necessary. Adding incorrect citations also does not reduce this recall. Reporting precision separately does not resolve this weakness in the headline metric. Moreover, the judge validation in Appendix C.2 reports similar mean scores and nonsignificant Friedman tests, which do not establish agreement on individual pass/fail decisions at the threshold of 4. For a benchmark centered on trustworthy attribution, I would need stronger evidence that SAA accepts supported answers and rejects unsupported ones. A breakdown by acceptance path and threshold-level agreement with the existing human judgments would help.
  2. The model comparisons confound attribution capability with input quality. Table 9 gives Gemini native PDF input, GPT full-resolution page images, and most open models downscaled images. Table 10 shows that this matters substantially: reducing Qwen3-VL-235B’s pixel budget by half lowers SAA from 22.5 to 11.8. The authors acknowledge this limitation, but the main discussion still draws strong conclusions about the grounding reliability of different model tiers. The claim that 1024 × 1024 is a saturation point is also unsupported by an ablation that only tests lower resolutions. A comparison using the same rendered inputs on a feasible subset would help separate these effects. Otherwise, the conclusions should be limited to the evaluated model-and-input configurations.
Questions for authors
  1. Metric clarification: For each model, how many SAA-positive predictions pass only through Rec. ≥ 0.6 while Rel. < 4? Do the existing human annotations indicate whether these citations adequately support the answers?
  2. Judge validation: On the 200 examples in Appendix C.2, what is the sample-level agreement between human and automated judgments for Ans. ≥ 4 and Rel. ≥ 4? Similar average scores could conceal disagreements that change SAA.
  3. Input comparability: Were any comparisons conducted using identical rendered pages and resolutions across model families? If so, do the main ranking and performance gaps persist? I am asking about existing controls rather than requesting a new full-benchmark evaluation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer not6 #1

We thank you for the review. The two main weaknesses you raise (SAA's pass condition and input comparability) and the three specific questions (rec_only samples, judge threshold agreement, and the same-input comparison) all bear on what is essential to benchmarking. We answer W1 together with Q1 and W2 together with Q3, and answer Q2 separately.


W1 + Q1. SAA's Pass Condition and the rec_only Samples

(1) The distribution of necessary evidence means that Rec. ≥ 0.6 is equivalent to "citing everything" on most questions. We counted the number of necessary evidence elements over the full evaluation set:

Necessary evidence per questionShare of questions
162.0%
224.2%
310.1%
≥43.6%

86.2% of questions have only 1–2 necessary elements. For these questions, Rec. ≥ 0.6 effectively requires citing all of them: 1/1 or 2/2; citing 1 of 2 gives only 0.5, which fails. The case you raise—citing 2 of 3 (2/3 = 0.667)—applies only to the 10.1% of questions that have exactly 3 necessary elements; for 4 or more (3.6%), a threshold of 0.6 requires at least 3/4, 3/5, and so on.

(2) Raising the Rec threshold to 1.0 leaves SAA almost unchanged. To test directly the effect of "2/3 is enough to pass", we used the paper's existing per-question results to raise the Rec branch of SAA from 0.6 to 1.0 (with all other conditions unchanged) and recomputed over the 20 models: the number of passes drops from 9,929 to 9,915, a decrease of only 14 (0.14%). We will add this re-computation to the revision's appendix as a sensitivity analysis.

(3) Per-model share of rec_only. The table below decomposes the passing samples of the 20 models by pass path into both, rel_only, and rec_only:

ModelPassesbothrel_onlyrec_onlyrec_only share of passes
Gemini-3.1-Pro1,441973437312.2%
Gemini-3-Flash1,241607587473.8%
GPT-5.41,117407669413.7%
Gemini-2.5-Pro8922815149710.9%
Qwen3-VL-235B427124288153.5%
All 20 models9,9293,5125,9824354.4%

Of the 435 rec_only samples, 421 (96.8%) already have Rec = 1.0; only 14 (3.2%) are partial citations with 0.6 ≤ Rec < 1.0. These samples have Rec of either 0.667 or 0.8 (i.e., 2/3 or 4/5)—exactly the case you describe; all of them satisfy Ans. ≥ 4, so the answer itself is correct. They account for 0.14% of all passes. Per model, only Gemini-2.5-Pro exceeds 10% (97/892 = 10.9%); the remaining 15 models range from 0.7% (Qwen3-VL-8B) to 6.8% (GLM-5V-Turbo).

(4) Additional citations do not lower Rec., and this is by design: what matters is that the necessary evidence is cited. You are right that Rec. does not decrease when extra citations are added. This is not a flaw in the metric but a deliberate difference between CiteVQA and retrieval-style metrics. SAA's pass condition is "the answer is correct and the necessary evidence is cited", not "the cited set must equal the minimal evidence set". The reasons are as follows: B_gt is split into B_crucial and B_other (Section 4.1), B_other is legitimate evidence, and a model should not be penalized for citing it; the same answer admits multiple possible evidence granularities (whole paragraph, sentence, table cell), and requiring a unique minimal set would mark reasonable citations as wrong. Redundant citations are reported separately by Precision/F1 (Table 12); SAA itself does not take on the responsibility of judging whether citations are minimal. If the headline metric also penalized redundancy, it would conflate "citing additional correct evidence" with "citing wrong evidence", weakening the metric's focus on attribution errors.

(5) On the rejection side, the gap between SAA and Ans. shows that SAA rejects many samples whose answers are correct but whose citations fall short. For example, Qwen3-VL-235B has a Single-Doc Ans. of 75.0 but an SAA of only 25.0; Gemini-3.1-Pro has an Ans. of 86.7 and an SAA of 76.0. The paper's decomposition of SAA = 0 samples (Figure 5, Appendix C.6) further shows that, among all failures, boundary imprecision accounts for only 1.0%–5.0%, whereas wrong-page and wrong-element citations together account for 34.8%–51.7%; among answer-correct failures, the dominant mode is genuinely citing the wrong evidence, not imprecise box drawing.


AuthorsOpenReview ↗

Response to Reviewer not6 #2

W2 + Q3. Input Comparability and the "Saturation Point"

(1) The "might yield marginal gains" claim in the paper is incorrect, and the new upward ablation confirms this directly. The original sentence in Appendix C.1 reads: "Although further increasing the resolution might yield marginal gains, 1024×1024 represents a critical 'saturation point'...". Our intended meaning was that, within a 256K context, 1024² is the highest uniform resolution that covers all documents; however, the "marginal gains" claim is inconsistent with the experiments—the upward ablation below shows that SAA improves substantially from 1024² to 2048². The revision will remove "marginal gains" and clarify "saturation point" as "the uniform upper limit under the context budget", to avoid confusion with performance saturation.

(2) The upward ablation and the same-input comparison show that 1024² is not a performance saturation point. On the 341 Single-Doc questions with at most 25 pages:

Model / InputSAARec.Rel.Ans.
Gemini-3.1-Pro, native PDF78.671.983.587.9
Gemini-3.1-Pro, 1024² page images54.353.163.087.9
Qwen3-VL-235B, 1024²28.715.945.377.7
Qwen3-VL-235B, 1536²49.035.063.482.9
Qwen3-VL-235B, 2048²63.145.774.082.5

Qwen's SAA rises from 28.7 to 49.0 and then to 63.1, and Rec. rises from 15.9 to 45.7, while Ans. rises only from 77.7 to 82.5: resolution mainly improves in-page localization and has little effect on answer correctness. The downward ablation in Table 10 (22.5→11.8) points in the same direction; together they show that 1024² is merely an operating point under the context constraint, not a performance ceiling. On the same 1024² page images, Gemini-3.1-Pro scores 54.3 and Qwen 28.7; even when Qwen is raised to 2048² (63.1), it remains below Gemini's 78.6 on native PDF.

Under the same input, both models' Ans. remains clearly higher than SAA, and Attribution Hallucination persists; the gap between closed-source and open-source models still lies mainly in attribution, rather than being caused by input differences.

(3) The choice of 1024² is a consequence of the context constraint. Qwen3-VL-235B has a native context of 256K. Using the per-page token counts measured on the 341 questions, we extrapolate to all 987 Single-Doc questions:

ResolutionTokens per page (approx.)Max pages within 256KQuestions that fit
1024²~930~261987/987 (100%)
1536²~2,200~110923/987 (94%)
2048²~4,000~61793/987 (80%)

The longest 185-page document requires about 737K tokens at 2048², far beyond 256K; Multi-Doc questions involve more pages in total, so the constraint is only tighter. A uniform evaluation can therefore stop only at 1024²: this is the highest resolution that covers the entire benchmark, not a saturation point for attribution performance.

(4) Input configuration and the scope of our conclusions. Most models in Table 9 (the 256K row: Qwen3.5, Qwen3-VL, Gemma-4, Kimi-K2.5, Seed-2.0-Pro) already use the same set of 1024² page images, the same rendering pipeline, and the same prompt; GLM-5V-Turbo uses a compact 768² input; only GPT-5.4/GPT-5.2/Qwen3.6-Plus use full-resolution page images, and Gemini uses native PDF. The same-input comparison is already given in (2): the relative ordering of the two models is unchanged, and the gap lies mainly in attribution. We agree that absolute scores under different input configurations are not directly comparable. The revision will state the input configuration of each model class in the main text, and will also make explicit the setting of "under the highest-fidelity input available to each model": subject to covering the full benchmark, each model is given the highest input fidelity its context allows, without artificially lowering long-context or native-PDF models just to unify the input—context length is itself part of model capability, and sustaining high-resolution input under a large context is a genuine advantage in real deployment.

AuthorsOpenReview ↗

Response to Reviewer not6 #3

Q2. Per-Sample Agreement Between Human and Automatic Judges at Threshold 4

The table below reports the threshold-decision agreement rate on the 200 samples of Appendix C.2. Specifically: for each sample, the human score and the automatic judge score are each binarized at threshold 4 (≥ 4 = pass, < 4 = fail); agreement scores 1 and disagreement scores 0; the values are the proportion of the 200 samples for that reasoning model on which the two decisions agree.

Reasoning modelThresholdHuman vs Qwen judgeHuman vs Gemini judge
GPT-5.4Rel. ≥ 487.5%89.0%
GPT-5.4Ans. ≥ 493.0%92.5%
Gemini-3.1-ProRel. ≥ 491.0%92.0%
Gemini-3.1-ProAns. ≥ 494.5%93.5%

This is consistent with the mean differences in Table 11 (0.02–0.16) and the Friedman tests (p = 0.14–0.53): there is no systematic bias between the automatic judges and the human experts.


Finally, we thank you for the careful review. The supplementary experiments and statistics above will be fully presented in the revision, and we would be grateful for further comments.

AuthorsOpenReview ↗

Summary of Our Responses to Reviewer not6

Thank you for your review. For your convenience, here is a summary of the main points of our earlier replies.

  • W1 + Q1 (SAA's pass condition and the rec_only samples). Rec. ≥ 0.6 is in fact equivalent to "cite everything" on the vast majority of questions — necessary evidence typically consists of only one or two elements, and citing one of two yields Rec. = 0.5, below the threshold. We also tested the effect of partial citations directly: recomputing all 20 models with the Rec. threshold raised from 0.6 to 1.0 changes the number of passes by only 0.14%, so this case is negligible for SAA. As for extra citations not lowering Rec., this is intentional: SAA passes when "the answer is correct and the necessary evidence is cited", not when "the cited set equals the minimal evidence set"; redundancy is reported separately by Precision/F1, and having the headline metric penalize it as well would conflate "citing additional correct evidence" with "citing wrong evidence". Conversely, the gap between SAA and Ans. shows that what SAA rejects is precisely the "correct answer, wrong citation" case — in the failure decomposition, loose box boundaries account for only a small fraction, while citing the wrong page or element dominates.
  • W2 + Q3 (input comparability and the "saturation point"). The original "might yield marginal gains" statement is wrong; the revision will remove it and define "saturation point" explicitly as "the uniform ceiling under the context budget", so that it is not confused with performance saturation. The upward ablation shows that 1024² is not a performance ceiling: raising resolution mainly improves in-page localization and affects answer correctness little. Stopping the uniform evaluation at 1024² is a context constraint — at higher resolutions a considerable fraction of long documents no longer fits, and 1024² is the highest uniform setting that covers the entire benchmark. With exactly the same 1024² input as the open-source models, the closed- vs. open-source gap persists and still lies mainly in attribution rather than answer correctness; Attribution Hallucination is present under every configuration. The revision will state each model class's input configuration in the main text, together with the principle of "the highest-fidelity input available to each model".
  • Q2 (judge threshold agreement). Binarizing the human and automatic judge scores at threshold 4 and comparing them sample by sample, agreement exceeds 87% throughout, consistent with the mean differences and the Friedman test: there is no systematic bias between the automatic judge and the human experts.

These additional experiments and statistics will be included in the revised paper. The discussion period is drawing to a close; should it be convenient for you, we would be grateful to hear your further comments before it ends, and we hope to continue the discussion. Thank you again for your careful review. Best regards.

Reviewer_kxkeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CiteVQA, a benchmark for evaluating evidence attribution in document visual question answering. It comprises 1,897 questions from 711 PDFs across seven domains, covering single- and multi-document scenarios. The authors also introduce an automated pipeline for generating questions and fine-grained evidence annotations, using the Supported Answer Accuracy (SAA) metric to assess answer correctness and evidence attribution. Experiments on 20 multimodal models reveal a pervasive gap between answer accuracy and evidence attribution performance.

Critical strengths
  1. Meaningful research problem. The paper identifies the gap between answer correctness and evidence attribution in existing document VQA benchmarks and introduces CiteVQA to address this limitation.

  2. Comprehensive benchmark construction. CiteVQA covers diverse domains, languages, and single- and multi-document scenarios. Its automated construction pipeline also provides fine-grained evidence annotations for systematic evaluation.

  3. Experimental evaluation. The evaluation of 20 multimodal models provides useful insights into current evidence attribution capabilities.

Critical weaknesses

I have the following concerns:

  1. Evidence annotations. The masking-based approach relies on an MLLM's responses to identify crucial evidence, which may not reliably establish the necessity of each annotated region. Although the authors conduct a human quality assessment, it remains unclear whether the crucial evidence annotations are sufficiently accurate or whether alternative supporting evidence is adequately considered.

  2. Validation of the SAA metric. The OR-based criterion allows samples to pass through either evidence relevance or localization recall, but the paper does not analyze the contributions of these two pathways. Moreover, the human evaluation does not directly assess agreement on the threshold-based SAA decisions.

  3. Evaluation configurations. Different models use varying document input strategies and image resolutions, which may affect attribution performance. Given the reported sensitivity to image resolution, it is unclear to what extent the observed performance differences reflect model capabilities.

  4. Comparison with existing benchmarks. The paper discusses related benchmarks such as ViDoRe V3 and SciEGQA. Can the author further clarify whether CiteVQA provides different evaluation insights or reveals model limitations that existing benchmarks cannot adequately capture.

  5. Figure readability. Some figures, such as Figures 3 and 4, contain text that is small to read.

Questions for authors

The main questions have been outlined in the Weaknesses section.

12 comments
AuthorsOpenReview ↗

Response to Reviewer kxke #1

We thank the reviewer for the evaluation. Your comments on the necessity of masking-based annotation, the two SAA pass paths, the comparability of input configurations, and the positioning of the benchmark are all specific, and we address them one by one. W1 and W2 both concern whether the annotations and the metric can support the claim of "trustworthy attribution", so we answer them together.


W1 + W2. Necessity of Evidence Annotation and Validation of SAA

(1) Masking is itself a test for alternative supporting evidence. The Evidence Package is proposed by MLLM agents from the parsed elements, but whether an element is "crucial" is not decided by the MLLM's free-form relevance judgment; it is determined by the outcome of a leave-one-out intervention: each time only one element is masked while the others remain visible, and the model is asked to answer. If the model can still answer correctly, the element can be substituted by the remaining evidence and is therefore not labeled crucial; only when the model fails after masking is the element labeled Crucial Evidence. In other words, "the existence of alternative supporting evidence" is exactly what this intervention is designed to identify: an element that can be substituted is not treated as the sole basis. To prevent the model from answering from parametric knowledge, candidate questions first go through a zero-document self-test, and questions that can be answered without any document are removed (Section 3.3). In addition, non-crucial elements are not removed from the ground truth but retained as supplemental evidence (Section 4.1 splits B_gt into B_crucial and B_other), and the evaluation does not require the model to cite only crucial evidence.

(2) Decomposition of the SAA OR paths. Using the paper's existing per-question results, we split all passing samples of 20 models into three paths: both (Rel ≥ 4 and Rec ≥ 0.6), rel_only (Rel ≥ 4 and Rec < 0.6), and rec_only (Rel < 4 and Rec ≥ 0.6, i.e., passing through Recall alone). The main models are as follows:

ModelPassesbothrel_onlyrec_onlyrec_only share of passes
Gemini-3.1-Pro1,441973437312.2%
Gemini-3-Flash1,241607587473.8%
GPT-5.41,117407669413.7%
Gemini-2.5-Pro8922815149710.9%
Qwen3-VL-235B427124288153.5%

Across all models, the three paths account for 35.4% (both), 60.2% (rel_only), and 4.4% (rec_only). In other words, samples that pass through Rec alone are rare: only 435 of the 9,929 passing samples. Per model, only Gemini-2.5-Pro exceeds 10% (10.9%), and all others are at or below 6.8%. The case of "citing only part of the crucial evidence and failing the relevance threshold yet still passing" therefore has little effect on SAA.

How should we understand the contribution of the two paths? For the strongest model (e.g., Gemini-3.1-Pro), both accounts for about two-thirds of its passing samples (68%), the highest among the 20 models, and the agreement between the Rel and Rec decisions is good. For weaker models, Rec is stricter: the model may produce smaller boxes (e.g., covering only a sentence rather than the whole element), so IoU falls below 0.5; or it may not distinguish the granularity required by the evaluation (complete paragraph/table/figure), so its boxes do not align with the GT element. These citations are not necessarily wrong; they simply do not reach the IoU threshold against element-level GT, and in this case Rel is a strong complement to Rec. This is also consistent with the observation in W1 that valid evidence may exist outside the GT.

SAA failures are not dominated by box-drawing precision. The paper's decomposition of SAA = 0 samples (Figure 5, Appendix C.6) shows that, among all failures, Boundary Imprecision alone accounts for only 1.0%–5.0%, whereas wrong-page and wrong-element citations jointly account for 34.8%–51.7%. Among answer-correct failures, the dominant mode is therefore genuinely citing the wrong evidence (Attribution Hallucination), not imprecise box drawing; the role of Rel is to catch cases where the citation is valid but at a different granularity, not to relax the judgment of incorrect citations.

AuthorsOpenReview ↗

Response to Reviewer kxke #2

W1 + W2. Necessity of Evidence Annotation and Validation of SAA (continued)

(3) Per-sample agreement between human and automatic judges at threshold 4. Regarding your point that "the human evaluation did not directly assess agreement on the thresholded SAA decision", we binarized human and automatic judge scores on the 200 samples of Appendix C.2 at threshold 4 (≥ 4 = pass, < 4 = fail) and computed the proportion of samples on which the two decisions agree (two reasoning models, GPT-5.4 and Gemini-3.1-Pro, × two judges, Qwen and Gemini):

Reasoning modelThresholdHuman vs Qwen judgeHuman vs Gemini judge
GPT-5.4Rel. ≥ 487.5%89.0%
GPT-5.4Ans. ≥ 493.0%92.5%
Gemini-3.1-ProRel. ≥ 491.0%92.0%
Gemini-3.1-ProAns. ≥ 494.5%93.5%

Agreement is 87.5%–92.0% for Rel. and 92.5%–94.5% for Ans. This is consistent with the mean differences in Table 11 (0.02–0.16) and the Friedman tests (p = 0.14–0.53), indicating no systematic bias between the automatic judges and the human experts. The revision will include this analysis in the appendix.


W3. Input Configuration and Resolution

(a) We acknowledge that resolution significantly affects attribution. On the 341 Single-Doc questions with at most 25 pages, we increased Qwen3-VL-235B's input resolution from 1024² to 1536² and 2048², and also report Gemini-3.1-Pro's results on native PDF and on the same set of 1024² page images:

Model / InputSAARec.Rel.Ans.
Gemini-3.1-Pro, native PDF78.671.983.587.9
Gemini-3.1-Pro, 1024² page images54.353.163.087.9
Qwen3-VL-235B, 1024²28.715.945.377.7
Qwen3-VL-235B, 1536²49.035.063.482.9
Qwen3-VL-235B, 2048²63.145.774.082.5

SAA rises from 28.7 to 49.0 and then to 63.1, Rec. rises from 15.9 to 45.7, while Ans. rises only from 77.7 to 82.5. Resolution mainly improves in-page localization accuracy and has little effect on answer correctness. This also shows that the judgment in Appendix C.1 that further increasing the resolution "might yield marginal gains" does not hold—from 1024² to 2048², SAA improves by about 34 points; we will correct this statement in the revision.

(b) The main experiments stop at 1024² because of the context constraint. Qwen3-VL-235B has a native context of 256K. We estimate, for all 987 Single-Doc questions:

ResolutionTokens per page (approx.)Max pages within 256KSingle-Doc questions that fit
1024²~930~261987/987 (100%)
1536²~2,200~110923/987 (94%)
2048²~4,000~61793/987 (80%)

At 1536² only 94% of the questions fit, and at 2048² only 80%; the longest document, at 185 pages, requires about 737K tokens at 2048², far beyond 256K. Multi-Doc questions involve more pages in total, so the constraint is only tighter. Hence 1024² is the highest uniform resolution that "lets all documents enter the context". Note that the paper's use of "saturation point" refers to a trade-off under the context budget: within the 256K context, 1024² is the highest resolution that covers all documents, not a saturation point for attribution performance. Our configuration principle is: subject to covering the full benchmark, give each model the highest input fidelity its context allows; long-context or native-PDF models (e.g., Gemini) already use higher-fidelity inputs, and we have not artificially lowered their capability just to unify the input. Context length is itself part of model capability, and sustaining high-resolution input under a large context is a genuine advantage in real deployment.

(c) With the input controlled, the core conclusions are unchanged. To avoid conflating input differences with model capability, we re-evaluated Gemini-3.1-Pro on exactly the same set of 1024² page images used for the open-source models (same prompt, same rendering pipeline). On the 341 short-document questions, under the same 1024², Gemini-3.1-Pro scores 54.3 versus Qwen3-VL-235B's 28.7, a gap of about 26 points; on all 987 Single-Doc questions, the gap is 47.1 versus 25.0, about 22 points. The gap is mainly in attribution: on the 987 questions, Rec. is 44.1 versus 15.2 and Ans. is 86.3 versus 75.0. Even when Qwen is raised to 2048² (63.1), it remains below Gemini's native-PDF 78.6.

(d) Conclusion. Resolution affects absolute scores but does not change the conclusion that there is a significant gap between closed-source and open-source models, and that the gap lies mainly in evidence attribution rather than answer correctness; Attribution Hallucination exists under every configuration. We will add the same-input comparison and the resolution ablation to the revision, and explain the input configuration in the main text—the initial submission did not elaborate on this due to space limitations.


AuthorsOpenReview ↗

Response to Reviewer kxke #3

W4. Comparison with Existing Benchmarks

ViDoRe V3 and SciEGQA both provide box-level evidence, but their goals differ. ViDoRe V3 is a retrieval benchmark; its evidence boxes are freely drawn by annotators, and the authors themselves report an inter-annotator IoU of only about 0.50 and inconsistent granularity. SciEGQA is confined to 80 scientific papers and relies on a costly cross-annotation protocol that does not scale. Neither provides a sample-level metric that jointly scores answer correctness and sub-page citations on multi-page, multi-domain PDFs (Table 4: Gran. = B, Joint = ×).

CiteVQA differs in that: (i) its element-level, parser-standardized ground-truth boxes can scale automatically to arbitrary domains; (ii) SAA binds answer correctness and evidence attribution into a single sample-level metric, directly exposing the "correct answer, wrong citation" Attribution Hallucination—a failure mode that neither answer-only nor retrieval benchmarks can measure; and (iii) in the multi-page/multi-document setting, we can further decompose failures into "wrong page" and "wrong in-page localization" (Page-level Recall and Rec./Prec.), an analysis dimension absent from existing benchmarks. This also explains the phenomenon we observe: open-source models have reasonable Ans. but very low SAA.


W5. Figure Readability

We will redraw Figures 3 and 4 in the revision with larger fonts and adjusted legends, and check the readability of the remaining figures.


Finally, we thank the reviewer again for the careful evaluation. The supplementary experiments and statistics above will be fully presented in the revision, and we would be grateful for further comments.

Reviewer_K9W9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A benchmark and automated construction/evaluation pipeline for multimodal evidence attribution in multi-page, multi-document PDFs, with element-level bounding-box citations and metrics that assess both answer correctness and evidence grounding.

Critical strengths
  • evaluates element-level citations in multi-page PDFs using bounding boxes
  • covers multiple domains, question types, and single-/multi-document settings
  • measures answer correctness alongside evidence relevance and localization
Critical weaknesses

The paper motivates CiteVQA as a realistic document benchmark, but does not specifically evaluate whether its LLM relevance judge handles a common challenge in real documents: repeated or overlapping information across passages, pages, or documents. A citation may contain the correct fact yet omit qualifications or surrounding context, while an alternative passage outside the annotated ‘crucial evidence’ may also validly support the answer. Although the judge prompt considers sufficiency and context, the reported human-validation study does not establish its reliability on these cases. Targeted tests with redundant evidence and context-sensitive hard negatives would strengthen the claim that the evaluation reflects real-world attribution.

Because CiteVQA evaluates answer generation and evidence attribution jointly, attribution performance is not fully disentangled from answer quality and response scope. Models that provide shorter or less complete answers may face a lower citation burden than models that make more detailed claims, making attribution scores difficult to compare directly across systems. A complementary evaluation using fixed answers would help isolate attribution quality.

The benchmark’s data-quality claims should be further validated. Although the appendix reports a human audit of 200 examples with a high mean evidence-quality score, it does not report inter-rater agreement, or a quantitative audit of bounding-box fidelity. Key stages are automated and evaluated largely through model-based checks, while the document-selection criteria are insufficiently specified to assess selection bias.

Questions for authors

Because CiteVQA includes questions, source documents, and gold evidence regions, it could also support a post-hoc attribution setting. Adding a track in which systems receive a fixed, previously generated answer and must identify the supporting evidence, with attribution quality evaluated separately from answer correctness. This would help distinguish systems that generate answers with citations jointly from those that can reliably ground an answer after it has been written.

3 comments
AuthorsOpenReview ↗

Response to Reviewer K9W9 #1

We thank you for the careful reading, and for raising what we consider the three most critical issues for a benchmark of this kind: whether the relevance judge is reliable when evidence is redundant or incomplete; whether attribution ability can be decoupled from answer generation; and whether data quality (inter-rater agreement, bounding-box fidelity, and document selection) is sufficiently validated. For the first, we add new experiments; for the second, we commit to a new evaluation track; for the third, we add statistical analyses and clarifications.


W1. Reliability of the relevance judge under redundant/overlapping evidence

We agree that average agreement is not enough to establish that a citation judge is reliable: it must also reject unsupported citations while accepting legitimately redundant evidence. We therefore ran a controlled probe: the answer text is fixed to the gold answer and only the cited boxes are replaced, so any change in Rel. is attributable to the citations alone. The probe covers 60 Single-Doc questions (stratified over the four question types; every question carries supplemental evidence, i.e., regions that the masking ablation marked non-necessary), with 360 judge calls in total, all successful. The judge is the same as in the paper (Qwen3-VL-235B-A22B-Instruct).

ConditionCited contentMean Rel.Judged supportive (Rel ≥ 4)Judged unsupportive (Rel < 4)
goldall necessary evidence4.4391.7%8.3%
gold + auxnecessary + supplemental evidence (redundant)4.6295.0%5.0%
aux onlysupplemental evidence only1.4313.3%86.7%
truncatedkeep only the upper half of each necessary box2.9545.0%55.0%
neighbornearest non-evidence text block on the same page0.271.7%98.3%
other pagenecessary boxes moved to the same coordinates on an adjacent page0.080.0%100.0%

Conclusions.

  1. Valid redundancy is not falsely rejected. Adding supplemental evidence raises the mean score from 4.43 to 4.62 and the pass rate from 91.7% to 95.0%, indicating that the judge accepts valid evidence outside the crucial set.
  2. Evidence lacking context or completeness is penalized. Truncating each necessary box to half lowers the mean from 4.43 to 2.95 and the pass rate to 45.0%.
  3. Hard negatives are rejected. We construct two types of hard negatives: (i) the non-evidence text block nearest to the gold evidence on the same page—it shares the page and layout context with the answer but does not carry the answer's facts; (ii) moving the necessary evidence region to the same coordinates on an adjacent page, so that the cited content becomes a region on the wrong page. Their pass rates are only 1.7% and 0.0%, respectively, with mean Rel. below 0.3 in both cases. This shows that the judge does not simply reward superficial proximity, but checks the cited content itself.

W2 / Question: Fixed-answer (post-hoc attribution) track

We agree that joint scoring entangles attribution with answer scope, and that models giving shorter or less complete answers may bear a lower citation burden. We will add a post-hoc attribution track in the revision:

  • Fixed answer. Use the dataset's gold answers directly; all systems receive the same answer rather than generating their own.
  • Evidence only. Systems receive the document, the question, and the gold answer, and return only the supporting evidence; attribution is scored with the same box metrics and relevance judge as in the paper, independent of answer correctness.
  • Comparability. This separates "generating an answer with citations" from "grounding an answer that has already been written," and removes the citation-burden asymmetry you identified.

The full benchmark contains 1,897 questions; running all of them requires full inference for some models and is time-consuming. We will add complete results in the revision and update this thread with results for the main models before the rebuttal period ends whenever possible.


AuthorsOpenReview ↗

Response to Reviewer K9W9 #2

W3. Data-quality validation and document selection

(a) Human audit (already in Appendix B.3). We randomly sampled 200 examples, independently scored by 3 PhD-level experts using the same 5-point rubric as the models (Appendix E.1). The expert means are: question difficulty 2.97, answer quality 4.43, evidence quality 4.91 (Table 7).

(b) Inter-rater agreement. The 200 examples were independently scored by 3 experts; Table 7 reports the average across the three experts. The additional variability and agreement statistics are as follows (the LLM judge in the human-vs-LLM α is Qwen3-VL-235B-A22B):

MetricMean ± stdProportion ≥ 4Krippendorff's α (among 3 experts)α (human vs. LLM)
Question difficulty2.97 ± 0.74—0.740.72
Answer quality4.43 ± 0.4891.5%0.780.75
Evidence quality4.91 ± 0.2497.5%0.820.80

Agreement among the three experts is α = 0.74–0.82, in the acceptable-to-good range; human–LLM agreement is close (α = 0.72–0.80). Answer quality (std = 0.48) and evidence quality (std = 0.24, with 97.5% of examples ≥ 4) show low dispersion, indicating stable output quality from the automated pipeline. We will add these statistics in the revision.

(c) Bounding-box fidelity. Ground-truth citations are not hand-drawn rectangles, but layout elements parsed by MinerU (complete paragraphs, complete tables, figures, or notes), standardized at the element level (Appendix A). Parsing the same document twice yields nearly identical boundaries, so the ground-truth boxes are deterministic and reproducible, with no annotator-to-annotator variation in box drawing. On the other hand, manual box annotation is not necessarily reliable and is hard to scale to arbitrary domains: ViDoRe V3 (Loison et al., 2026) reports an inter-annotator IoU of about 0.50 for its free-form box annotations and notes that annotators differed in granularity. We therefore use a deterministic parser to define element boundaries; in a future version we will consider adding a human sampling audit (e.g., boundary error rate), but we will not use hand-drawn boxes as ground truth.

(d) Document selection. The main text gives the funnel (over 100 million raw PDFs → about 250k stratified candidates → 711), but the inclusion criteria are indeed under-specified. The actual process is an engineering multi-stage filter: candidates are first deduplicated, then annotated by an MLLM in two stages—coarse-grained primary domain and language, followed by fine-grained sub-category classification within each domain; we then perform balanced sampling over page count, language, sub-category distribution, and layout-element complexity, so that the selected documents cover, as evenly as possible, 7 domains, 30 sub-categories, languages, and layout complexity (both complex and simple documents). This is mainly engineering detail that the main text does not expand on; the revision will provide a complete description of the selection criteria.

We will also state the residual bias explicitly: the corpus comes from Common Crawl and reflects the publicly indexable PDFs it covers, excluding access-restricted or unindexed documents.


Finally, we thank you again for the careful review and constructive comments, which have helped us delineate the scope and validation boundaries of the benchmark more clearly. We will add the complete results of the fixed-answer attribution evaluation requested in W2 and the Questions in the revision, and will update the main-model results in this thread during the rebuttal period whenever possible. We would be grateful if you could follow up and continue to point out issues.

AuthorsOpenReview ↗

Summary of Our Responses and First Results for the Fixed-Answer Attribution Track

Thank you for your review. Beyond W2, a brief recap of the two points we have already addressed: W1 (judge reliability) — with the answer fixed and only the cited boxes replaced, 360 judge probes show that the nearest non-evidence block on the same page and the same-coordinate block on an adjacent page pass only 1.7% and 0.0% of the time, i.e., the judge checks the cited content itself rather than superficial proximity; W3 (data quality) — annotation quality and agreement both hold up: on the 200 samples independently re-scored by three experts, evidence quality averages 4.91/5 with 97.5% of samples at 4 or above, agreement among the experts is α = 0.74–0.82, and human–LLM agreement is close to the same level; ground-truth boxes are produced by a standardized parser rather than hand-drawn, so there is no annotator-to-annotator variation in box drawing; and the corpus's restriction to publicly indexable PDFs from Common Crawl is stated explicitly as a limitation.

Below are the first results for the W2 fixed-answer attribution track. All four models share exactly the same setting: the 987 Single-Doc questions (the same set of 1024² page images, the same prompt, with only the task changed to "the standard answer is given; return only the supporting <bbox /> tags"), temperature 0, and the paper's own deterministic box metrics (no LLM judge).

ModelnRec.Prec.F1PageRec.
Gemini-3.1-Pro98749.9144.3544.1266.61
Gemini-3-Flash98748.8336.9938.6069.46
Qwen3.5-397B-A17B98516.3416.3114.5166.44
Kimi-K2.59857.968.837.0950.09

Time constraints allowed only the 987 Single-Doc questions and these four models; Multi-Doc questions and additional models will be completed in the revision.

The conclusions are consistent with the paper: (i) the model ordering is unchanged, so joint scoring does not systematically favor any class of model; (ii) giving Gemini-3.1-Pro the answer raises all four metrics by only about 5 points (Rec. 44.1→49.9, Prec. 39.8→44.4, F1 39.1→44.1, PageRec. 60.3→66.6) and it still misses about half of the necessary evidence, showing that attribution is a bottleneck independent of answer generation; (iii) Qwen3.5-397B's PageRec. of 66.4 is nearly identical to Gemini-3.1-Pro's 66.6 while its Rec. is only 16.3 — precisely why the paper uses page-level and box-level metrics to separate wrong-page failures from wrong in-page localization.

This track, together with its setting and prompt changes, will be included in the revised paper.

The discussion period is drawing to a close; should it be convenient for you, we would be grateful to hear your further comments before it ends, and we hope to continue the discussion. Thank you again for your careful review, and for how much these comments have helped clarify the boundaries of our evaluation and the scope of our claims — whatever the outcome, this exchange has made the benchmark's presentation more rigorous. Best regards.

Comments on the paper

2 comments
Area_Chair_CmbBOpenReview ↗

Please Respond to the Authors’ Rebuttal

Dear Reviewers,

We are now roughly halfway through the Author–Reviewer Discussion period (which runs through 18 November). The authors have submitted their response and, in some cases, a revised manuscript.

Could you please take a moment to:

  • Read the authors’ rebuttal and any updated version of the paper;
  • Leave a reply to the authors on the points you raised, indicating whether your concerns have been addressed.

Even if your evaluation remains unchanged, a short reply confirming your final position is very helpful, as it allows me to track each reviewer’s stance and write an accurate meta-review.

If you are unable to participate further, please let me know as soon as possible.

Thank you for your time and careful reviewing.

Best regards,

Area Chair, ICLR 2027

AuthorsOpenReview ↗

Global Response

To the Reviewers and the Area Chair,

Thank you for your careful reviews during the discussion period. We understand how busy this period is for you, and we would be grateful if you could take a moment to look through this summary and give us your feedback. Over the past week we have replied to each point and added new experiments and statistics; for easy reference, we summarize the most important points below, with full details remaining in the individual responses.

  1. Resolution and input configuration: we evaluate each model at its strongest, and this setting will be stated in the paper. Our principle is to give every model the highest input fidelity its context allows: context length is itself part of a model's capability, and we do not artificially degrade strong models for the sake of formal uniformity. Appendix C.1 describes 1024² as a "saturation point" and states that higher resolution "might yield marginal gains"; both statements are inaccurate, and the revision will correct them — the "marginal gains" claim will be removed, and "saturation point" will be replaced by "the uniform operating point at which the entire benchmark fits within the context budget (for Qwen and other open-source model families)". The data support this correction: raising resolution substantially improves attribution — for Qwen3-VL-235B, SAA rises from 28.7 at 1024² to 49.0 at 1536² and 63.1 at 2048² — showing that fixing all models at 1024² systematically underestimates the attribution ability of strong models. Under exactly the same 1024² input, Gemini-3.1-Pro scores 47.1 against Qwen3-VL-235B's 25.0, with the gap still mainly in attribution, showing that the gap between open- and closed-source models is not entirely due to input fidelity.

  2. Human validation and judge reliability. On the 200 samples independently re-scored by three experts, evidence quality averages 4.91/5, 97.5% of samples score at least 4, and inter-expert agreement is α = 0.74–0.82; binarizing human and automatic judge scores at threshold 4 and comparing them sample by sample gives agreement of 87.5%–94.5%, consistent with the mean differences and the Friedman test. A 360-call probe with the answer fixed and only the cited boxes replaced further shows: necessary plus auxiliary evidence 95.0%, auxiliary evidence alone 13.3%, half-truncated evidence 45.0%, the nearest non-evidence block on the same page 1.7%, and the same-coordinate block on an adjacent page 0.0%.

  3. SAA's pass condition holds up. Splitting all passes of the 20 models by path, only 4.4% pass on Rec. ≥ 0.6 alone (Rel. < 4); raising the Rec. threshold from 0.6 to 1.0 and recomputing changes the number of passes by only 0.14%. Among SAA = 0 failures, boundary imprecision accounts for only 1.0%–5.0%, while wrong page and wrong element together account for 34.8%–51.7%.

  4. New fixed-answer post-hoc attribution track. The standard answer is given and the model only returns the supporting boxes: 987 Single-Doc questions, 4 models, temperature 0, deterministic box metrics only. Gemini-3.1-Pro's Rec. rises from 44.1 in the joint setting to 49.9, and it still misses about half of the necessary evidence, while the model ordering is unchanged — attribution is a bottleneck independent of answer generation.

In addition, the positioning relative to existing benchmarks (retrieval-oriented vs. joint scoring of answer correctness and sub-page citations), the redrawn Figures 3 and 4, the document-filtering criteria, and the input-configuration description will all be completed in the revision. All of the above experiments, statistics, and corrections will be incorporated into the revision.

The discussion period ends on November 18 at 23:59 AoE (Anywhere on Earth). Should it be convenient for you, we would be grateful for any further comments before then, or for a confirmation of our responses. Thank you again for your reviews — these comments have made our statements about the boundaries of the evaluation and the scope of our claims considerably more rigorous.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.