- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
CiteVQA evaluates document question answering together with element-level bounding-box citations. It contains 1,897 questions over 711 PDFs across seven domains and two languages. An automated pipeline constructs questions and evidence annotations, using masking ablations to identify crucial evidence. The paper introduces Strict Attributed Accuracy (SAA), evaluates 20 multimodal models, and finds that correct answers often come with incorrect citations. Additional analyses include human audits, input-resolution ablations, and training-based validation of the annotation pipeline.
Critical strengths
- The task addresses a useful gap in document evaluation. Requiring a model to identify the specific passage, table, or figure supporting its answer makes failures visible that answer-only evaluation misses. The inclusion of long documents and multi-document questions makes this relevant to practical document assistants.
- The paper makes a substantial effort to validate the resource. Beyond evaluating 20 models, the authors provide a 200-instance annotation audit and an auxiliary training comparison against manually annotated data (Appendices B.3–B.4). These results offer useful evidence that the automated pipeline produces meaningful supervision.
Critical weaknesses
- SAA does not reliably enforce the attribution requirement claimed by the paper. In Section 4.1, an answer passes whenever Ans. ≥ 4 and Rec. ≥ 0.6, even if its relevance score is low. For example, citing two of three crucial elements would suffice despite omitting evidence explicitly labeled necessary. Adding incorrect citations also does not reduce this recall. Reporting precision separately does not resolve this weakness in the headline metric. Moreover, the judge validation in Appendix C.2 reports similar mean scores and nonsignificant Friedman tests, which do not establish agreement on individual pass/fail decisions at the threshold of 4. For a benchmark centered on trustworthy attribution, I would need stronger evidence that SAA accepts supported answers and rejects unsupported ones. A breakdown by acceptance path and threshold-level agreement with the existing human judgments would help.
- The model comparisons confound attribution capability with input quality. Table 9 gives Gemini native PDF input, GPT full-resolution page images, and most open models downscaled images. Table 10 shows that this matters substantially: reducing Qwen3-VL-235B’s pixel budget by half lowers SAA from 22.5 to 11.8. The authors acknowledge this limitation, but the main discussion still draws strong conclusions about the grounding reliability of different model tiers. The claim that 1024 × 1024 is a saturation point is also unsupported by an ablation that only tests lower resolutions. A comparison using the same rendered inputs on a feasible subset would help separate these effects. Otherwise, the conclusions should be limited to the evaluated model-and-input configurations.
Questions for authors
- Metric clarification: For each model, how many SAA-positive predictions pass only through Rec. ≥ 0.6 while Rel. < 4? Do the existing human annotations indicate whether these citations adequately support the answers?
- Judge validation: On the 200 examples in Appendix C.2, what is the sample-level agreement between human and automated judgments for Ans. ≥ 4 and Rel. ≥ 4? Similar average scores could conceal disagreements that change SAA.
- Input comparability: Were any comparisons conducted using identical rendered pages and resolutions across model families? If so, do the main ranking and performance gaps persist? I am asking about existing controls rather than requesting a new full-benchmark evaluation.
4 comments
Response to Reviewer not6 #2
W2 + Q3. Input Comparability and the "Saturation Point"
(1) The "might yield marginal gains" claim in the paper is incorrect, and the new upward ablation confirms this directly. The original sentence in Appendix C.1 reads: "Although further increasing the resolution might yield marginal gains, 1024×1024 represents a critical 'saturation point'...". Our intended meaning was that, within a 256K context, 1024² is the highest uniform resolution that covers all documents; however, the "marginal gains" claim is inconsistent with the experiments—the upward ablation below shows that SAA improves substantially from 1024² to 2048². The revision will remove "marginal gains" and clarify "saturation point" as "the uniform upper limit under the context budget", to avoid confusion with performance saturation.
(2) The upward ablation and the same-input comparison show that 1024² is not a performance saturation point. On the 341 Single-Doc questions with at most 25 pages:
| Model / Input | SAA | Rec. | Rel. | Ans. |
|---|---|---|---|---|
| Gemini-3.1-Pro, native PDF | 78.6 | 71.9 | 83.5 | 87.9 |
| Gemini-3.1-Pro, 1024² page images | 54.3 | 53.1 | 63.0 | 87.9 |
| Qwen3-VL-235B, 1024² | 28.7 | 15.9 | 45.3 | 77.7 |
| Qwen3-VL-235B, 1536² | 49.0 | 35.0 | 63.4 | 82.9 |
| Qwen3-VL-235B, 2048² | 63.1 | 45.7 | 74.0 | 82.5 |
Qwen's SAA rises from 28.7 to 49.0 and then to 63.1, and Rec. rises from 15.9 to 45.7, while Ans. rises only from 77.7 to 82.5: resolution mainly improves in-page localization and has little effect on answer correctness. The downward ablation in Table 10 (22.5→11.8) points in the same direction; together they show that 1024² is merely an operating point under the context constraint, not a performance ceiling. On the same 1024² page images, Gemini-3.1-Pro scores 54.3 and Qwen 28.7; even when Qwen is raised to 2048² (63.1), it remains below Gemini's 78.6 on native PDF.
Under the same input, both models' Ans. remains clearly higher than SAA, and Attribution Hallucination persists; the gap between closed-source and open-source models still lies mainly in attribution, rather than being caused by input differences.
(3) The choice of 1024² is a consequence of the context constraint. Qwen3-VL-235B has a native context of 256K. Using the per-page token counts measured on the 341 questions, we extrapolate to all 987 Single-Doc questions:
| Resolution | Tokens per page (approx.) | Max pages within 256K | Questions that fit |
|---|---|---|---|
| 1024² | ~930 | ~261 | 987/987 (100%) |
| 1536² | ~2,200 | ~110 | 923/987 (94%) |
| 2048² | ~4,000 | ~61 | 793/987 (80%) |
The longest 185-page document requires about 737K tokens at 2048², far beyond 256K; Multi-Doc questions involve more pages in total, so the constraint is only tighter. A uniform evaluation can therefore stop only at 1024²: this is the highest resolution that covers the entire benchmark, not a saturation point for attribution performance.
(4) Input configuration and the scope of our conclusions. Most models in Table 9 (the 256K row: Qwen3.5, Qwen3-VL, Gemma-4, Kimi-K2.5, Seed-2.0-Pro) already use the same set of 1024² page images, the same rendering pipeline, and the same prompt; GLM-5V-Turbo uses a compact 768² input; only GPT-5.4/GPT-5.2/Qwen3.6-Plus use full-resolution page images, and Gemini uses native PDF. The same-input comparison is already given in (2): the relative ordering of the two models is unchanged, and the gap lies mainly in attribution. We agree that absolute scores under different input configurations are not directly comparable. The revision will state the input configuration of each model class in the main text, and will also make explicit the setting of "under the highest-fidelity input available to each model": subject to covering the full benchmark, each model is given the highest input fidelity its context allows, without artificially lowering long-context or native-PDF models just to unify the input—context length is itself part of model capability, and sustaining high-resolution input under a large context is a genuine advantage in real deployment.
Response to Reviewer not6 #3
Q2. Per-Sample Agreement Between Human and Automatic Judges at Threshold 4
The table below reports the threshold-decision agreement rate on the 200 samples of Appendix C.2. Specifically: for each sample, the human score and the automatic judge score are each binarized at threshold 4 (≥ 4 = pass, < 4 = fail); agreement scores 1 and disagreement scores 0; the values are the proportion of the 200 samples for that reasoning model on which the two decisions agree.
| Reasoning model | Threshold | Human vs Qwen judge | Human vs Gemini judge |
|---|---|---|---|
| GPT-5.4 | Rel. ≥ 4 | 87.5% | 89.0% |
| GPT-5.4 | Ans. ≥ 4 | 93.0% | 92.5% |
| Gemini-3.1-Pro | Rel. ≥ 4 | 91.0% | 92.0% |
| Gemini-3.1-Pro | Ans. ≥ 4 | 94.5% | 93.5% |
This is consistent with the mean differences in Table 11 (0.02–0.16) and the Friedman tests (p = 0.14–0.53): there is no systematic bias between the automatic judges and the human experts.
Finally, we thank you for the careful review. The supplementary experiments and statistics above will be fully presented in the revision, and we would be grateful for further comments.
Summary of Our Responses to Reviewer not6
Thank you for your review. For your convenience, here is a summary of the main points of our earlier replies.
- W1 + Q1 (SAA's pass condition and the rec_only samples). Rec. ≥ 0.6 is in fact equivalent to "cite everything" on the vast majority of questions — necessary evidence typically consists of only one or two elements, and citing one of two yields Rec. = 0.5, below the threshold. We also tested the effect of partial citations directly: recomputing all 20 models with the Rec. threshold raised from 0.6 to 1.0 changes the number of passes by only 0.14%, so this case is negligible for SAA. As for extra citations not lowering Rec., this is intentional: SAA passes when "the answer is correct and the necessary evidence is cited", not when "the cited set equals the minimal evidence set"; redundancy is reported separately by Precision/F1, and having the headline metric penalize it as well would conflate "citing additional correct evidence" with "citing wrong evidence". Conversely, the gap between SAA and Ans. shows that what SAA rejects is precisely the "correct answer, wrong citation" case — in the failure decomposition, loose box boundaries account for only a small fraction, while citing the wrong page or element dominates.
- W2 + Q3 (input comparability and the "saturation point"). The original "might yield marginal gains" statement is wrong; the revision will remove it and define "saturation point" explicitly as "the uniform ceiling under the context budget", so that it is not confused with performance saturation. The upward ablation shows that 1024² is not a performance ceiling: raising resolution mainly improves in-page localization and affects answer correctness little. Stopping the uniform evaluation at 1024² is a context constraint — at higher resolutions a considerable fraction of long documents no longer fits, and 1024² is the highest uniform setting that covers the entire benchmark. With exactly the same 1024² input as the open-source models, the closed- vs. open-source gap persists and still lies mainly in attribution rather than answer correctness; Attribution Hallucination is present under every configuration. The revision will state each model class's input configuration in the main text, together with the principle of "the highest-fidelity input available to each model".
- Q2 (judge threshold agreement). Binarizing the human and automatic judge scores at threshold 4 and comparing them sample by sample, agreement exceeds 87% throughout, consistent with the mean differences and the Friedman test: there is no systematic bias between the automatic judge and the human experts.
These additional experiments and statistics will be included in the revised paper. The discussion period is drawing to a close; should it be convenient for you, we would be grateful to hear your further comments before it ends, and we hope to continue the discussion. Thank you again for your careful review. Best regards.
Response to Reviewer not6 #1
We thank you for the review. The two main weaknesses you raise (SAA's pass condition and input comparability) and the three specific questions (rec_only samples, judge threshold agreement, and the same-input comparison) all bear on what is essential to benchmarking. We answer W1 together with Q1 and W2 together with Q3, and answer Q2 separately.
W1 + Q1. SAA's Pass Condition and the rec_only Samples
(1) The distribution of necessary evidence means that Rec. ≥ 0.6 is equivalent to "citing everything" on most questions. We counted the number of necessary evidence elements over the full evaluation set:
86.2% of questions have only 1–2 necessary elements. For these questions, Rec. ≥ 0.6 effectively requires citing all of them: 1/1 or 2/2; citing 1 of 2 gives only 0.5, which fails. The case you raise—citing 2 of 3 (2/3 = 0.667)—applies only to the 10.1% of questions that have exactly 3 necessary elements; for 4 or more (3.6%), a threshold of 0.6 requires at least 3/4, 3/5, and so on.
(2) Raising the Rec threshold to 1.0 leaves SAA almost unchanged. To test directly the effect of "2/3 is enough to pass", we used the paper's existing per-question results to raise the Rec branch of SAA from 0.6 to 1.0 (with all other conditions unchanged) and recomputed over the 20 models: the number of passes drops from 9,929 to 9,915, a decrease of only 14 (0.14%). We will add this re-computation to the revision's appendix as a sensitivity analysis.
(3) Per-model share of rec_only. The table below decomposes the passing samples of the 20 models by pass path into both, rel_only, and rec_only:
Of the 435 rec_only samples, 421 (96.8%) already have Rec = 1.0; only 14 (3.2%) are partial citations with 0.6 ≤ Rec < 1.0. These samples have Rec of either 0.667 or 0.8 (i.e., 2/3 or 4/5)—exactly the case you describe; all of them satisfy Ans. ≥ 4, so the answer itself is correct. They account for 0.14% of all passes. Per model, only Gemini-2.5-Pro exceeds 10% (97/892 = 10.9%); the remaining 15 models range from 0.7% (Qwen3-VL-8B) to 6.8% (GLM-5V-Turbo).
(4) Additional citations do not lower Rec., and this is by design: what matters is that the necessary evidence is cited. You are right that Rec. does not decrease when extra citations are added. This is not a flaw in the metric but a deliberate difference between CiteVQA and retrieval-style metrics. SAA's pass condition is "the answer is correct and the necessary evidence is cited", not "the cited set must equal the minimal evidence set". The reasons are as follows: B_gt is split into B_crucial and B_other (Section 4.1), B_other is legitimate evidence, and a model should not be penalized for citing it; the same answer admits multiple possible evidence granularities (whole paragraph, sentence, table cell), and requiring a unique minimal set would mark reasonable citations as wrong. Redundant citations are reported separately by Precision/F1 (Table 12); SAA itself does not take on the responsibility of judging whether citations are minimal. If the headline metric also penalized redundancy, it would conflate "citing additional correct evidence" with "citing wrong evidence", weakening the metric's focus on attribution errors.
(5) On the rejection side, the gap between SAA and Ans. shows that SAA rejects many samples whose answers are correct but whose citations fall short. For example, Qwen3-VL-235B has a Single-Doc Ans. of 75.0 but an SAA of only 25.0; Gemini-3.1-Pro has an Ans. of 86.7 and an SAA of 76.0. The paper's decomposition of SAA = 0 samples (Figure 5, Appendix C.6) further shows that, among all failures, boundary imprecision accounts for only 1.0%–5.0%, whereas wrong-page and wrong-element citations together account for 34.8%–51.7%; among answer-correct failures, the dominant mode is genuinely citing the wrong evidence, not imprecise box drawing.