acceptodds
Under review as a conference paper at ICLR 2027

DocComp: Query-Conditioned Semantic Component Selection for Efficient Document VQA

Abstract

Document visual question answering (DocVQA) requires high-resolution inputs to read small text and structured content, but full-page processing turns even question-irrelevant regions into visual tokens. Existing visual-token pruning reduces the sequence only after visual processing, whereas aggressive region selection can lose answer-supporting context, and both can be tied to a specific VLM. To address this, we propose DocComp, a plug-and-play framework for query-conditioned semantic component selection that keeps only question-relevant layout regions before downstream VLM processing. DocComp decomposes a page into semantic layout components and ranks them against the question with a late-interaction visual retriever adapted from page-level to within-page retrieval. Retrieval confidence then sets a question-specific evidence budget in place of a fixed Top-k. Finally, a reconstruction layer serializes the selected evidence for the downstream image processor, giving compatibility across VLM families while the detector and generative VLM remain frozen. Component-level adaptation raises within-page R@1 from 37.78% for the base ColSmol page-level retriever to 92.56%. Across nine frozen backbones, DocComp reduces visual tokens by 62.34–63.26% for Qwen with ANLS changes of −1.9 to +0.8 points, and by 33.20% for InternVL with changes of −4.4 to +3.0 points, improving three of four models. At similar reductions, it exceeds VScan and VisionZip by 14.14 and 16.92 ANLS points, and, counting its own detection and retrieval, cuts page-amortized compute by 61.04% on the reliable subset. These results support pre-VLM semantic evidence selection as an effective alternative to processing the full page first and pruning afterward.

Primary area:
foundation or frontier models, including LLMs
Keywords:
document visual question answering, vision-language models, inference efficiency, visual token reduction, visual document retrieval, late interaction, document layout analysis, query-conditioned evidence selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DhgXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes DocComp, a framework that reduces the computational cost of document visual question answering through query-conditioned semantic component selection. It first uses a frozen layout detector to generate candidate regions, then adapts ColSmol from page-level retrieval to within-page component ranking using answer-location supervision, same-page negatives, and text-side LoRA. The framework adaptively selects the number of components based on retrieval score gaps and reconstructs the selected evidence for different image processors before passing it to frozen downstream vision-language models.The authors evaluate answer quality and visual-token usage on 41,180 answer-valid BoundingDocs examples across nine Qwen and InternVL models. Retrieval, ablation, and computational analyses additionally use a subset of 35,349 examples with reliable spatial annotations. The results show substantial improvements in within-page retrieval and reduced visual-token usage, with comparable or improved overall ANLS for some models. However, answer quality varies across models and document sources, and practical latency improvements depend on reusing page-level preprocessing.

Critical strengths

S1. Component-level adaptation delivers substantial retrieval gains supported by statistical analysis. On identical candidate regions and positive assignments, DocComp improves within-page R@1 from 37.78% for the base ColSmol retriever to 92.56%, with paired document-cluster bootstrap confidence intervals supporting the improvement. The fixed-Top-3 downstream experiment also improves ANLS from 0.5898 to 0.8179. These results provide substantive evidence for adapting page-level retrievers specifically to within-page component ranking. Although the downstream comparison does not use identical visual-token budgets, this does not diminish the significance of the retrieval gains measured on the same candidate sets.

S2. Processor-aware reconstruction is practically valuable and evaluated across multiple models. The paper considers not only which regions to select, but also how different image processors convert them into visual tokens, including avoiding the enlargement of small crops and reducing redundant overhead in fixed-tile processing. A single retriever serves nine frozen downstream models. For Qwen, the method reduces visual tokens by approximately 62–63%, with overall ANLS changes ranging from −1.90 to +0.84 points. The InternVL reconstruction comparisons further demonstrate the importance of preserving region scale for answer quality. These results support the practical value of the framework as a reusable front end.

Critical weaknesses

W1. The incremental contribution over the closest region-selection approaches is not sufficiently established empirically. The related-work section discusses PinPoint and RegionRAG, but the main compression comparison includes only VScan and VisionZip. Since the central contribution is the combination of component supervision, region selection, and input reconstruction, these comparisons do not sufficiently establish its incremental value over existing region-selection approaches. The experiments convincingly demonstrate an advantage over the evaluated post-hoc pruning configurations, but they do not establish an advantage over the closest region-based alternatives. A limited comparison with clearly specified task and resource conditions, or a substantive explanation of why direct comparison is inappropriate, would considerably strengthen the contribution’s positioning.

W2. Essential experimental protocols and practical efficiency gains are not sufficiently independently verifiable. The reliable subset has 100% candidate coverage, but the paper does not fully specify its filtering procedure, the conversion of multipage documents into QA–page examples, the dataset revision, or document-level split checks explicitly covering all 41,180 evaluation examples. Essential training, detection, and latency-measurement configurations are also incomplete. Moreover, uncached execution is 4.75% slower than Full-page, so practical speedups depend on caching or page-level amortization. It remains unclear whether Full-page uses equivalent opportunities to reuse cacheable visual features. These gaps directly affect the scope of the retrieval results, reproducibility, and the assessment of the efficiency contribution. They do not establish data leakage or an unfair comparison, but they require clarification.

Questions for authors

1.What is the key incremental value of DocComp relative to PinPoint and RegionRAG? Could you provide a limited region-selection comparison with clearly specified training resources, downstream generators, and input budgets, or explain which task or model constraints make a direct comparison inappropriate? This would affect my assessment of the contribution’s significance. 2.Please specify the executable rules and exclusion counts by category for constructing the 41,180 answer-valid examples and the 35,349-example reliable subset. Which dataset revision was used? How were multipage documents converted into QA–page examples, and were input pages selected using answer locations? Is the entire evaluation set document-disjoint from the training and validation sets? These details would affect my assessment of the evaluation scope and protocol reliability. 3.Please report the hardware, batch size, numerical precision, timing boundaries, absolute latency, and a breakdown of detection, component encoding, retrieval, and VLM inference time. Does Full-page reuse cacheable visual features? Does the page-amortized latency advantage persist when both methods operate under consistent reuse conditions? This would affect my assessment of the practical efficiency contribution. 4.Please provide the LoRA, optimizer, training-duration, detector, and context-margin configurations, and clarify whether positive-assignment rules are identical during training and evaluation. Using the existing predictions, could you distinguish failures to retain an answer-bearing component from incorrect answers despite retaining such a component, and provide diagnostics indicating whether the latter involve missing context or reconstruction issues? This would help explain the relationship between high answer-location retention and final QA quality.

2 comments
AuthorsOpenReview ↗

Response to Reviewer DhgX — Part 1/2

We thank the reviewer. Corrections are in the general response: GT-free re-evaluation, disjointness and latency.

W1 / Q1. Incremental value over PinPoint and RegionRAG.

All methods were run on all 41,180 known-page QA, with paired document-cluster bootstrap (5,000 resamples).

BackboneFull-pageBaselineDocCompDocComp − BaselineVisual tokens (F / B / D)
Qwen2.5-VL-7B, RegionRAG0.81640.81310.8229+0.0098 [+0.0051, +0.0149]1,091 / 1,448 / 408
Qwen2-VL-7B, PinPoint0.80170.63620.7622+0.126 [+0.111, +0.141]1,651 / 1,017 / 635
LLaVA-NeXT-7B, PinPoint0.53230.49870.5474+0.049 [+0.032, +0.063]2,360 / 1,768 / 1,533
  • Resources.
    • RegionRAG: official RegionRet adapter, Top-4. PinPoint: the authors' checkpoints.
    • Both are applied zero-shot. DocComp's retriever (500M, 17.4M LoRA parameters) is trained in-domain.
    • Within each backbone, all methods use the same frozen generator weights, prompt and decoding. For RegionRAG, Full-page and DocComp were re-generated under the exact generator configuration of the RegionRAG run.
  • Limits.
    • On Qwen2-VL, the PinPoint gain is +0.050 without DeepForm, and DocComp's EM is −0.038 lower.
    • Under a low page budget (max 262,144 pixels), DocComp is −0.061 [−0.086, −0.038] below RegionRAG.
  • Selection footprint (1,000 QA). RegionRAG's regions cover 98.71% of the page area, vs 31.62% for DocComp. All answer words are kept (each ≥ 80% covered) in 99.60% vs 90.70% of cases.
  • Incremental value. DocComp uses the fewest visual tokens of the three region methods. On Qwen2.5-VL-7B it also matches or exceeds Full-page and RegionRAG. We will add both comparisons.

W2 / Q2. Data protocol.

  • Dataset. letxbe/BoundingDocs snapshot 47db6d2b, official splits.
    • Training: 479,947 train instances (one per annotated answer occurrence; 193,033 questions) and 60,278 validation instances.
    • Evaluation: test only.
  • QA-page examples. 65,444 test rows are grouped by (doc_id, question_id, page) into 41,180 examples.
    • These come from 25,463 questions and 4,824 documents.
    • A question answered on several pages gives one example per page.
  • Page selection. Yes, the input is the annotated answer page for every method. We added a whole-document run:
Pages12–34–10>10
Whole document0.8010.9030.8250.614
Answer page given0.8010.8790.8550.829
  • Reliable subset. It is used to evaluate the retriever (R@1, answer-region retention), which needs exactly one well-defined positive component. End-to-end accuracy is evaluated on all 41,180 QA. A QA-page is kept if any of its rows is classified reliable. Replaying the rule reproduces the 35,349 IDs exactly.
Rule (in order)ExcludedRemaining
No fatal quality issue (missing geometry/OCR, OCR–box mismatch, no overlapping detector candidate, weak match)3,33937,841
Exactly one plausible positive component44137,400
Answer tokens not split across components42936,971
Answer box not crossing a component boundary42936,542
Match score ≥ 0.551,19335,349

The rule uses detector candidates, so the 100% coverage is partly by construction. Coverage on all 41,180 QA is 98.16%.

  • Disjointness. An ID and image-hash check found 3 test documents (44 QA) that also appear in the training split. There is no question-ID or validation overlap.
    • We remove these documents, so evaluation is document-disjoint.
    • On the remaining 41,136 QA, every ANLS changes by at most 0.0003.
    • DocComp − Full stays +0.0066 [+0.0016, +0.0119].
AuthorsOpenReview ↗

Response to Reviewer DhgX — Part 2/2

W2 / Q3. Latency.

  • Setup. One H200, batch 1, BF16, SDPA, greedy decoding (64 tokens), CUDA-synchronised timers. Fixed 500 reliable QA, 10 warm-up queries.
  • Breakdown (ms).
    • Full-page 1,068.8 = loading 410.9 + processor 228.8 + vision 231.5 + prefill/decoding 197.6.
    • DocComp 1,119.6 = component encoding and ranking 590.8 + detection + VLM path 410.6.
    • Detection: 55.1 ms/page.
  • Does Full-page reuse cacheable features? Not in the paper. We now cache Full-page visual embeddings per page and measure on the same protocol. Inputs are token-identical to the uncached run.
ms per QAFull-pageDocComp
No reuse1,068.81,119.6
Page-level reuse, DocComp only (paper)1,068.8755.3
Page-level reuse, both (new)625.7 [571, 686]755.3
  • Does the advantage persist under consistent reuse? No. We remove the latency claim.
  • What remains:
    • decoder visual tokens −63%;
    • VLM stage −45% (642.2 vs 354.0 ms);
    • analytical FLOPs −51% cold and −46% to −49% with equal reuse.

W2 / Q4. Configuration, positive rules, failures.

  • Retriever.

    • ColSmolVLM-500M (650243e9) with text-side LoRA on q/k/v/o/gate/up/down: r 32, α 32, dropout 0.05.
    • Multi-positive InfoNCE (τ 1.0), AdamW with linear warmup and decay, BF16. Batch of 48 queries, ≤ 32 negatives per query.
    • 2 epochs: epoch 1 at lr 2e-4 (10,121 steps); epoch 2 at lr 5e-5. One H200.
  • Detector. DocLayout-YOLO (YOLOv10m, D4LA), frozen. Input 1024², confidence 0.01, NMS IoU 0.45.

  • Selection. Top-1 if gap(1,2) ≥ 4.0; else Top-3 if gap(3,4) ≥ 1.5; else Top-5.

  • Context margin. 24 px.

  • Positive rules. They differ between training and evaluation:

    • training uses at most one positive: a component covering ≥ 0.8 of the answer anchor and containing its centre;
    • evaluation accepts every component covering ≥ 0.8 of any answer box.

    Under the stricter training rule, reliable retention is 99.50% (99.53% under the evaluation rule).

  • Failures (reliable subset, exact match, 9,071 errors).

CaseQA
Answer-bearing component not retained134 (1.5%)
Retained, but wrong8,937 (98.5%)
— of which Full-page is correct1,921

We re-ran the 1,921 cases with one change each, keeping the same selection.

ChangeExact match
Unchanged0.027
2× crop resolution0.645
96 px context0.523
Top-50.425

Most errors come from resolution and local context, not from missing regions.

Reviewer_ZEb4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DocComp, a plug-and-play efficiency framework for reducing the number of visual tokens processed by VLMs for DocVQA. Layout regions detected using DocLayout-YOLO are ranked by a question/query-conditioned retriever model. An adaptive Top-k policy selects the regions to be passed to a frozen VLM for obtaining the final answer while having a lower token count consumption compared to default full-document setup. Experiments with Qwen and InternVL model family variants show substantial reductions in visual token counts. However, the overall effect of the token pruning strategy on answer accuracy varies across models and datasets.

Critical strengths

The idea of determining candidate regions where the answer is likely to be present is quite reasonable. It can actually make a big difference when the answer is highly localized to a small region.

The proposed method does not require modifying the downstream VLM. This represents savings in training budget as well.

Critical weaknesses
  1. The method implicitly assumes that the evidence required to answer a question is localized to a small number of layout components. But the method does not seem suitable for abstractive questions, or questions requiring reasoning across multiple regions of the document. For example, a question like "What are the main reasons for the decline in sales?" may require information from multiple tables and paragraphs in the document. Similarly, a question like "How did the ocean temperatures change compared to last year?" may require looking at several sections of the document. In such cases, selecting only 1, 3 or 5 regions can discard information essential to answering the question. Once these regions are removed by ranking mechanism, the downstream VLM has no access to them. Basically, the method also has no mechanism to determine whether a question requires localized or global evidence. This is a fundamental limitation of the proposed formulation, not merely an implementation issue.

Related to above, the paper does not provide a question-type-wise evaluation to assess how the method performs on abstractive or distributed evidence questions.

  1. Sec. 3.2 defines positive components based on overlap with annotated answer locations. Components that do not overlap the answer are used as negatives. But there may still be some use for these so-called negative regions which may provide some secondary evidence for answering effectively. Reusing the same example from above - "What are the main reasons for the decline in sales?" , the explanation may occur in one paragraph, but supporting evidence could be present in tables elsewhere in the document. Such regions could be treated as negatives simply because they do not contain the answer text.

  2. In the current formulation, the retriever is optimized for answer localization. It does not necessarily identify all the regions which could potentially help answer the question. This is also a fundamental issue. The fact that BoundingDocs annotations are based on answer strings and not on evidence regions leaves no scope for enabling a retriever to use such information.

The formulation does not seem general and robust enough to support DocVQA in a general sense.

Questions for authors
  1. How does the method handle abstractive questions requiring information from multiple regions, or potentially the entire document? Is there any analysis of performance by question type (extractive, comparative, arithmetic, abstractive, etc.)?

  2. Why is answer-location supervision appropriate for training an evidence retriever? In particular, how are regions containing useful supporting evidence, but not the answer text, handled during training?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZEb4 — Part 1/2

We thank the reviewer. We agree that the adaptive Top-1/3/5 rule reflects retriever confidence. It does not decide whether a question needs localized or global evidence, and we will state this.

W1 / Q1. Question types, multi-region and whole-document questions.

Question types.

  • On BoundingDocs-1000 (rule labels), 986 of 1,000 QA are single-value or row/column-conditioned lookups. Full-page scores .827 / .883 and DocComp .826 / .934. The other 14 carry only a wording cue, since BoundingDocs answers must match the page OCR. We therefore also ran TAT-DQA (financial reports) with the unchanged pipeline.
  • TAT-DQA uses official question types, with arithmetic sub-types taken from the gold derivation.
TAT-DQA question typenFullDocCompΔ [95% CI]
Extractive: single value691.767.643−.124 [−.156, −.092]
Extractive: multi-part209.474.383−.091 [−.144, −.041]
Comparative: difference276.406.366−.040 [−.090, +.011]
Comparative: % change181.055.088+.033 [.000, +.067]
Other arithmetic240.175.158−.017 [−.052, +.017]
All1,637.500.431−.070 [−.088, −.052]
  • Comparative and arithmetic questions show no significant loss. The operands are retained in ≥ 95% of cases.
  • The out-of-domain loss comes from retrieval misses. On single-block TAT-DQA questions, DocComp is within −.004 of Full-page when the evidence block is retained (552 QA) and drops by −.586 when it is missed (128 QA); retention is post hoc, measured with GT boxes.
  • Free-form abstractive questions remain untested: neither benchmark contains them.
  • On all 41,180 QA, natural-language questions are −0.025 [−0.044, −0.015] vs Full-page. Form fields are +0.016 [+0.011, +0.021].

Evidence spread over several regions. This is the reviewer's main concern. Multi-part TAT-DQA questions lose −.091. The operands of arithmetic questions usually lie in one table (see W2/Q2), so evidence that is separated across tables and paragraphs remains untested. We list it as a limitation.

Where DocComp fails. DocComp fails on 9,071 reliable QA.

  • In 98.5% of them, the answer region was retained.
  • For the 1,921 failures that Full-page answers correctly, the same selection recovers .645 with 2× resolution, .523 with a 96 px context and .425 with Top-5.
  • A wider context is not a free fix. It helps natural questions (+.037) but hurts form fields (−.057).

Whole-document questions. DocComp received each question's whole document (25,463 questions).

Pages12–34–10>10
Whole document.801.903.825.614
Answer page given.801.879.855.829

DocComp is not designed for long-document reasoning. We report this as a limitation.

AuthorsOpenReview ↗

Response to Reviewer ZEb4 — Part 2/2

W2, W3 / Q2. Answer-location supervision and supporting regions.

  • Why we used it. Evidence-region labels are rare. Answer locations exist at BoundingDocs scale (479,947 training instances). The resulting signal still finds non-answer evidence (below).

  • Agreed: answer location is a localization proxy, not evidence supervision.

    • Each training instance has one positive: the tightest component covering ≥ 80% of the answer box.
    • Up to 32 same-page components are negatives.
  • Supporting regions can be penalized.

    • In 48.5% of matched form-field labels, the label lies inside a sampled negative.
    • When an answer occurs several times on a page, the other occurrences can be negatives (33.5% of rows).
  • The learned signal is not tied to the answer string. We used 640 TAT-DQA arithmetic QA whose computed answer appears nowhere on the page. Without further training:

    • the retriever keeps all annotated operands in 97.7% of cases, vs 22.5% for random selection (+75.1 [+73.0, +77.1]);
    • it ranks the operand component first in 85%.

    The operands usually lie in one table, so this shows co-located evidence, not separated evidence.

  • Secondary evidence. There is no dedicated mechanism. Secondary evidence is kept only when it lies in the answer's component or when the rule selects 3 or 5 components. Our retention (97.5% of all QA, 99.5% reliable) is answer-bearing retention, not evidence sufficiency.

Manuscript revisions.

  • Restrict the claim to document QA where the answer and its identifying context lie in a few layout regions of a known page.
  • State that the budget reflects retriever confidence, not evidence scope.
  • Add the question-type, TAT-DQA, failure and whole-document results.
  • List abstractive, long-document and separated multi-region evidence as limitations.
Reviewer_ZEb4OpenReview ↗

Strong AI-generated vibe to response

The replies have a strong AI-generated vibe to them in terms of tone and language. Also, aspects such as confidence intervals were never asked for, but they are reported - the confidence intervals are a characteristic tell about AI models. In this case, they add no additional value to the response.

Since there are methodological issues with the approach, I retain my score.

Reviewer_Gt3UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DocComp reduces document VQA cost by detecting layout components, ranking their image crops with an adapted ColSmol retriever, selecting up to five components with a score gap policy, and reconstructing inputs for frozen Qwen and InternVL models. The main contribution is a practical combination of component supervision, adaptive selection, and processor specific reconstruction. The results show substantial token savings across nine backbones, although the incremental advantage over prior learned region selection remains unclear.

Critical strengths
  • Component adaptation has a substantial measured effect. On the 35,349 reliable examples with identical candidates, R@1 rises from 37.78% to 92.56%. With a common Top 3 policy, downstream ANLS rises from 0.5898 to 0.8179, with a document bootstrap interval excluding zero. The authors appropriately disclose that selected crop sizes and token counts differ (Tables 2 and 3, Appendix D).

  • The shared selector works across nine frozen backbones. Qwen token reductions are 62.34% to 63.26%, with overall ANLS changes between minus 1.90 and plus 0.84 points. InternVL results demonstrate why reconstruction matters: balanced reconstruction improves three of four models at 33.20% token reduction (Table 1).

  • Efficiency reporting distinguishes VLM only estimates from pipeline costs. On the reliable subset, including the front end, page amortized compute falls 61.04% and latency falls 29.33%. Reporting that uncached latency instead increases 4.75% makes the deployment limitation clear (§5.7, Appendix B).

Critical weaknesses
  • The experimental distinction from prior learned region selection could be clearer. RegionRAG learns query relevant regions and uses an off the shelf generator without additional training (AAAI 2026, §4.1). Its corpus retrieval setting differs from DocComp's within page ranking, and §2 already identifies DocComp's detector components and adaptive budget. Tables 2 and 4 establish gains over unadapted retrievers and post hoc pruning, but do not measure the advantage of this component design over a learned region proposal alternative. A comparison restricted to the known page, or a more precise account of why that comparison is unsuitable, would strengthen the contribution.

  • The reliable subset needs a reproducible selection rule. The paper explicitly limits its 100% candidate coverage and 99.55% answer location retention results to this curated population. Appendix A explains OCR annotation errors, but does not specify how the 35,349 examples are selected from 41,180. The positive assignment rule in §3.2 does not resolve that population definition. This leaves the effect of curation on detector coverage unclear. Full set QA results still provide evidence for overall answer quality (§4.1, §5.1, §5.3).

  • Implementation details are insufficient for reproducing the principal claims. The manuscript specifies the objective and adapter parameter count, but omits the LoRA configuration, optimizer, learning rate, batch size, and training duration. The context margin has no numerical value. Profiling lacks hardware and timing methodology, making the latency improvements difficult to reproduce (§3.2, §3.3, §4.2, Appendices A and B).

Questions for authors
  • What additional accuracy or pipeline cost benefit does detector based component selection provide over learned region proposals in a controlled known page setting? If RegionRAG is unsuitable for that comparison, which specific methodological differences prevent it?

  • What exact rules produce the reliable subset? Do they condition on detector coverage, and what are the detector failures among the remaining examples?

  • Can you provide the missing training configuration, context margin, processor budgets, and profiling hardware and timing protocol?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Gt3U — Part 1/2

We thank the reviewer. Two strengths cited in the review use numbers we have since corrected (see the general response):

  • GT-free values: R@1 92.40%, Fixed-Top-3 ANLS 0.8178, retention 99.53%, Qwen token reduction 61.67–62.61%, InternVL Balanced 31.38% (still improves 3 of 4 models).
  • The −29.33% latency and −61.04% page-amortized compute gave page reuse to DocComp only. Under equal reuse DocComp is not faster, so we remove the latency claim (Q3); the analytical compute reduction becomes −50.7% on the reliable subset (−49.0% on all 41,180 QA).

W1 / Q1. Benefit over a learned region proposal on the known page.

We ran RegionRAG (official RegionRet adapter, Top-4) on all 41,180 known-page QA. All three use the same frozen Qwen2.5-VL-7B configuration: snapshot, library versions, prompt, greedy decoding and batching. Full-page and DocComp were re-generated under the exact configuration of the RegionRAG run. Statistics are paired document-cluster bootstrap with 5,000 resamples.

Full-pageRegionRAGDocComp
ANLS (41,180)0.81640.81310.8229
ANLS (reliable 35,349)0.81750.81370.8295
Visual tokens1,0911,448408
  • DocComp − RegionRAG: +0.0098 [+0.0051, +0.0149] ANLS, with 72% fewer tokens.
  • Why they differ. On a known page, RegionRAG's regions cover almost the whole page. DocComp selects a much smaller footprint. On 1,000 QA (actual crop footprints, answer-word coverage ≥ 80%):
Page area selectedAll answer words keptVisual tokens
RegionRAG98.71%99.60%1,446
DocComp31.62%90.70%414

RegionRAG also gives each region its own processor window, so it uses more tokens than the full page. DocComp keeps the page's pixel density. Its all-word criterion is stricter than the component retention (97.5%) reported elsewhere.

  • Caveats.
    • RegionRAG is applied zero-shot; DocComp's retriever is trained in-domain.
    • Gains are source-dependent: positive on DeepForm, FATURA and VRDU; negative on MP-VQA, XFUND, DUDE and Kleister-Charity.
    • Under a low page budget (max 262,144 pixels), DocComp is −0.061 [−0.086, −0.038] below RegionRAG.
  • A PinPoint comparison is in our response to DhgX: +0.126 on Qwen2-VL and +0.049 on LLaVA-NeXT.

W2 / Q2. Reliable-subset rule, detector coverage, detector failures.

  • Purpose. The reliable subset is used to evaluate the retriever (R@1, answer-region retention), which needs exactly one well-defined positive component. End-to-end accuracy is evaluated on all 41,180 QA; reliable-subset ANLS is a secondary view.
  • Rule. 65,444 test rows are grouped into 41,180 QA-page examples. An example is kept if any of its rows is reliable. Replaying the rule reproduces the 35,349 IDs exactly.
Rule (in order)ExcludedRemaining
No fatal quality issue (missing geometry/OCR, OCR–box mismatch, no overlapping detector candidate, weak match)3,33937,841
Exactly one plausible positive component44137,400
Answer tokens not split across components42936,971
Answer box not crossing a component boundary42936,542
Match score ≥ 0.551,19335,349
  • Does it condition on detector coverage? Yes. The 100% coverage is partly by construction, and we will state this in §4.1.
  • Detector status of the 5,831 excluded QA.
QAShare
A candidate covers the answer5,07387.0%
Candidates exist, none covers the answer3766.4%
No candidate (Full-page fallback)3826.6%
  • Over all 41,180 QA, coverage is 98.16%.
AuthorsOpenReview ↗

Response to Reviewer Gt3U — Part 2/2

W3 / Q3. Training configuration, context margin, processor budgets, profiling.

  • Retriever.
    • ColSmolVLM-500M (650243e9) with text-side LoRA on q/k/v/o/gate/up/down: r 32, α 32, dropout 0.05, 17.37M parameters.
    • Multi-positive InfoNCE (τ 1.0), AdamW with linear warmup and decay, BF16.
    • Batch of 48 queries, ≤ 512 crops per batch, ≤ 32 negatives per query.
    • Training data: 479,947 train instances (one per annotated answer occurrence; 193,033 questions) and 60,278 validation instances.
    • 2 epochs: epoch 1 at lr 2e-4 (10,121 steps); epoch 2 at lr 5e-5, weight decay 0.01.
    • Hardware: one NVIDIA H200.
  • Detector. DocLayout-YOLO (YOLOv10m, D4LA), frozen. TensorRT, input 1024², confidence 0.01, NMS IoU 0.45, 4 px padding.
  • Selection. Top-1 if gap(1,2) ≥ 4.0; else Top-3 if gap(3,4) ≥ 1.5; else Top-5.
  • Context margin. 24 px of the original image around each box.
  • Processor budgets.
    • Qwen. min/max pixels 153,664 / 1,003,520. Crops keep the full-page density and are not resized.
    • InternVL. 448² tiles, 256 tokens per tile, at most 12 tiles plus a thumbnail.
      • Full-page: mean 2,735 tokens.
      • Efficient: one tile per component plus a thumbnail, 870 tokens (−68.2%).
      • Balanced: components spanning ≤ 2 tiles become one tile; others keep their tiles at page scale; plus a thumbnail. 1,877 tokens (−31.4%).
  • Profiling.
    • Setup: one H200, batch 1, BF16, SDPA, greedy decoding (64 tokens), transformers 4.51.3.
    • CUDA-synchronised stage timers, 10 warm-up queries, fixed 500 reliable QA.
    • Full-page 1,068.8 ms = loading 410.9 + processor 228.8 + vision 231.5 + prefill/decoding 197.6.
    • DocComp 1,119.6 ms = component encoding and ranking 590.8 + detection + VLM path 410.6. Detection is 55.1 ms/page.
  • Equal reuse (new). Full-page visual embeddings are cached per page. Inputs are token-identical to the uncached run.
ms per QAFull-pageDocComp
No reuse1,068.81,119.6
Page-level reuse, DocComp only (paper)1,068.8755.3
Page-level reuse, both (new)625.7755.3

We remove the latency claim. What remains:

  • decoder tokens −63%;
  • VLM stage −45%;
  • analytical FLOPs −46% to −51%.

All settings above will be added to Appendix A and B.

Thank the authors for detailed rebuttal within a limited time period.

Further details about experimental setting and implementation resolved concerns regarding reproducibility and clarity.

However

  • Under the comparable setting (page-amortized), latency improvement does not hold anymore, which was a key strengths in the initial review.
  • Also, regarding comparison with relevant work RegionRAG, as RegionRAG is zero-shot and DocComp is trained in-domain, quantitative results (+0.0098 gain and lower performance on low budget setting, although with 72% fewer tokens) is not very competitive.

Due to these issues, I keep the original score of 2: Weak rejection.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers. Their questions led us to re-audit the pipeline. Below we summarize the corrections and the new results. Details are in the individual responses.

1. GT-free re-evaluation. The historical test candidate cache contained an answer-anchor fallback crop, built for training supervision. It reached the generator in 908 of 41,180 test QA (172 of 35,349 reliable). We rebuilt detector-only candidates, re-ranked them with the frozen retriever, and re-ran all nine backbones.

OriginalGT-free
R@1, base → adapted37.78% → 92.56%37.66% → 92.40%
Fixed-Top-3 ANLS, base → adapted0.5898 → 0.81790.5959 → 0.8178
Answer-region retention, reliable / all99.55% / –99.53% / 97.50%
Qwen token reduction62.34–63.26%61.67–62.61%
Qwen ANLS vs Full-page−1.90 to +0.84 pt−2.13 to +0.74 pt
InternVL Balanced token reduction33.20%31.38% (still improves 3 of 4 models)

2. Document disjointness. Three test documents (44 QA) have identical page images in the retriever's training split. We remove them from the test set, so evaluation is document-disjoint by document ID and page-image hash. On the remaining 41,136 QA, every ANLS changes by at most 0.0003.

3. Latency under equal reuse. The reported −29.33% gave page-level reuse to DocComp only. With the same reuse for Full-page, measured on the paper's protocol:

ms per QAFull-pageDocComp
No reuse1,068.81,119.6
Page-level reuse, DocComp only (paper)1,068.8755.3
Page-level reuse, both (new)625.7755.3

We therefore remove the latency claim. DocComp reduces decoder visual tokens by 63% and analytical FLOPs by 46–51%. The VLM stage is 45% faster.

4. New results.

Comparison (41,180 QA)ΔANLS [95% CI]Visual tokens
DocComp − Full-page (Qwen2.5-VL-7B)+0.0066 [+0.0017, +0.0116]408 vs 1,091
DocComp − RegionRAG+0.0098 [+0.0051, +0.0149]408 vs 1,448
DocComp − PinPoint (Qwen2-VL-7B)+0.126 [+0.111, +0.141]635 vs 1,017
DocComp − PinPoint (LLaVA-NeXT-7B)+0.049 [+0.032, +0.063]1,533 vs 1,768
  • Baselines use their released weights zero-shot. Under a low page budget (max 262,144 pixels), DocComp is −0.061 below RegionRAG.
  • Whole-document input: unchanged up to 3 pages; beyond 10 pages ANLS falls from 0.829 to 0.614.
  • Failures: 98.5% of DocComp errors keep the answer region. Higher resolution or a wider context recovers most of them.
  • Question types: no significant loss on TAT-DQA comparative and arithmetic questions; −12 pt on out-of-domain extractive lookups.

5. Revised claim. DocComp reduces visual tokens by about 62% with frozen generators. Accuracy is comparable to or better than Full-page on Qwen2.5-VL-7B, and setting-dependent elsewhere. It targets questions whose answer and identifying context lie in a few layout regions of a known page.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.