acceptodds
Under review as a conference paper at ICLR 2027

HiToc: Hierarchical Token Compression Across the Vision-Language Pipeline for Long Document Question Answering

Abstract

Long-document question answering is computationally expensive due to long visual-token sequences. Compressing this context without disrupting document structure or losing cross-page evidence remains challenging. We propose HiToc, a training-free hierarchical token compression framework that progressively refines visual context across the vision-language pipeline. After visual encoding, Bottom-up Background Token Merging reduces background redundancy while preserving layout cues. During prefilling, Adaptive Page-level Token Assignment converts question-conditioned attention into soft page budgets and then selects tokens within each page, reducing attention-sink effects and preserving complementary cross-page evidence. During generation, layer-adaptive dynamic KV-cache compression reselects active visual context across decoder depths at each step. Across four document QA benchmarks, HiToc reduces the final active visual-token budget to under 20% of the original input while preserving 94.94–98.72% of baseline performance and achieving – LLM-stage speedups.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Token Compression, Document Question Answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1TNpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HiToc applies three compression stages. Bottom-up Background Token Merging uses image-processing cues to identify background patches and merges their encoded tokens. Adaptive Page-level Token Assignment uses question-to-visual attention to allocate soft page budgets before selecting tokens within each page. Finally, dynamic KV selection activates different visual subsets in shallow and deep decoder stages during generation.

The main experiments use Qwen3-VL-4B on four document-QA benchmarks, with additional Qwen3-VL-8B and official test-split results in the appendix. The method retains 3.6%3.6\%–18.2%18.2\% of the original visual-token count in its final active subset and reports LLM-stage speedups of 1.50×1.50\times–4.05×4.05\times, with relatively small losses in answer quality.

Critical strengths
  1. The evaluation covers meaningful long-document workloads. Four benchmarks include substantially different document lengths and question types. Official DUDE and MP-DocVQA test results and an 8B extension strengthen the evidence beyond the main 4B validation results (pp. 6–7 and 14).
  2. The component evaluation is informative. The paper reports cumulative module effects, a per-sample token-matched global-selection comparison, and alternative decoder-stage configurations (Tables 1–3). These comparisons support the practical usefulness of page-to-token allocation.
  3. Efficiency is measured in several ways. The paper reports visual-stage overhead, prefill and decode latency, incremental GPU memory, and FLOPs, rather than relying exclusively on token retention (pp. 8–9, Tables 4–6).
  4. The appendix examines cross-page and unanswerable questions. These analyses help assess whether compression disproportionately harms important aspects of document QA (pp. 18–20).
Critical weaknesses

Post-rebuttal update: I have updated my rating from 2 (weak rejection) to 3 (weak acceptance). The clarification of candidate KV storage and cost accounting, the paired statistical analysis, and the frozen-stage-set ablation substantially address my original concerns. I also acknowledge that the compared LLM-stage pruning methods use the same pruning depth as APTA, and withdraw my earlier implication to the contrary. On balance, the evidence supports weak acceptance. The weaknesses below reflect my initial assessment; my final assessment is provided in the discussion response.

  1. The dynamic cache mechanism needs a precise storage contract. The method reselects from post-APTA candidates at each generation step and allows inactive entries to become active again. It is unclear where these entries reside, which layers retain the larger candidate cache, and how rescoring and gathering are implemented. Final active-token retention of 3.6%3.6\%–18.2%18.2\% is different from the reported incremental-memory reduction of 2.42%2.42\%–16.37%16.37\%; these quantities should be explicitly connected (pp. 5 and 9).
  2. Matching final token counts does not match computation. APTA runs at layer 20, after substantial processing of the larger sequence. Comparing full HiToc against methods forced to reach the same final retention before question-conditioned processing does not establish an advantage at matched FLOPs, latency, or memory. The stage-aligned BBTM comparison is useful, but does not close this gap for the full pipeline (pp. 5–7).
  3. Novelty is primarily compositional. Sobel-based foreground detection, token merging, attention aggregation, and top-kk selection are established ingredients. The page rule ρp=sp/max⁡p′sp′\rho_p=s_p/\max_{p'}s_{p'} does not enforce a fixed global budget, and summing attention over a page can introduce page-length dependence. The extent to which this rule specifically mitigates attention sinks needs stronger evidence.
  4. Whole-pipeline gains are smaller than the headline LLM-stage gains. Summing the reported visual and LLM times gives approximately 881.80/572.58≈1.54×881.80/572.58\approx1.54\times on MMLongBench-Doc and approximately 1.21×1.21\times on DUDE. The abstract correctly labels its speedups as LLM-stage, but practical comparisons should emphasize the complete pipeline. TokenCorr has lower total latency on three of the four datasets in Table 9 (pp. 9 and 15).
  5. Some supporting evidence is incomplete. Accuracy comparisons lack paired uncertainty estimates, final active budgets vary by dataset, and attention-distribution similarity is not direct evidence that the correct supporting pages were retained. The duplicated discussion after Table 8 also needs correction.
Questions for authors

Post-rebuttal update: I have considered the authors’ responses to the questions below and have no further questions.

  1. Where are inactive KV entries stored, which layers retain them, and what is the precise sequence of rescoring, gathering, attention, and reactivation at each decoding step? Can you provide pseudocode and per-layer physical KV sizes?
  2. Can you compare methods on quality versus measured end-to-end latency, FLOPs, and memory curves, rather than only matching final active-token counts?
  3. What results are obtained with a single active-budget setting across all datasets? Which independent data were used to choose the pruning layers and other hyperparameters?
  4. Can you provide document-level paired confidence intervals and direct evidence-retention measurements for cross-page questions, including failure cases involving attention sinks or non-text visual content?
  5. Can you correct the paragraph following Table 8 and explain the actual transfer results on Qwen3-VL-8B?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1TNp – Part 1

We thank the reviewer for the constructive feedback. We address each concern below and will incorporate all additions into the revision.

Q1: More details on KV storage and the execution order during decoding.

A1: At each decoding step: (1) Rescore the retained visual candidates using q_t’s full-candidate attention, aggregated by the average across heads. (2) Select the top-k_d% visual KV entries for each stage. (3) Gather the selected entries from each layer’s own candidate cache; only previously inactive entries selected at the current step are reactivated, while all unselected visual entries remain inactive. (4) Compute attention using the selected visual KV and all text KV, including the newly appended text tokens.

Inactive entries remain in each layer’s GPU-resident candidate cache, restricted to tokens surviving prefill compression. Both candidate storage and active buffers are included in profiled memory, and scoring, selection, gathering, and attention are included in LLM-forward latency. We will clarify this attention-based selection and subsequent KV-access sequence in Section 3.3.

The pseudo codes are listed as follows. Cache_l is layer l's candidate KV cache; Gather reads the selected visual entries and all text entries without adding text KV again.

for each decoding step t:
    for each decoder layer l, in order:
        q_t = CurrentQuery(l, t)
        AppendCurrentTextKV(Cache_l)
        if l provides scores for stage g in {s, d}:
            A = ComputeFullCandidateAttention(q_t, Cache_l)
            I_g = TopPercent(AggregateHeads(A)[K_vis], k_d)
        elif l uses stage g's selection:
            KV_active = Gather(Cache_l, I_g ∪ all_text_IDs)
            ComputeAttention(q_t, KV_active)
        else:
            ComputeFullCandidateAttention(q_t, Cache_l)
# Only selected visual entries are active in compressed layers.

For physical sizes, let CℓC_\ell denote the number of retained visual candidates at layer ℓ\ell, TT the number of cached text tokens, and AgA_g the number of active visual tokens in stage gg. Each K or V tensor has shape B×Hkv×n×dB \times H_{\mathrm{kv}} \times n \times d, where BB is the batch size, HkvH_{\mathrm{kv}} the number of KV heads, nn the corresponding cache length, and dd the head dimension. Let bb denote bytes per element.

LayersResident candidate lengthAttention length
0≤ℓ≤Ls0 \leq \ell \leq L_sCℓ+TC_\ell + TCℓ+TC_\ell + T
Ls+1≤ℓ≤Ld−1L_s+1 \leq \ell \leq L_d-1Cℓ+TC_\ell + TA0+TA_0 + T
ℓ=Ld\ell = L_dCℓ+TC_\ell + TCℓ+TC_\ell + T
Ld+1≤ℓ≤35L_d+1 \leq \ell \leq 35Cℓ+TC_\ell + TA1+TA_1 + T

The candidate K/V pair at layer ℓ\ell occupies Mℓcand=2BHkv(Cℓ+T)db.M_\ell^{\mathrm{cand}} = 2 B H_{\mathrm{kv}} (C_\ell + T) d b. A separately materialized active K/V pair for stage gg adds Mℓ,gactive=2BHkv(Ag+T)db.M_{\ell,g}^{\mathrm{active}} = 2 B H_{\mathrm{kv}} (A_g + T) d b. Both expressions give storage in bytes. We will include this pseudocode and per-layer storage specification in the revision.

Q2: Can you compare methods on quality versus measured end-to-end latency, FLOPs, and memory curves, rather than only matching final active-token counts?

A2: Our FastV, DivPrune and SparseVLM implementations prune at the same decoder layer as APTA (layer 20), so they also receive question-conditioned processing before selection. HiToc’s higher quality in Table 2 therefore cannot simply be attributed to later pruning.

HiToc additionally applies BBTM before the LLM, so fewer visual tokens enter the LLM than in the compared LLM-stage pruning methods. It further reduces the active visual KV subset during decoding, targeting costs across multiple inference stages. Different compression stages and architectural designs make exact matching of latency, memory, or TFLOPs difficult. Comparing end-to-end performance at matched final token-pruning ratios is therefore a reasonable and commonly used evaluation protocol.

Q3: Results with a single active-budget setting across all datasets. Which independent data were used to choose the pruning layers and other hyperparameters?

A3: We additionally evaluated full HiToc on MMLongBench-Doc with k_d=45%, reducing the active visual KV budget to 739.9 tokens, approximately one page (768 tokens). This extends the roughly one-page active-budget setting to all four benchmarks. Under this aggressive compression setting—retaining only 2.03% of the original visual tokens —HiToc still achieves 26.53% accuracy and 22.49% F1, retaining 95.57% and 94.89% of the uncompressed baseline's accuracy and F1, respectively.

We held out 10% of the DUDE training set as an independent development set to select the pruning layers and other hyperparameters. The trends observed during tuning were consistent with the ablation results on the DUDE validation set shown in Figure 3. We will clarify this parameter-selection protocol in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 1TNp – Part 2

Q4: More direct evidence-retention measurements for cross-page questions.

A4: Using official evidence annotations in MMLongBench-Doc, we evaluate both answering quality and evidence retention. With 20,000 document-level paired bootstrap resamples, BBTM+APTA improves accuracy over token-matched DirectChoose by 4.04 percentage points on 358 cross-page questions (95% CI: [0.69, 6.05]) and by 5.11 points on the 193 involving charts or figures ([1.34, 9.03]). Both intervals exclude zero.

The table measures the fraction of visual tokens originating from evidence pages, averaged over questions. Its three nested subsets comprise 838 answerable questions with valid evidence-page IDs, 354 requiring evidence from multiple pages, and 190 of those involving charts or figures. These counts exclude annotations referring to pages absent from the inputs. All four selectors use identical input visual tokens and retain the same number per question; the uncompressed baseline uses all original visual tokens.

SubsetUncompressed baselineAPTADirectChooseFastVDivPrune
All valid evidence annotations (838)6.27%18.55%8.89%8.66%6.53%
Cross-page (354)10.21%20.03%12.29%12.12%10.36%
Cross-page charts/figures (190)10.03%19.51%12.34%12.17%10.05%

On cross-page questions, evidence pages account for 10.21% of visual tokens in the uncompressed input and 20.03% of APTA's retained tokens, approximately a 1.96-fold increase in evidence-page concentration. APTA also exceeds the three token-matched selectors.

We recorded attention, token retention, and answers in the same run for two cross-page counting failures. In both, the non-evidence page containing attention sinks ranks first, while evidence pages rank lower. APTA keeps only 39/590 and 40/588 candidate tokens on chart evidence pages 8 and 12, and 112/653 and 346/644 on person evidence pages 3 and 4, respectively.

Case (pred. / GT)Evidence-page rank (all pages)Retained tokens by APTA per evidence pageRetained grid coverage
N/A counting (2 / 3)p8: 12/23; p12: 10/23p8: 39; p12: 405/7
Person counting (5 / 6)p3: 3/26; p4: 2/26p3: 112; p4: 34636/37

Although the evidence pages rank below the sink-dominated page and receive limited token budgets, most answer-region spatial support survives, as shown in the last column. This is consistent with page-to-token selection: evidence tokens need not individually outrank background tokens on other pages. The final failed answer may involve visual recognition or counting limitations of the VLM backbone.

Q5: Correct the paragraph for Qwen3-VL-8B following Table 8.

A5: The paragraph following Table 8 mistakenly repeats the official test-split discussion from Table 7. We will replace it with the following analysis of transfer to Qwen3-VL-8B.

Overall, HiToc remains effective when transferred to a larger backbone. On DUDE and MP-DocVQA, HiToc still achieves strong performance under approximately 20% of the original token budget. Specifically, on DUDE, HiToc obtains an ANLS of 0.5877, retaining about 98.2% of the baseline performance (0.5987). On MP-DocVQA, it achieves 0.8509 ANLS and 74.86% accuracy, corresponding to about 98.3% and 97.0% of the baseline performance, respectively. On MMLongBench-Doc, HiToc still maintains strong performance under a merely 3.6% token budget, retaining 97.0% of the baseline Accuracy and 98.2% of the baseline F1-score.

A notable observation is that, on the 8B backbone, introducing APTA leads to more degradation compared to BBTM alone. Compared with BBTM, which removes redundant background regions while preserving most semantic content, APTA further reduces the token budget from roughly 80% to 20% through hierarchical page-wise pruning. Such structured selection may mildly affect continuous semantic patterns or long-range evidence utilization that the larger backbone can exploit. Nevertheless, the performance of APTA-based variants remains strong overall. Moreover, the final HiToc model recovers part of the performance drop after APTA on DUDE and MP-DocVQA and improves MMLongBench-Doc F1, supporting the effectiveness of the proposed layer-adaptive KV-Cache compression in dynamically preserving useful key-value pairs according to both decoding context and layer depth.

On MMDA, HiToc retains 98.3% of baseline ACNLS (0.5950→0.5851) and 94.1% of AIC-Acc (70.10%→65.98%). The larger reduction in AIC-Acc may reflect the metrics’ different sensitivities: ACNLS gives partial credit for lexical overlap, whereas AIC-Acc evaluates answer correctness. Compression may occasionally remove a decisive detail, such as a number, entity, or qualifier, causing an otherwise similar answer to be judged incorrect. This can produce a larger drop in AIC-Acc while leaving ACNLS relatively stable.

Reviewer_HYCsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HiToc combines conservative background token merging after visual encoding, question conditioned page budgets during prefilling, and visual KV selection at two decoder depths during generation. On four document QA benchmarks, it retains 3.6% to 18.2% of the original visual tokens as the final active budget, with limited quality degradation and 1.50× to 4.05× LLM stage speedups. The strongest contribution is page allocation and its controlled comparison with global selection.

Critical strengths
  • APTA has a particularly useful control. Against DirectChoose with identical token counts for each sample, it improves MMLongBench accuracy from 23.10% to 27.37% and DUDE ANLS from 0.5620 to 0.5831 (Table 3). This supports page budgeting beyond retaining more tokens.

  • The appendix strengthens the evaluation. Official test results, a larger Qwen3 backbone, cross page question results, and answerability analysis address several plausible generalization and evidence preservation concerns (Tables 7, 8, 12, 15). Cross page accuracy retains 96.29% of baseline performance.

  • The paper measures actual latency and incremental memory beyond token counts. It separates visual and LLM costs and reports the retrieval overhead and total latency of document specific baselines (Tables 4 to 6 and 9).

Critical weaknesses
  • Dynamic KV selection needs a more complete specification (Section 3.3). Temporary eviction and later reactivation are described, but inactive KV storage, the application of selected token IDs to each layer's own cache, and aggregation of dynamic selection scores across heads remain unclear. APTA does specify head aggregation in Eq. 5. Clarifying the decoding procedure would make the active attention budget and resident cache cost reproducible.

  • Matched final token ratios are not matched compute budgets. APTA acts at layer 20, and decoding activates 70% to 80% of its candidates. Nevertheless, Tables 2 and 5 show higher quality and lower total LLM latency than SparseVLM on all four benchmarks at the reported settings. The identical-token DirectChoose comparison also supports the value of page allocation. An optional quality versus measured cost sweep could test how broadly this advantage holds across budgets.

  • The dynamic cache evidence establishes depth complementarity more clearly than temporal adaptation. SingleStage versus two stages changes the number of selection depths, while Appendix C.3 measures overlap across depths. Neither isolates the benefit of reselecting at every generation step relative to freezing the two stage specific sets. This matters because temporal reactivation is a central claimed distinction.

  • The operational distinction from cited prior work could be more explicit. ZipVL (ICCV 2025) selects layer specific caches during prefilling and discards unselected KV entries, while Lethe (AAAI 2026) repeatedly prunes a growing language model KV cache during reasoning. HiToc describes reselection of active visual entries from the post-APTA candidate set. Its document hierarchy and this reactivation are meaningful differences, and a direct comparison of candidate retention and eviction rules would clarify the scope of its decoding contribution.

Questions for authors
  • Can you specify the decoding cache state and selection procedure, including head aggregation, inactive entry storage, layer specific KV access, and their inclusion in measured memory and latency?

  • With the two selection depths and candidate budget fixed, how much quality and latency benefit comes specifically from reselection over generation steps versus frozen stage specific sets?

  • Could Section 2.3 explicitly compare HiToc's candidate retention and reactivation with the eviction mechanisms of ZipVL and Lethe?

3 comments
AuthorsOpenReview ↗

Response to Reviewer HYCs

We thank the reviewer for the constructive feedback. We address each concern below and will incorporate all additions into the revision.

Q1: Can you specify the decoding cache state and selection procedure, including head aggregation, inactive entry storage, layer specific KV access, and their inclusion in measured memory and latency?

A1: HiToc maintains a candidate KV cache and a stage-specific active set. Each layer retains its own candidate K/V on the GPU; inactive entries are excluded from that layer’s active attention but remain available for reselection. At each decoding step and selection depth, we score the full visual candidates using the current decoding query’s attention, taking the average across heads, and select the top-k_d% entries under the stage budget. Layers within the same stage share selected token IDs but access their own K/V tensors.

All of these costs are included in our reported measurements. The synchronized LLM-forward timing includes attention scoring, top-k selection, KV gathering, and attention computation. Table 6 reports incremental peak GPU memory measured by profiling the complete generation call and subtracting the pre-call allocation. This measurement includes resident inactive candidate K/V, active KV buffers, and auxiliary selection tensors. Thus, the reported latency and memory already account for maintaining and accessing the candidate pool. We will make these implementation and measurement details explicit in the revision.

Q2: With the two selection depths and candidate budget fixed, how much quality and latency benefit comes specifically from reselection over generation steps versus frozen stage specific sets?

A2: We have completed the frozen-stage-set evaluation on all four benchmarks. Each stage independently selects its initial visual KV subset and keeps those visual IDs fixed throughout generation.

BenchmarkHiToc qualityFrozen qualityHiToc decode (min)Frozen decode (min)
DUDEANLS 0.5791ANLS 0.573983.44 ± 3.5275.71 ± 1.91
MMDAACNLS 0.5525; AIC-Acc 61.98%ACNLS 0.5449; AIC-Acc 59.41%9.70 ± 0.1711.17 ± 1.05
MP-DocVQAANLS 0.8275; Acc 70.99%ANLS 0.8106; Acc 68.34%50.07 ± 1.1160.40 ± 1.26
MMLongBench-DocAcc 26.63%; F1 22.50%Acc 26.98%; F1 22.21%26.43 ± 2.5425.35 ± 0.27

HiToc achieves higher quality on six of seven metrics, with particularly clear gains on MMDA and MP-DocVQA. Decoding times show mixed outcomes across benchmarks, with neither variant consistently faster. Overall, the reported results favor dynamic selection for answer quality without showing a consistent increase in total decoding time.

Q3: Could Section 2.3 explicitly compare HiToc's candidate retention and reactivation with the eviction mechanisms of ZipVL and Lethe?

A3: We will compare the methods at the same inference stage. During prefilling, ZipVL uses a global ranking of normalized attention scores with layer-adaptive budgets; APTA allocates budgets across pages before within-page token selection. This explicitly protects locally important tokens on lower-scoring pages from competition with globally higher-scoring tokens. Table 3 provides a token-matched control for this allocation choice, and Figure 4 Q2 illustrates robustness to an incorrect top-page ranking. Neither method recovers tokens already removed during prefilling.

During decoding, Lethe evicts entries from its growing cache, whereas HiToc selects active visual entries from retained post-APTA candidates. Previously inactive candidates can therefore participate again, at the cost of retaining their storage. Section 2.3 will distinguish page-wise allocation from decoding-time recoverability rather than conflate them into a single eviction comparison.

Thanks the authors' detailed rebuttal, including details about the implementation, further analysis, and comparison with related works. I recommend the authors to include these details in the final version of the paper.

I will keep my score of 3 (weak acceptance).

We sincerely thank the reviewer for the constructive feedback. We will incorporate the implementation details, additional analyses, and comparisons with related work into the final version to make the methodology clearer and the evaluation more comprehensive.

Reviewer_jiKGOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

HiToc introduces a training-free framework for long-document question answering. It combines Bottom-up Background Token Merging (BBTM), Adaptive Page-level Token Assignment (APTA), and layer-adaptive dynamic KV-cache compression to progressively reduce visual context. Results on four DocQA benchmarks show substantial compression with limited performance loss and reduced LLM-stage latency.

Critical strengths
  • APTA outperforms DirectChoose across all four benchmarks while retaining the same number of tokens for each sample, directly supporting page-to-token allocation over global token ranking.

  • HiToc preserves most baseline performance on the official DUDE and MP-DocVQA test splits and on Qwen3-VL-8B, supporting its effectiveness across evaluation splits and model sizes.

Critical weaknesses
  • APTA’s page-level relevance score still includes attention-sink weights. If these weights make a non-target page the highest-scoring page, that page retains all tokens while other pages receive lower retention ratios. Figure 4’s Q2 illustrates this ranking error. The results support APTA’s overall effectiveness and tolerance to such errors, but do not isolate reduced attention-sink interference as the cause of its gains.

  • SingleStage and ThreeStage vary KV selection across decoder depths, while chance-corrected Jaccard similarity measures differences between shallow and deep stages. These analyses support selection across depths but do not isolate the benefit of reselection at each decoding step. Different selected subsets also do not establish that they provide complementary visual evidence as generation progresses.

  • Table 2 matches the final active visual-token budget without matching the visual context available throughout inference. HiToc applies APTA at layer 20 and can reselect from a larger post-APTA candidate pool during decoding. The quality differences therefore combine token selection, compression timing, and candidate-pool size. The measured latency supports practical efficiency gains but does not resolve this attribution issue.

  • The effective number of attended pages, the number of pages receiving at least 5% attention mass, and normalized attention variance in Table 13 are insensitive to page identities. They can remain unchanged even when attention shifts to incorrect pages. Table 12 supports retained cross-page answering performance, but Table 13 establishes only similar attention spread and cannot identify which evidence pages remain attended.

Questions for authors

Please see my "weaknesses" comments.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jiKG

We thank the reviewer for the constructive feedback. We address each concern below and will incorporate all additions into the revision.

Q1: The results support APTA’s overall effectiveness and tolerance to page ranking errors caused by attention sinks, but do not isolate reduced attention-sink interference as the cause of its gains.

A1: Our claim is that APTA mitigates the impact of attention sinks on token selection rather than eliminating the sinks themselves. Its page-to-token design allows evidence tokens to survive without individually outranking high-attention background tokens on other pages.

DirectChoose ranks visual tokens globally, making evidence tokens compete directly with high-attention sink tokens. APTA mitigates this vulnerability through page-specific budgets followed by within-page selection. Under identical input visual tokens and matched retained-token budgets, APTA improves the accuracy from 10.48% on DirectChoose to 14.52% (document-level paired 95% CI for the gain: [0.69, 6.05] percentage points) on 358 cross-page questions in MMLongBench-Doc. These results support greater robustness to attention-sink interference in token selection, even though the sinks themselves and page-ranking errors may persist.

Q2: Analyses on the benefit of reselection at each decoding step. Different selected subsets also do not establish that they provide complementary visual evidence as generation progresses.

A2: To examine the benefit of reselection across generation steps, we evaluated frozen stage-specific sets on all four benchmarks. Each stage only selects its visual KV subset at the first decoding step and reuses the same visual IDs throughout generation. The two selection depths and dataset-specific token budgets follow the paper’s configuration.

BenchmarkHiTocFrozen stage-specific sets
DUDEANLS 0.5791ANLS 0.5739
MMDAACNLS 0.5525; AIC-Acc 61.98%ACNLS 0.5449; AIC-Acc 59.41%
MP-DocVQAANLS 0.8275; Acc 70.99%ANLS 0.8106; Acc 68.34%
MMLongBench-DocAcc 26.63%; F1 22.50%Acc 26.98%; F1 22.21%

Relative to the paper’s HiToc results, frozen sets score lower on six of seven metrics, including accuracy decreases of 2.57 percentage points on MMDA and 2.65 points on MP-DocVQA.

Different selected subsets indicate that the model chooses different visual tokens. Combined with the overall better answering performance compared with frozen stage-specific sets, these results support adapting visual access to the evolving decoding context to select information more relevant to answering the question.

Q3: Questions on different visual context available throughout inference. The quality differences therefore combine token selection, compression timing, and candidate-pool size.

A3: The results in Table 2 aim to evaluate end-to-end performance against state-of-the-art compression methods at matched final token-retention ratios, a common evaluation protocol. We already align pre-LLM token budgets for ViT-stage methods and pruning depths for LLM-stage methods for a more fair end-to-end comparision. Although HiToc retains a larger candidate pool during decoding, BBTM reduces the visual context available before layer 20 compared with LLM-stage pruning methods in Table 2. HiToc therefore does not uniformly access more visual context throughout inference. Due to the multi-stage compression design principle of HiToc, it is hard to precisely align token selection, compression timing, and candidate-pool size simultanously, we have tried our best to make the comparisons as fair as possible.

Q4: Table 13 establishes only similar attention spread and cannot identify which evidence pages remain attended.

A4: APTA employs attention scores to determine which visual tokens survive compression; it does not change the overall page-attention distribution since page with higher attention still retain more tokens. In this way, the proposed APTA can reduce irrelevent tokens while keeping the attention distribution as similar as possible, which prevents attention shifts to incorrect pages.

To directly evaluate page identity, we additionally measure the retention using official evidence-page annotations in MMLongBench-Doc. With identical input visual tokens and matched retained-token budgets, evidence pages account for 20.03% of APTA’s retained tokens on cross-page questions, versus 12.29% for DirectChoose, 12.12% for FastV, and 10.36% for DivPrune. In uncompressed baseline, the evidence page tokens only account for 10.21% of the whole document tokens. These measurements complement Tables 12–13 by directly showing that APTA concentrates more of its retained-token budget on annotated evidence pages.

Reviewer_jiKGOpenReview ↗

Response to Author Rebuttal

Thanks to all the authors for their detailed responses and clarifications, which have resolved my doubts. I suggest that the authors incorporate these contents into the final version.

We sincerely thank the reviewer for the careful reassessment and positive feedback. We are pleased that our responses resolved your concerns. We will incorporate these clarifications and additional results into the final version to strengthen the experimental evaluation and improve the clarity of our presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.