- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HiToc applies three compression stages. Bottom-up Background Token Merging uses image-processing cues to identify background patches and merges their encoded tokens. Adaptive Page-level Token Assignment uses question-to-visual attention to allocate soft page budgets before selecting tokens within each page. Finally, dynamic KV selection activates different visual subsets in shallow and deep decoder stages during generation.
The main experiments use Qwen3-VL-4B on four document-QA benchmarks, with additional Qwen3-VL-8B and official test-split results in the appendix. The method retains – of the original visual-token count in its final active subset and reports LLM-stage speedups of –, with relatively small losses in answer quality.
Critical strengths
- The evaluation covers meaningful long-document workloads. Four benchmarks include substantially different document lengths and question types. Official DUDE and MP-DocVQA test results and an 8B extension strengthen the evidence beyond the main 4B validation results (pp. 6–7 and 14).
- The component evaluation is informative. The paper reports cumulative module effects, a per-sample token-matched global-selection comparison, and alternative decoder-stage configurations (Tables 1–3). These comparisons support the practical usefulness of page-to-token allocation.
- Efficiency is measured in several ways. The paper reports visual-stage overhead, prefill and decode latency, incremental GPU memory, and FLOPs, rather than relying exclusively on token retention (pp. 8–9, Tables 4–6).
- The appendix examines cross-page and unanswerable questions. These analyses help assess whether compression disproportionately harms important aspects of document QA (pp. 18–20).
Critical weaknesses
Post-rebuttal update: I have updated my rating from 2 (weak rejection) to 3 (weak acceptance). The clarification of candidate KV storage and cost accounting, the paired statistical analysis, and the frozen-stage-set ablation substantially address my original concerns. I also acknowledge that the compared LLM-stage pruning methods use the same pruning depth as APTA, and withdraw my earlier implication to the contrary. On balance, the evidence supports weak acceptance. The weaknesses below reflect my initial assessment; my final assessment is provided in the discussion response.
- The dynamic cache mechanism needs a precise storage contract. The method reselects from post-APTA candidates at each generation step and allows inactive entries to become active again. It is unclear where these entries reside, which layers retain the larger candidate cache, and how rescoring and gathering are implemented. Final active-token retention of – is different from the reported incremental-memory reduction of –; these quantities should be explicitly connected (pp. 5 and 9).
- Matching final token counts does not match computation. APTA runs at layer 20, after substantial processing of the larger sequence. Comparing full HiToc against methods forced to reach the same final retention before question-conditioned processing does not establish an advantage at matched FLOPs, latency, or memory. The stage-aligned BBTM comparison is useful, but does not close this gap for the full pipeline (pp. 5–7).
- Novelty is primarily compositional. Sobel-based foreground detection, token merging, attention aggregation, and top- selection are established ingredients. The page rule does not enforce a fixed global budget, and summing attention over a page can introduce page-length dependence. The extent to which this rule specifically mitigates attention sinks needs stronger evidence.
- Whole-pipeline gains are smaller than the headline LLM-stage gains. Summing the reported visual and LLM times gives approximately on MMLongBench-Doc and approximately on DUDE. The abstract correctly labels its speedups as LLM-stage, but practical comparisons should emphasize the complete pipeline. TokenCorr has lower total latency on three of the four datasets in Table 9 (pp. 9 and 15).
- Some supporting evidence is incomplete. Accuracy comparisons lack paired uncertainty estimates, final active budgets vary by dataset, and attention-distribution similarity is not direct evidence that the correct supporting pages were retained. The duplicated discussion after Table 8 also needs correction.
Questions for authors
Post-rebuttal update: I have considered the authors’ responses to the questions below and have no further questions.
- Where are inactive KV entries stored, which layers retain them, and what is the precise sequence of rescoring, gathering, attention, and reactivation at each decoding step? Can you provide pseudocode and per-layer physical KV sizes?
- Can you compare methods on quality versus measured end-to-end latency, FLOPs, and memory curves, rather than only matching final active-token counts?
- What results are obtained with a single active-budget setting across all datasets? Which independent data were used to choose the pruning layers and other hyperparameters?
- Can you provide document-level paired confidence intervals and direct evidence-retention measurements for cross-page questions, including failure cases involving attention sinks or non-text visual content?
- Can you correct the paragraph following Table 8 and explain the actual transfer results on Qwen3-VL-8B?
2 comments
Response to Reviewer 1TNp – Part 2
Q4: More direct evidence-retention measurements for cross-page questions.
A4: Using official evidence annotations in MMLongBench-Doc, we evaluate both answering quality and evidence retention. With 20,000 document-level paired bootstrap resamples, BBTM+APTA improves accuracy over token-matched DirectChoose by 4.04 percentage points on 358 cross-page questions (95% CI: [0.69, 6.05]) and by 5.11 points on the 193 involving charts or figures ([1.34, 9.03]). Both intervals exclude zero.
The table measures the fraction of visual tokens originating from evidence pages, averaged over questions. Its three nested subsets comprise 838 answerable questions with valid evidence-page IDs, 354 requiring evidence from multiple pages, and 190 of those involving charts or figures. These counts exclude annotations referring to pages absent from the inputs. All four selectors use identical input visual tokens and retain the same number per question; the uncompressed baseline uses all original visual tokens.
| Subset | Uncompressed baseline | APTA | DirectChoose | FastV | DivPrune |
|---|---|---|---|---|---|
| All valid evidence annotations (838) | 6.27% | 18.55% | 8.89% | 8.66% | 6.53% |
| Cross-page (354) | 10.21% | 20.03% | 12.29% | 12.12% | 10.36% |
| Cross-page charts/figures (190) | 10.03% | 19.51% | 12.34% | 12.17% | 10.05% |
On cross-page questions, evidence pages account for 10.21% of visual tokens in the uncompressed input and 20.03% of APTA's retained tokens, approximately a 1.96-fold increase in evidence-page concentration. APTA also exceeds the three token-matched selectors.
We recorded attention, token retention, and answers in the same run for two cross-page counting failures. In both, the non-evidence page containing attention sinks ranks first, while evidence pages rank lower. APTA keeps only 39/590 and 40/588 candidate tokens on chart evidence pages 8 and 12, and 112/653 and 346/644 on person evidence pages 3 and 4, respectively.
| Case (pred. / GT) | Evidence-page rank (all pages) | Retained tokens by APTA per evidence page | Retained grid coverage |
|---|---|---|---|
| N/A counting (2 / 3) | p8: 12/23; p12: 10/23 | p8: 39; p12: 40 | 5/7 |
| Person counting (5 / 6) | p3: 3/26; p4: 2/26 | p3: 112; p4: 346 | 36/37 |
Although the evidence pages rank below the sink-dominated page and receive limited token budgets, most answer-region spatial support survives, as shown in the last column. This is consistent with page-to-token selection: evidence tokens need not individually outrank background tokens on other pages. The final failed answer may involve visual recognition or counting limitations of the VLM backbone.
Q5: Correct the paragraph for Qwen3-VL-8B following Table 8.
A5: The paragraph following Table 8 mistakenly repeats the official test-split discussion from Table 7. We will replace it with the following analysis of transfer to Qwen3-VL-8B.
Overall, HiToc remains effective when transferred to a larger backbone. On DUDE and MP-DocVQA, HiToc still achieves strong performance under approximately 20% of the original token budget. Specifically, on DUDE, HiToc obtains an ANLS of 0.5877, retaining about 98.2% of the baseline performance (0.5987). On MP-DocVQA, it achieves 0.8509 ANLS and 74.86% accuracy, corresponding to about 98.3% and 97.0% of the baseline performance, respectively. On MMLongBench-Doc, HiToc still maintains strong performance under a merely 3.6% token budget, retaining 97.0% of the baseline Accuracy and 98.2% of the baseline F1-score.
A notable observation is that, on the 8B backbone, introducing APTA leads to more degradation compared to BBTM alone. Compared with BBTM, which removes redundant background regions while preserving most semantic content, APTA further reduces the token budget from roughly 80% to 20% through hierarchical page-wise pruning. Such structured selection may mildly affect continuous semantic patterns or long-range evidence utilization that the larger backbone can exploit. Nevertheless, the performance of APTA-based variants remains strong overall. Moreover, the final HiToc model recovers part of the performance drop after APTA on DUDE and MP-DocVQA and improves MMLongBench-Doc F1, supporting the effectiveness of the proposed layer-adaptive KV-Cache compression in dynamically preserving useful key-value pairs according to both decoding context and layer depth.
On MMDA, HiToc retains 98.3% of baseline ACNLS (0.5950→0.5851) and 94.1% of AIC-Acc (70.10%→65.98%). The larger reduction in AIC-Acc may reflect the metrics’ different sensitivities: ACNLS gives partial credit for lexical overlap, whereas AIC-Acc evaluates answer correctness. Compression may occasionally remove a decisive detail, such as a number, entity, or qualifier, causing an otherwise similar answer to be judged incorrect. This can produce a larger drop in AIC-Acc while leaving ACNLS relatively stable.
Response to Reviewer 1TNp – Part 1
We thank the reviewer for the constructive feedback. We address each concern below and will incorporate all additions into the revision.
Q1: More details on KV storage and the execution order during decoding.
A1: At each decoding step: (1) Rescore the retained visual candidates using q_t’s full-candidate attention, aggregated by the average across heads. (2) Select the top-k_d% visual KV entries for each stage. (3) Gather the selected entries from each layer’s own candidate cache; only previously inactive entries selected at the current step are reactivated, while all unselected visual entries remain inactive. (4) Compute attention using the selected visual KV and all text KV, including the newly appended text tokens.
Inactive entries remain in each layer’s GPU-resident candidate cache, restricted to tokens surviving prefill compression. Both candidate storage and active buffers are included in profiled memory, and scoring, selection, gathering, and attention are included in LLM-forward latency. We will clarify this attention-based selection and subsequent KV-access sequence in Section 3.3.
The pseudo codes are listed as follows. Cache_l is layer l's candidate KV cache; Gather reads the selected visual entries and all text entries without adding text KV again.
For physical sizes, let Cℓ denote the number of retained visual candidates at layer ℓ, T the number of cached text tokens, and Ag the number of active visual tokens in stage g. Each K or V tensor has shape B×Hkv×n×d, where B is the batch size, Hkv the number of KV heads, n the corresponding cache length, and d the head dimension. Let b denote bytes per element.
The candidate K/V pair at layer ℓ occupies Mℓcand=2BHkv(Cℓ+T)db. A separately materialized active K/V pair for stage g adds Mℓ,gactive=2BHkv(Ag+T)db. Both expressions give storage in bytes. We will include this pseudocode and per-layer storage specification in the revision.
Q2: Can you compare methods on quality versus measured end-to-end latency, FLOPs, and memory curves, rather than only matching final active-token counts?
A2: Our FastV, DivPrune and SparseVLM implementations prune at the same decoder layer as APTA (layer 20), so they also receive question-conditioned processing before selection. HiToc’s higher quality in Table 2 therefore cannot simply be attributed to later pruning.
HiToc additionally applies BBTM before the LLM, so fewer visual tokens enter the LLM than in the compared LLM-stage pruning methods. It further reduces the active visual KV subset during decoding, targeting costs across multiple inference stages. Different compression stages and architectural designs make exact matching of latency, memory, or TFLOPs difficult. Comparing end-to-end performance at matched final token-pruning ratios is therefore a reasonable and commonly used evaluation protocol.
Q3: Results with a single active-budget setting across all datasets. Which independent data were used to choose the pruning layers and other hyperparameters?
A3: We additionally evaluated full HiToc on MMLongBench-Doc with k_d=45%, reducing the active visual KV budget to 739.9 tokens, approximately one page (768 tokens). This extends the roughly one-page active-budget setting to all four benchmarks. Under this aggressive compression setting—retaining only 2.03% of the original visual tokens —HiToc still achieves 26.53% accuracy and 22.49% F1, retaining 95.57% and 94.89% of the uncompressed baseline's accuracy and F1, respectively.
We held out 10% of the DUDE training set as an independent development set to select the pruning layers and other hyperparameters. The trends observed during tuning were consistent with the ablation results on the DUDE validation set shown in Figure 3. We will clarify this parameter-selection protocol in the revised manuscript.