acceptodds
Under review as a conference paper at ICLR 2027

Shallow Pool, Deep Prune: Accelerating Vision-Language Models via Hybrid Token Reduction

Abstract

Vision Language Models (VLMs) process hundreds or even thousands of visual tokens to perform complex visual understanding and reasoning tasks. However, such long input sequences result in substantial computational overhead during inference. While a range of token reduction approaches have been proposed to accelerate VLMs, they generally encounter early-stage context loss, inadequate salient token retention and suboptimal pruning allocation, leading to significant performance degradation. To address these issues, we propose a novel hybrid token reduction method, termed HTR-VLM, motivated by the depth-dependent variation in compression objectives across VLM layers. Specifically, we introduce the FFN Token Pooling with Residual Approximation (FTP) module, which leverages residual connections to approximate full image-feature updates with pooled tokens, reducing shallow FFN computation while preserving full context. Building on this, Salient Token Retention with Multi-Criteria Scoring (STR) selects and preserve important tokens for full computation by integrating multiple cues. Finally, a Depth-Aware Token Pruning (DATP) mechanism is adopted to enable more precise token reduction in deeper layers. Extensive experiments demonstrate that HTR-VLM substantially reduces computational overhead and maintains higher model performance compared to state-of-the-art token reduction methods. We will release our code upon paper acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Vision Language Models, Visual Token Pruning, Efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gJ7FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes HTR-VLM, a hybrid visual-token reduction scheme for LLaVA-style VLMs that treats shallow and deep LLM layers differently. In shallow layers (3–10), attention keeps all visual tokens, while the FFN is evaluated on a c×cc\times c average-pooled grid and its update is bilinearly up-sampled back through the residual connection (FTP). A subset of tokens selected by last-text-token attention (with raw vision features added to the keys at layer 4) and by feature diversity bypasses pooling and receives the exact FFN update (STR). In deep layers, tokens are pruned at two layers (11 and 20) chosen from valleys of a top-kk attention-mass curve (DATP). The model is obtained by one epoch of LLaVA SFT (665K samples) with HTR in place.

Validation is on LLaVA-1.5-7B over seven benchmarks against eight token-reduction baselines whose numbers are copied from their papers. At 128/64/32 "retained" tokens HTR keeps 98.0%/98.0%/97.2% of full-model performance (best baseline DivPrune: 96.9%/94.4%/91.3%).

Critical strengths

S1. FTP is a simple, plausible way to cut shallow-layer cost without dropping context. Instead of discarding tokens early, attention stays at full length and only the FFN is evaluated on pooled tokens, with the update returned through the residual. This keeps positions and cross-token interaction intact. In the ablation it accounts for almost all of the gain (average $51.1 \rightarrow 68.4 $ at 64 tokens), and the pooling/up-sampling ablation (average + bilinear best) is sensible. This is the one idea in the paper that could transfer to other models.

S2. Strong accuracy retention under aggressive budgets, with measured latency. Reported retention stays at 97–98% from 128 down to 32 tokens, whereas baselines degrade to 84–91% at 32. The paper also reports wall-clock latency and FLOPs on real hardware, not only token counts. If the comparison were confirmed to be fair, this would be a meaningful practical result.

Critical weaknesses

W1. The main comparison is confounded by training and by budget definition. HTR-VLM is obtained by one epoch of SFT on the 665K LLaVA data, while most baselines in Table 1 (FastV, SparseVLM, VisionZip, DART, DivPrune, …) are training-free, and all baseline numbers are copied from their papers. The ablation baseline in Table 3 (avg. 51.1) is numerically identical to training-free FastV at 64 tokens in Table 1, so the reported +33.9% from FTP likely mixes the effect of FTP with the effect of fine-tuning.

The token budget is also not like-for-like. Under HTR, layers 0–10 attend over all 576 visual tokens, and pruning to 64 (then to 4) happens only at layers 11 and 20. Many baselines retain 64 tokens from the LLM input onward. "Average retained tokens" is not defined, and FLOPs/latency are reported only for HTR. Without a training-matched comparison and an equal-FLOPs comparison, the central claim — better accuracy at the same budget — is not established. This is the main reason for my rating.

W2. Limited novelty and evidence of generality. Beyond FTP, STR combines FastV-style attention and DivPrune-style diversity scores, and DATP reduces to two fixed pruning layers. Those layers were chosen from attention statistics on MMBench and TextVQA, which are also evaluation benchmarks, and Table 7 selects (11, 20) on the same benchmarks. Everything is shown on one model (LLaVA-1.5-7B, fixed 576-token grid), with no high-resolution or dynamic-resolution model, no video, and no variance across runs. STR and DATP each add under 1 point on average. It is therefore unclear whether the recipe generalizes or is tuned to this one setting.

Questions for authors
  1. Training-matched comparison. How does HTR perform without the extra SFT epoch, i.e., applied training-free to the released LLaVA-1.5-7B? Conversely, how do strong baselines (e.g., VisionZip, DivPrune, PDrop) perform after the same one-epoch SFT? Is the Table 3 baseline row trained or training-free?
  2. Budget definition and compute. How exactly is "average 64 retained tokens" computed, given full-length attention in layers 0–10 and pruning at layers 11 and 20? Please report prefill FLOPs and latency for the baselines on the same L20X setup, so methods can be compared at equal compute rather than at nominal token counts.
1 comment

Q1. Training setting and additional comparisons

HTR-VLM is training-based, not a training-free plug-in. Compression is incorporated into one epoch of SFT on the 665K LLaVA samples to adapt to approximate FFN updates and token pruning. The training-based baselines also use one SFT epoch, but follow their reported configurations rather than an identical-SFT protocol.

For comparison with VisionZip++, Avg. below excludes MME. HTR-VLM scores 68.5 / 68.6 / 68.1 at 128 / 64 / 32 tokens. It is comparable to VisionZip++ at 128 tokens (68.5 vs. 68.6), and leads by 0.7 and 1.7 points at 64 and 32 tokens, respectively.

From 128 to 32 tokens, HTR-VLM drops only 0.4 points, versus 2.2 points for VisionZip++ (68.6 to 66.4); from 64 to 32, the drop is 0.5 points. At 32 tokens, it retains 97.6% of the full-model six-benchmark average, versus 95.2% for VisionZip++.

These results support accuracy retention under the reported configurations, but do not isolate design gains from SFT. We have not reported training-free HTR-VLM or baseline reruns with our exact SFT recipe. We will label training settings and clarify the Table 3 baseline's training status; its reported +33.9% gain cannot be attributed solely to FTP without a training-matched baseline.

Q2. Token-budget definition and compute

The 64-token budget denotes visual tokens remaining after the first prune at layer 11: 576−512=64576-512=64. Table 1's 88.9% reduction is (576−64)/576(576-64)/576. This is a post-first-pruning budget, not a layer-averaged sequence length. We will correct the term "average retained tokens."

The visual-token schedule is:

  • Layers 0–2: attention and FFN both use 576 tokens.
  • Layers 3–10: attention uses 576 tokens; the FFN processes about 62 token equivalents.
  • Layers 11–19: attention and FFN both use 64 tokens.
  • Layers 20–31: 4 tokens remain after removing another 60.

For the shallow FFN, c=6c=6 and Ma=Mb=0.04∣Nv∣M^{a}=M^{b}=0.04|N_v| give:

57662+2×0.04×576=62.08≈62.\frac{576}{6^2}+2\times0.04\times576=62.08\approx62.

This reduces FFN computation without shortening shallow attention.

Likewise, the 128-token setting removes 448 then 114 tokens, leaving 14; the 32-token setting removes 544 then 30, leaving 2.

Table 2 reports prefill FLOPs of 1.67T / 1.17T / 0.96T at 128 / 64 / 32 tokens, versus 7.63T for the full model. Accounting must separate attention and FFN token counts: counting all operations in layers 3–19 at 64 tokens omits full-length shallow attention. We will verify these values and correct them if needed.

Table 2 also reports end-to-end latency on the same NVIDIA L20X:

  • MMBench: 791.35 ms (full model) versus 136.00 / 116.61 / 99.44 ms at 128 / 64 / 32 tokens.
  • TextVQA: 344.49 ms (full model) versus 114.47 / 105.55 / 102.92 ms at 128 / 64 / 32 tokens.

Table 1 baselines are from their papers at the same nominal token budgets; we do not report their FLOPs or latency on the same L20X. Since token schedules and pruning depths differ, equal nominal token counts do not imply equal compute. We will clarify this limitation and not claim superiority at matched FLOPs or latency.

Comparison with Other SOTA Methods

LLaVA-1.5-7B results. To compare with VisionZip++, Avg. excludes MME and averages the other six benchmarks; Rel. uses the corresponding full-model average. "--" denotes missing results; bold marks the best available results within each budget.

Full Model: All 576 Tokens (100%)

MethodVQAv2GQASQA-IMGTextVQAPOPEMMEMMBAvg.Rel.
LLaVA-1.5-7B78.561.969.558.285.91506.564.769.8100.0%

Retain 128 Tokens (77.8% token reduction)

MethodVQAv2GQASQA-IMGTextVQAPOPEMMEMMBAvg.Rel.
VisionZip++77.460.168.257.884.9--63.468.698.4%
PDrop--57.161.656.6----61.6----
HTR-VLM (Ours)77.061.367.655.986.11462.462.968.598.1%

Retain 64 Tokens (88.9% token reduction)

MethodVQAv2GQASQA-IMGTextVQAPOPEMMEMMBAvg.Rel.
VisionZip++76.658.968.357.083.7--62.667.997.2%
PDrop--47.569.056.6----58.8----
DivePrune--55.367.9--83.11330.058.2----
HTR-VLM (Ours)76.760.868.355.586.01453.264.368.698.3%

Retain 32 Tokens (94.4% token reduction)

MethodVQAv2GQASQA-IMGTextVQAPOPEMMEMMBAvg.Rel.
VisionZip++74.257.068.856.080.9--61.566.495.2%
HTR-VLM (Ours)76.260.068.254.885.61427.963.868.197.6%
Reviewer_Nvn8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes HTR-VLM, a hybrid visual-token reduction scheme that treats shallow and deep LLM layers differently. In layers 3–10, attention still runs over all visual tokens, but the FFN is applied only to average-pooled tokens plus a few salient tokens, and the pooled updates are up-sampled back into the residual stream (FTP + STR). In deeper layers, tokens are explicitly pruned at two depths chosen from an attention-concentration curve (DATP). The model is obtained with one epoch of LLaVA-1.5 SFT with the reduction in the loop. On LLaVA-1.5-7B, it reports the best average over seven benchmarks at 128, 64 and 32 tokens (e.g., 97.2% of the full model at 32 tokens).

Critical strengths
  1. Keeping full attention in shallow layers and approximating only the FFN update from pooled tokens is a simple way to avoid early, irreversible token dropping.
  2. Accuracy holds up well under aggressive budgets: at 32 tokens, HTR-VLM averages 68.6 vs. 64.4 for DivPrune (Table 1), and it barely changes from 128 to 32 tokens.
  3. The ablations cover the main design choices: each module (Table 3), the pooling/up-sampling operators (Table 4), the salient-token criteria (Table 5) and the pruning layers (Table 7).
Critical weaknesses
  1. The main comparison is confounded by training. HTR-VLM is fine-tuned for a full epoch of LLaVA SFT, while the Table 1 baselines are mostly training-free numbers copied from their papers; in fact, the "no module" row of Table 3 is exactly FastV's training-free 64-token row.
  2. The compute budgets are not matched. The "64-token" setting keeps all 576 tokens in layers 0–2 and in attention up to layer 10, which by my count costs more than a static 128-token baseline, and the FLOPs in Table 2 appear to assume that layers 3–10 run on the reduced tokens.
  3. Beyond FTP, the other modules add little and are fragile. STR and DATP add +0.2 and +0.6 (Table 3), while shifting the second pruning layer by one or two layers costs 2–3 points (Table 7), which suggests the layers were tuned on the evaluation benchmarks.
  4. The closest prior work, ProxyV (Wu et al., 2025a), which also keeps the full token sequence and routes heavy computation through pooled proxy tokens, is cited only in passing and not compared.
  5. The scope is narrow: all experiments use LLaVA-1.5-7B with a fixed 24×24 grid, and latency is compared only against the dense model, not against any baseline.
Questions for authors
  1. How does HTR-VLM perform when applied training-free to the released LLaVA-1.5-7B, and how do the strongest baselines perform after the same one-epoch SFT?
  2. Can you report per-budget FLOPs that count full-length attention in layers 0–10, and end-to-end latency for the baselines on the same GPU?
  3. Were the pruning layers (11/20) chosen on the evaluation benchmarks, and how does HTR-VLM do on a higher-resolution or newer backbone (e.g., LLaVA-NeXT or Qwen2.5-VL)?
2 comments

Q1. Training-based setting and additional comparisons

HTR-VLM is a training-based compression framework, not a training-free plug-in. Compression is incorporated into one epoch of LLaVA SFT to adapt the model to approximate FFN updates and token pruning. The tables below compare training-based methods under their reported configurations.

HTR-VLM shows limited degradation as compression increases. Its six-benchmark average excluding MME is 68.5, 68.6, and 68.1 at 128, 64, and 32 tokens, respectively. From 128 to 32 tokens, its average drops by only 0.4 points, versus 2.2 points for VisionZip++ (68.6 to 66.4). From 64 to 32 tokens, the drop is only 0.5 points.

HTR-VLM is comparable to VisionZip++ at 128 tokens (68.5 vs. 68.6), while its advantage increases to 0.7 and 1.7 points at 64 and 32 tokens. At 32 tokens, it retains 97.6% of the full-model six-benchmark average, versus 95.2% for VisionZip++. These results support its stability under aggressive compression in the evaluated settings. Comparisons follow each method's reported training configuration, not an identical-SFT protocol.

Q2. FLOPs calculation and end-to-end latency

Table 2 reports prefill FLOPs of 1.67T, 1.17T, and 0.96T at 128, 64, and 32 retained tokens, respectively, versus 7.63T for all 576 tokens. We sum layer-wise costs using each layer's token count.

  • Layers 0–2 are counted at 576 tokens.
  • Layers 3–19 are counted at the retained budget of that setting: 128, 64, or 32 tokens.

The shallow FFN widths are controlled by cc, MaM^{a}, and MbM^{b}:

\frac{576}{4^{2}} + 2 \times 0.08 \times 576 &= 128.16 \approx 128, \\ \frac{576}{6^{2}} + 2 \times 0.04 \times 576 &= 62.08 \approx 62, \\ \frac{576}{8^{2}} + 2 \times 0.02 \times 576 &= 32.04 \approx 32. \end{aligned}$$ **After layer 20**, we apply the second pruning step stated in the implementation details: - **128-token setting:** removing 448 tokens and then 114 tokens leaves 14 tokens. - **64-token setting:** removing 512 tokens and then 60 tokens leaves 4 tokens. - **32-token setting:** removing 544 tokens and then 30 tokens leaves 2 tokens. This schedule is used to compute the FLOPs in Table 2. The end-to-end latency on the same NVIDIA L20X is also reported in Table 2: - **MMBench:** latency decreases from **791.35 ms** to **136.00 / 116.61 / 99.44 ms** at 128 / 64 / 32 tokens, respectively. - **TextVQA:** latency decreases from **344.49 ms** to **114.47 / 105.55 / 102.92 ms** at 128 / 64 / 32 tokens, respectively. The baselines in Table 1 are cited from their original papers under the same retained-token counts. We align the comparisons by the retained-token counts in Table 1 and report the corresponding FLOPs in Table 2. ### Q3. Pruning-depth selection and scope Yes. We computed attention mass using all samples from multiple evaluation benchmarks and observed broadly consistent depth-wise trends across datasets. The final pruning depths (11, 20) were selected using these attention-mass profiles together with evaluation scores. We have not yet conducted comparisons at higher resolutions or on other backbones. ### Comparison with Other SOTA Methods LLaVA-1.5-7B results under three token budgets. For comparison with VisionZip++, **Avg.** is computed over the six benchmarks excluding MME; **Rel.** is relative to the full-model average over the same six benchmarks. "--" denotes missing results; **bold** marks the best available results within each token budget. **Full Model: All 576 Tokens (100%)** |Method|VQAv2|GQA|SQA-IMG|TextVQA|POPE|MME|MMB|Avg.|Rel.| |:---|---:|---:|---:|---:|---:|---:|---:|---:|---:| |LLaVA-1.5-7B|78.5|61.9|69.5|58.2|85.9|1506.5|64.7|69.8|100.0%| **Retain 128 Tokens (77.8% token reduction)** |Method|VQAv2|GQA|SQA-IMG|TextVQA|POPE|MME|MMB|Avg.|Rel.| |:---|---:|---:|---:|---:|---:|---:|---:|---:|---:| |VisionZip++|**77.4**|60.1|**68.2**|**57.8**|84.9|--|**63.4**|**68.6**|**98.4%**| |PDrop|--|57.1|61.6|56.6|--|--|61.6|--|--| |**HTR-VLM (Ours)**|77.0|**61.3**|67.6|55.9|**86.1**|**1462.4**|62.9|68.5|98.1%| **Retain 64 Tokens (88.9% token reduction)** |Method|VQAv2|GQA|SQA-IMG|TextVQA|POPE|MME|MMB|Avg.|Rel.| |:---|---:|---:|---:|---:|---:|---:|---:|---:|---:| |VisionZip++|76.6|58.9|68.3|**57.0**|83.7|--|62.6|67.9|97.2%| |PDrop|--|47.5|**69.0**|56.6|--|--|58.8|--|--| |DivePrune|--|55.3|67.9|--|83.1|1330.0|58.2|--|--| |**HTR-VLM (Ours)**|**76.7**|**60.8**|68.3|55.5|**86.0**|**1453.2**|**64.3**|**68.6**|**98.3%**| **Retain 32 Tokens (94.4% token reduction)** |Method|VQAv2|GQA|SQA-IMG|TextVQA|POPE|MME|MMB|Avg.|Rel.| |:---|---:|---:|---:|---:|---:|---:|---:|---:|---:| |VisionZip++|74.2|57.0|**68.8**|**56.0**|80.9|--|61.5|66.4|95.2%| |**HTR-VLM (Ours)**|**76.2**|**60.0**|68.2|54.8|**85.6**|**1427.9**|**63.8**|**68.1**|**97.6%**|

Thank you for the additional comparisons and for clarifying how the pruning depths were selected. The comparison with VisionZip++ strengthens the empirical case for HTR-VLM at the 32-token setting.

My main concern remains the novelty of the shallow-layer design. The ablations show that FTP is useful within HTR-VLM, but I still find it difficult to assess what it adds beyond the pooled-proxy computation in ProxyV. A direct comparison at comparable compute would help clarify this. I also remain concerned about generality. All results use LLaVA-1.5-7B with a fixed 24×24 grid, and the pruning depths were selected partly using evaluation scores. It therefore remains unclear whether the same design choices transfer to a newer backbone or higher-resolution inputs.

Reviewer_bGeQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HTR-VLM, a hybrid visual-token reduction framework for accelerating vision-language models. In shallow layers, FFN Token Pooling with Residual Approximation (FTP) computes updates on pooled tokens and interpolates them back to the full sequence. Salient Token Retention (STR) preserves full FFN computation for selected tokens using attention and feature-diversity criteria. Depth-Aware Token Pruning (DATP) subsequently removes tokens at layers chosen from an attention-concentration profile. The framework is integrated into supervised fine-tuning. Experiments on LLaVA-1.5-7B across seven benchmarks show substantial computational savings and relatively stable performance under aggressive compression.

Critical strengths
  1. Intuitive design that separates context preservation from computation reduction. In shallow layers, the method approximates FFN updates with pooled tokens while keeping all visual tokens in the sequence, so attention in subsequent layers can still access the full visual input. This separation is intuitive and provides a reasonable starting point for applying different compression strategies at different depths.
  2. Stable performance under high compression. Performance remains relatively stable as the token budget decreases. With 32 retained tokens, the model still keeps 97.2% of the full model's average performance.
Critical weaknesses
  1. The depth-dependent design principle could be better supported by analysis. Non-salient tokens are updated through pooled FFN approximations, yet correlated inputs do not necessarily yield similar updates after a nonlinear FFN. The paper does not quantify this approximation error or how the relative sensitivity to pooling versus pruning varies with depth, so the "shallow pool, deep prune" principle is mainly supported by empirical results.
  2. The rationale behind DATP could be further validated. The paper assumes that a lower Top-k attention mass indicates higher redundancy and thus greater robustness to pruning. However, diffuse attention may also be spread over complementary rather than redundant content, and it would be helpful to show that this statistic correlates with the actual performance drop caused by pruning.
  3. Lack of comparison with closely related methods. ProxyV [1] also reduces computation through proxy tokens while updating the full visual sequence, and recent methods such as VisPruner [2] and CDPruner [3] report strong results under similar settings. A comparison and discussion with these methods would help clarify the contribution of HTR-VLM.
  4. Insufficient details for reproducibility. The paper does not specify the trainable parameters, optimization settings, or how the compression operations are applied during fine-tuning, and the code is not yet available. Providing these details would make the reported results easier to reproduce.
Questions for authors
  1. (Pruning criterion) Could the authors provide evidence that a lower Top-k attention mass corresponds to greater robustness to pruning, for example by relating this statistic to the actual performance drop when pruning at different layers?
  2. (Comparison with related work) Could the authors compare HTR-VLM with closely related methods such as ProxyV, VisPruner, and CDPruner? Since HTR-VLM involves fine-tuning, a comparison under the same training budget would also help separate the effect of the proposed design from that of fine-tuning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.