- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes HTR-VLM, a hybrid visual-token reduction scheme for LLaVA-style VLMs that treats shallow and deep LLM layers differently. In shallow layers (3–10), attention keeps all visual tokens, while the FFN is evaluated on a average-pooled grid and its update is bilinearly up-sampled back through the residual connection (FTP). A subset of tokens selected by last-text-token attention (with raw vision features added to the keys at layer 4) and by feature diversity bypasses pooling and receives the exact FFN update (STR). In deep layers, tokens are pruned at two layers (11 and 20) chosen from valleys of a top- attention-mass curve (DATP). The model is obtained by one epoch of LLaVA SFT (665K samples) with HTR in place.
Validation is on LLaVA-1.5-7B over seven benchmarks against eight token-reduction baselines whose numbers are copied from their papers. At 128/64/32 "retained" tokens HTR keeps 98.0%/98.0%/97.2% of full-model performance (best baseline DivPrune: 96.9%/94.4%/91.3%).
Critical strengths
S1. FTP is a simple, plausible way to cut shallow-layer cost without dropping context. Instead of discarding tokens early, attention stays at full length and only the FFN is evaluated on pooled tokens, with the update returned through the residual. This keeps positions and cross-token interaction intact. In the ablation it accounts for almost all of the gain (average $51.1 \rightarrow 68.4 $ at 64 tokens), and the pooling/up-sampling ablation (average + bilinear best) is sensible. This is the one idea in the paper that could transfer to other models.
S2. Strong accuracy retention under aggressive budgets, with measured latency. Reported retention stays at 97–98% from 128 down to 32 tokens, whereas baselines degrade to 84–91% at 32. The paper also reports wall-clock latency and FLOPs on real hardware, not only token counts. If the comparison were confirmed to be fair, this would be a meaningful practical result.
Critical weaknesses
W1. The main comparison is confounded by training and by budget definition. HTR-VLM is obtained by one epoch of SFT on the 665K LLaVA data, while most baselines in Table 1 (FastV, SparseVLM, VisionZip, DART, DivPrune, …) are training-free, and all baseline numbers are copied from their papers. The ablation baseline in Table 3 (avg. 51.1) is numerically identical to training-free FastV at 64 tokens in Table 1, so the reported +33.9% from FTP likely mixes the effect of FTP with the effect of fine-tuning.
The token budget is also not like-for-like. Under HTR, layers 0–10 attend over all 576 visual tokens, and pruning to 64 (then to 4) happens only at layers 11 and 20. Many baselines retain 64 tokens from the LLM input onward. "Average retained tokens" is not defined, and FLOPs/latency are reported only for HTR. Without a training-matched comparison and an equal-FLOPs comparison, the central claim — better accuracy at the same budget — is not established. This is the main reason for my rating.
W2. Limited novelty and evidence of generality. Beyond FTP, STR combines FastV-style attention and DivPrune-style diversity scores, and DATP reduces to two fixed pruning layers. Those layers were chosen from attention statistics on MMBench and TextVQA, which are also evaluation benchmarks, and Table 7 selects (11, 20) on the same benchmarks. Everything is shown on one model (LLaVA-1.5-7B, fixed 576-token grid), with no high-resolution or dynamic-resolution model, no video, and no variance across runs. STR and DATP each add under 1 point on average. It is therefore unclear whether the recipe generalizes or is tuned to this one setting.
Questions for authors
- Training-matched comparison. How does HTR perform without the extra SFT epoch, i.e., applied training-free to the released LLaVA-1.5-7B? Conversely, how do strong baselines (e.g., VisionZip, DivPrune, PDrop) perform after the same one-epoch SFT? Is the Table 3 baseline row trained or training-free?
- Budget definition and compute. How exactly is "average 64 retained tokens" computed, given full-length attention in layers 0–10 and pruning at layers 11 and 20? Please report prefill FLOPs and latency for the baselines on the same L20X setup, so methods can be compared at equal compute rather than at nominal token counts.
Q1. Training setting and additional comparisons
HTR-VLM is training-based, not a training-free plug-in. Compression is incorporated into one epoch of SFT on the 665K LLaVA samples to adapt to approximate FFN updates and token pruning. The training-based baselines also use one SFT epoch, but follow their reported configurations rather than an identical-SFT protocol.
For comparison with VisionZip++, Avg. below excludes MME. HTR-VLM scores 68.5 / 68.6 / 68.1 at 128 / 64 / 32 tokens. It is comparable to VisionZip++ at 128 tokens (68.5 vs. 68.6), and leads by 0.7 and 1.7 points at 64 and 32 tokens, respectively.
From 128 to 32 tokens, HTR-VLM drops only 0.4 points, versus 2.2 points for VisionZip++ (68.6 to 66.4); from 64 to 32, the drop is 0.5 points. At 32 tokens, it retains 97.6% of the full-model six-benchmark average, versus 95.2% for VisionZip++.
These results support accuracy retention under the reported configurations, but do not isolate design gains from SFT. We have not reported training-free HTR-VLM or baseline reruns with our exact SFT recipe. We will label training settings and clarify the Table 3 baseline's training status; its reported +33.9% gain cannot be attributed solely to FTP without a training-matched baseline.
Q2. Token-budget definition and compute
The 64-token budget denotes visual tokens remaining after the first prune at layer 11: 576−512=64. Table 1's 88.9% reduction is (576−64)/576. This is a post-first-pruning budget, not a layer-averaged sequence length. We will correct the term "average retained tokens."
The visual-token schedule is:
For the shallow FFN, c=6 and Ma=Mb=0.04∣Nv∣ give:
62576+2×0.04×576=62.08≈62.
This reduces FFN computation without shortening shallow attention.
Likewise, the 128-token setting removes 448 then 114 tokens, leaving 14; the 32-token setting removes 544 then 30, leaving 2.
Table 2 reports prefill FLOPs of 1.67T / 1.17T / 0.96T at 128 / 64 / 32 tokens, versus 7.63T for the full model. Accounting must separate attention and FFN token counts: counting all operations in layers 3–19 at 64 tokens omits full-length shallow attention. We will verify these values and correct them if needed.
Table 2 also reports end-to-end latency on the same NVIDIA L20X:
Table 1 baselines are from their papers at the same nominal token budgets; we do not report their FLOPs or latency on the same L20X. Since token schedules and pruning depths differ, equal nominal token counts do not imply equal compute. We will clarify this limitation and not claim superiority at matched FLOPs or latency.
Comparison with Other SOTA Methods
LLaVA-1.5-7B results. To compare with VisionZip++, Avg. excludes MME and averages the other six benchmarks; Rel. uses the corresponding full-model average. "--" denotes missing results; bold marks the best available results within each budget.
Full Model: All 576 Tokens (100%)
Retain 128 Tokens (77.8% token reduction)
Retain 64 Tokens (88.9% token reduction)
Retain 32 Tokens (94.4% token reduction)