- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that in many layers of a multimodal LLM, the self-attention or MLP computation on visual tokens contributes little to the answer, and that where this happens depends on the input and the module type. AdaVSkip attaches a lightweight router to every attention and MLP module that reads the prompt's text hidden states and decides, during prefill, whether visual tokens take part, with the backbone frozen. The routers are trained on targets from a per-module KL necessity score and then refined with GRPO. On three backbones and eight benchmarks, the paper claims full-model accuracy at much lower prefill cost and complementarity with token pruning: with CDPruner on LLaVA-NeXT it keeps 97.2% of average performance at 6.9% of the FLOPs.
Critical strengths
-
AdaVSkip degrades far more gracefully than fixed vertical skipping as the budget tightens. It beats ShortV and VTW at every skipping budget on all three backbones, and the margin widens at the most aggressive setting, where on LLaVA-1.5 it retains 95.1% of average performance against 85.8% for ShortV. This is direct evidence that input-dependent, module-wise decisions pay off where they matter most.
-
The method is cheap, leaves the backbone untouched, and plugs into existing token pruners. Router training and offline profiling each finish within hours per backbone, the trained routers are reused on pruned token sets without retraining, and the FLOPs accounting is explicit. The finding that attention and MLP differ in redundancy is well supported and broadly useful.
Critical weaknesses
-
The hybrid results do not yet isolate what adaptive routing contributes. Section 4.1 and the conclusion credit adaptive routing for the hybrid's gains over token compression alone. That requires holding the retained token count fixed and changing only the skipping rule. Instead, at the same equivalent budget the hybrid keeps more visual tokens than the standalone pruner and then skips modules (twice as many in the one Qwen3-VL configuration described), and for the LLaVA backbones the hybrid's token count and skipping budget are not reported. A fixed rule is already strong, with ShortV within two points of AdaVSkip at the roughly half-FLOPs budget on both LLaVA backbones, so much of the gain could come from a token and depth trade-off that any reasonable skipping rule would provide. As it stands, the evidence shows that pruning and vertical skipping combine well, not that adaptive routing makes the combination work.
-
The efficiency evidence supports the FLOPs reduction but not yet a general wall-clock speedup. Latency is measured on a single benchmark, without stating hardware, batch size or output length, and without the two closest competitors, CDPruner alone and ShortV. Because routing acts only during prefill, decode cost depends on whether generated tokens still attend to visual keys and values in skipped layers, which the paper leaves open; VisionZip and MMTok also show identical prefill but clearly different end-to-end times, without explanation. Routers are also trained separately per budget with no seeds or variance reported, so the training-component ablation gains, all below one point, cannot be told apart from noise. The practical speedup and the value of individual training components therefore remain suggestive.
Questions for authors
- Can you report the same pruner at your hybrid's retained token count combined with ShortV or the Fixed-Budget variant of Table 5, at matched FLOPs, and state the token count and skipping budget for every hybrid row?
- What hardware, batch size and generation length does Table 6 use, and how do CDPruner alone and ShortV compare under that setup, including batched inference where samples take different routing paths?
- How much do results vary across seeds for both training stages, particularly for the sub-point steps of the training-component ablation?
- What does AdaVSkip add beyond published methods that learn layer or module skipping for visual tokens in MLLMs, such as gamma-MoD (Luo et al.), p-MoD (Zhang et al.), Skip-Vision (Zeng et al.) and AdaLLaVA (Xu et al.)? Most fine-tune the backbone, so a discussion in Section 2.3 would suffice.
If the hybrid still clearly beats a token pruner plus ShortV or Fixed-Budget routing at matched token count and FLOPs, and the latency setup is fully specified, I would raise my rating; if that control closes most of the gap, I would lower it.
3 comments
Question 2: What hardware, batch size and generation length does Table 6 use, and how do CDPruner alone and ShortV compare under that setup, including batched inference where samples take different routing paths?
Response: Thank you for raising this question. The latency results in Table 6 were measured on MME using a single NVIDIA RTX 5090 GPU with a batch size of 1. Each response consists of two generated tokens: an answer token (“yes” or “no”) followed by an end-of-sequence token. We compare AdaVSkip with ShortV and evaluate the effect of adding AdaVSkip to CDPruner. Compared with ShortV, AdaVSkip reduces end-to-end latency by 14.3 ms and improves the MME score by 9.4 points. Compared with CDPruner alone, CDPruner + AdaVSkip improves the MME score by 48.6 points at comparable end-to-end latency.
| Method | End-to-end latency (ms) ↓ | MME ↑ |
|---|---|---|
| ShortV (N=19) | 123.6 (1.57×) | 1526.0 |
| AdaVSkip(K=16) | 109.3 (1.78×) | 1535.4 |
| CDPruner | 63.5 (3.06×) | 1425.3 |
| CDPruner + AdaVSkip(K=16) | 64.6 (3.01×) | 1473.9 |
During batched inference, each sample makes independent attention and MLP routing decisions at every layer. For attention, we form up to two sub-batches: samples retaining visual attention process both text and visual tokens, while those skipping it process only text tokens. Different sequence lengths are accommodated by padding within each sub-batch. For the MLP, we regroup samples according to their MLP routing decisions. The keep group processes both text and visual tokens, while the skip group processes only text tokens. We then write the updates back to their original positions, leaving the skip group’s visual states unchanged. We report end-to-end speedups relative to vanilla LLaVA-NeXT-7B at the same batch size. At B=8, AdaVSkip achieves a 2.10× speedup compared with 1.78× for ShortV. CDPruner + AdaVSkip reaches 6.6× compared with 6.17× for CDPruner alone. As batch size increases, AdaVSkip achieves greater end-to-end speedups, highlighting its potential for efficient batched inference.
| Method | B=1 | B=2 | B=4 | B=8 |
|---|---|---|---|---|
| ShortV (N=19) | 1.57× | 1.70× | 1.77× | 1.78× |
| AdaVSkip(K=16) | 1.78× | 1.94× | 2.05× | 2.10× |
| CDPruner | 3.06× | 4.65× | 5.76× | 6.17× |
| CDPruner + AdaVSkip(K=16) | 3.01× | 4.59× | 5.84× | 6.60× |
Question 3: How much do results vary across seeds for both training stages, particularly for the sub-point steps of the training-component ablation?
Response: Thank you for raising this question. We repeated the Stage I and Stage II training-component ablations in Table 4 using three random seeds for each configuration. In Stage I, adding the downstream CE loss to module-wise routing with necessity targets increased mean relative performance from 98.1% to 98.4%. Adaptive keep counts raised it further to 99.2%, which was 0.5 percentage points above the full layer-wise baseline of 98.7%. In Stage II, RL refinement raised mean relative performance from 99.2% to 99.9% without the skip-consistency reward (SCR) and to 99.8% with SCR. Adding SCR reduced mean retained FLOPs from 53.4% to 52.1%, a saving of 1.3 percentage points, with a 0.1-point decrease in mean relative performance.
| Method | Paper Avg. (%) | Seed 1 Avg. (%) | Seed 2 Avg. (%) | Seed 3 Avg. (%) | Avg. (%), mean ± SD | Retained FLOPs (%), mean ± SD |
|---|---|---|---|---|---|---|
| Layer-wise Routing (Full Stage I) | 98.9 | 98.9 | 98.5 | 98.6 | 98.7 ± 0.2 | 52.5 ± 0.5 |
| Module-wise + Necessity Targets | 98.2 | 98.1 | 98.1 | 98.2 | 98.1 ± 0.1 | 54.2 ± 0.3 |
| + Downstream CE Loss | 98.5 | 98.3 | 98.5 | 98.4 | 98.4 ± 0.1 | 53.8 ± 0.6 |
| + Adaptive Keep Counts | 99.3 | 99.4 | 99.0 | 99.3 | 99.2 ± 0.2 | 51.7 ± 0.3 |
| + RL Refinement w/o SCR | 100.0 | 100.2 | 99.6 | 99.9 | 99.9 ± 0.3 | 53.4 ± 0.2 |
| + Skip-Consistency Reward | 100.0 | 100.0 | 99.6 | 99.8 | 99.8 ± 0.2 | 52.1 ± 0.1 |
Question 4: What does AdaVSkip add beyond published methods that learn layer or module skipping for visual tokens in MLLMs, such as gamma-MoD (Luo et al.), p-MoD (Zhang et al.), Skip-Vision (Zeng et al.) and AdaLLaVA (Xu et al.)? Most fine-tune the backbone, so a discussion in Section 2.3 would suffice.
Response: Thank you for highlighting these closely related works. AdaVSkip differs from them in both what it routes and how it learns the routing policy. (1) γ-MoD and p-MoD route individual tokens through decoder blocks. Skip-Vision bypasses FFNs for selected visual tokens, while AdaLLaVA chooses a latency-constrained execution plan shared by image and text. AdaVSkip targets vertical redundancy by adaptively determining how many modules process visual tokens and which ones do so for each input, with separate routing decisions for self-attention and MLP. This design helps preserve task performance while skipping visual computation and can be directly combined with other token compression methods that target horizontal redundancy.(2) p-MoD and AdaLLaVA learn routing through answer prediction. γ-MoD adds auxiliary token-selection supervision, while Skip-Vision fine-tunes under a prescribed skipping rule. AdaVSkip trains its routers in two stages. In Stage I, we measure how skipping visual computation in individual modules changes the model’s answer-token distributions, yielding fine-grained, input-specific routing supervision that is combined with the language-modeling loss. Given that routing decisions jointly affect the final answer, we refine the policy in Stage II through reinforcement learning on complete sampled paths, with rewards for answer correctness and consistency with each input’s Stage-I retention rates. We have revised Section 2.3 to discuss all four methods and clarify these distinctions.
Weakness 2: Because routing acts only during prefill, decode cost depends on whether generated tokens still attend to visual keys and values in skipped layers, which the paper leaves open; VisionZip and MMTok also show identical prefill but clearly different end-to-end times, without explanation.
Response: Thank you for raising this question. We would like to clarify that when AdaVSkip skips visual computation in a self-attention layer during prefill, the visual tokens bypass the module entirely and their hidden states pass through unchanged. The layer computes no visual keys or values, so prompt text tokens cannot attend to visual tokens at that layer. At the end of prefill, the layer’s cache contains only prompt-text K/V. During decoding, only text K/V are appended to this cache, so generated tokens cannot attend to visual tokens at that layer either. Generated tokens can still attend to visual tokens at layers where visual K/V were cached during prefill.
We expand on Table 6 below to break down end-to-end latency into vision encoding, token compression, LLM prefill, decoding, and other overhead. Although VisionZip and MMTok have the same LLM prefill latency of 25.0 ms, their end-to-end latencies differ because of the time spent on other stages. In particular, MMTok incurs substantially higher token compression overhead: 16.9 ms compared with 1.9 ms for VisionZip.
| Method | ViT | Compression | Prefill | Decoding | Other | E2E | KV cache (MB) |
|---|---|---|---|---|---|---|---|
| Vanilla | 13.1 | 0.0 | 161.1 | 16.6 | 3.5 | 194.3 | 1087.7 |
| AdaVSkip-only | 13.1 | 0.0 | 77.1 | 15.6 | 3.5 | 109.3 | 517.5 |
| VisionZip | 18.3 | 1.9 | 25.0 | 14.6 | 3.5 | 63.3 | 112.2 |
| MMTok | 13.1 | 16.9 | 25.0 | 14.8 | 3.5 | 73.3 | 112.5 |
| CDPruner + AdaVSkip | 13.1 | 7.1 | 26.4 | 14.5 | 3.5 | 64.6 | 103.7 |
Thank you for your careful review and constructive feedback.
Question 1: Can you report the same pruner at your hybrid's retained token count combined with ShortV or the Fixed-Budget variant of Table 5, at matched FLOPs, and state the token count and skipping budget for every hybrid row?
Response: Thank you for suggesting this control. We evaluated both LLaVA-1.5-7B and LLaVA-NeXT-7B at two visual-token budgets per model. In each setting, we compared ShortV, Fixed-Budget AdaVSkip, and full AdaVSkip, each paired with CDPruner, at comparable prefill FLOPs. These results support two conclusions in the hybrid setting. First, across both models and both budgets, replacing ShortV with Fixed-Budget AdaVSkip improves average performance retention by 0.7–1.3 percentage points at nearly identical prefill FLOPs. This suggests that, even with a fixed skip count, the router is better at identifying modules where visual-token computation can be skipped with minimal impact on accuracy. Second, full AdaVSkip allows the router to adjust how many modules skip visual-token computation for each input. This yields a further 0.6–2.3 percentage-point gain in average performance retention over Fixed-Budget AdaVSkip while reducing average prefill FLOPs in all four settings. Overall, these results demonstrate the effectiveness of AdaVSkip in the hybrid setting: when combined with CDPruner, it consistently achieves higher average performance retention at lower average prefill FLOPs than CDPruner + ShortV across both models and both visual-token budgets.
LLaVA-1.5-7B
LLaVA-NeXT-7B
For the hybrid configurations reported in the original paper, each hybrid combines token pruning at twice the standalone retention ratio with AdaVSkip at a nominal 50% skipping budget. For LLaVA, the 22.2%/11.1%/5.6% equivalent budgets refer to 44.4%/22.2%/11.1% token retention combined with AdaVSkip (K=16). For Qwen3-VL, the 40%/20%/10% equivalent budgets similarly combine 80%/40%/20% token retention with AdaVSkip (K=18).