acceptodds
Under review as a conference paper at ICLR 2027

AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLM Inference

Abstract

Multimodal large language models (MLLMs) require substantial computation to process numerous visual tokens across all transformer layers. Most methods for efficient MLLM inference exploit horizontal redundancy by compressing visual tokens. Beyond token reduction, recent studies exploit vertical redundancy through early exit or fixed-layer skipping. However, we find that the extent and distribution of this redundancy vary across inputs and differ between self-attention and MLP modules. Motivated by these observations, we propose AdaVSkip, which equips each layer with two lightweight routers that independently determine whether visual tokens participate in the computation of the self-attention and MLP modules. These decisions collectively define an input-specific visual computation path. To train these routers, we develop a progressive two-stage training framework that updates only the routers while keeping the backbone frozen. Stage I establishes an initial routing policy by combining supervision from input-specific targets derived from module-wise necessity scores with the downstream language modeling loss. To better account for how routing decisions jointly affect answer quality, we introduce Stage II to further refine the policy. This stage uses reinforcement learning with feedback from answers generated under complete sampled routing paths. It combines an answer correctness reward with a skip-consistency reward that discourages excessive retention of visual token computation. Across three MLLM backbones, AdaVSkip maintains strong task performance with substantially less computation. On LLaVA-NeXT-7B, AdaVSkip reduces language model prefill FLOPs by 53.2% while preserving the original model's average performance. Combining it with visual token compression increases this reduction to 93.1%, while retaining 97.2% of the original performance on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLMs, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xLsROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that in many layers of a multimodal LLM, the self-attention or MLP computation on visual tokens contributes little to the answer, and that where this happens depends on the input and the module type. AdaVSkip attaches a lightweight router to every attention and MLP module that reads the prompt's text hidden states and decides, during prefill, whether visual tokens take part, with the backbone frozen. The routers are trained on targets from a per-module KL necessity score and then refined with GRPO. On three backbones and eight benchmarks, the paper claims full-model accuracy at much lower prefill cost and complementarity with token pruning: with CDPruner on LLaVA-NeXT it keeps 97.2% of average performance at 6.9% of the FLOPs.

Critical strengths
  1. AdaVSkip degrades far more gracefully than fixed vertical skipping as the budget tightens. It beats ShortV and VTW at every skipping budget on all three backbones, and the margin widens at the most aggressive setting, where on LLaVA-1.5 it retains 95.1% of average performance against 85.8% for ShortV. This is direct evidence that input-dependent, module-wise decisions pay off where they matter most.

  2. The method is cheap, leaves the backbone untouched, and plugs into existing token pruners. Router training and offline profiling each finish within hours per backbone, the trained routers are reused on pruned token sets without retraining, and the FLOPs accounting is explicit. The finding that attention and MLP differ in redundancy is well supported and broadly useful.

Critical weaknesses
  1. The hybrid results do not yet isolate what adaptive routing contributes. Section 4.1 and the conclusion credit adaptive routing for the hybrid's gains over token compression alone. That requires holding the retained token count fixed and changing only the skipping rule. Instead, at the same equivalent budget the hybrid keeps more visual tokens than the standalone pruner and then skips modules (twice as many in the one Qwen3-VL configuration described), and for the LLaVA backbones the hybrid's token count and skipping budget are not reported. A fixed rule is already strong, with ShortV within two points of AdaVSkip at the roughly half-FLOPs budget on both LLaVA backbones, so much of the gain could come from a token and depth trade-off that any reasonable skipping rule would provide. As it stands, the evidence shows that pruning and vertical skipping combine well, not that adaptive routing makes the combination work.

  2. The efficiency evidence supports the FLOPs reduction but not yet a general wall-clock speedup. Latency is measured on a single benchmark, without stating hardware, batch size or output length, and without the two closest competitors, CDPruner alone and ShortV. Because routing acts only during prefill, decode cost depends on whether generated tokens still attend to visual keys and values in skipped layers, which the paper leaves open; VisionZip and MMTok also show identical prefill but clearly different end-to-end times, without explanation. Routers are also trained separately per budget with no seeds or variance reported, so the training-component ablation gains, all below one point, cannot be told apart from noise. The practical speedup and the value of individual training components therefore remain suggestive.

Questions for authors
  1. Can you report the same pruner at your hybrid's retained token count combined with ShortV or the Fixed-Budget variant of Table 5, at matched FLOPs, and state the token count and skipping budget for every hybrid row?
  2. What hardware, batch size and generation length does Table 6 use, and how do CDPruner alone and ShortV compare under that setup, including batched inference where samples take different routing paths?
  3. How much do results vary across seeds for both training stages, particularly for the sub-point steps of the training-component ablation?
  4. What does AdaVSkip add beyond published methods that learn layer or module skipping for visual tokens in MLLMs, such as gamma-MoD (Luo et al.), p-MoD (Zhang et al.), Skip-Vision (Zeng et al.) and AdaLLaVA (Xu et al.)? Most fine-tune the backbone, so a discussion in Section 2.3 would suffice.

If the hybrid still clearly beats a token pruner plus ShortV or Fixed-Budget routing at matched token count and FLOPs, and the latency setup is fully specified, I would raise my rating; if that control closes most of the gap, I would lower it.

3 comments

Thank you for your careful review and constructive feedback.

Question 1: Can you report the same pruner at your hybrid's retained token count combined with ShortV or the Fixed-Budget variant of Table 5, at matched FLOPs, and state the token count and skipping budget for every hybrid row?

Response: Thank you for suggesting this control. We evaluated both LLaVA-1.5-7B and LLaVA-NeXT-7B at two visual-token budgets per model. In each setting, we compared ShortV, Fixed-Budget AdaVSkip, and full AdaVSkip, each paired with CDPruner, at comparable prefill FLOPs. These results support two conclusions in the hybrid setting. First, across both models and both budgets, replacing ShortV with Fixed-Budget AdaVSkip improves average performance retention by 0.7–1.3 percentage points at nearly identical prefill FLOPs. This suggests that, even with a fixed skip count, the router is better at identifying modules where visual-token computation can be skipped with minimal impact on accuracy. Second, full AdaVSkip allows the router to adjust how many modules skip visual-token computation for each input. This yields a further 0.6–2.3 percentage-point gain in average performance retention over Fixed-Budget AdaVSkip while reducing average prefill FLOPs in all four settings. Overall, these results demonstrate the effectiveness of AdaVSkip in the hybrid setting: when combined with CDPruner, it consistently achieves higher average performance retention at lower average prefill FLOPs than CDPruner + ShortV across both models and both visual-token budgets.

LLaVA-1.5-7B

Visual token retention ratioMethodFLOPs Avg. (%)GQATextVQAPOPEMMEMMBench ENSEED-imgMMStarAvg. (%)
22.2%ShortV (N=19)22.759.152.587.11437.661.363.534.296.5
22.2%Fixed-Budget AdaVSkip (K=16)22.659.054.287.21427.762.263.934.697.3
22.2%AdaVSkip (K=16)21.660.156.487.51424.063.064.134.998.4
11.1%ShortV (N=19)17.958.051.886.61385.660.062.033.294.5
11.1%Fixed-Budget AdaVSkip (K=16)17.857.953.187.51407.160.862.432.895.2
11.1%AdaVSkip (K=16)17.358.555.087.61405.960.662.632.995.8

LLaVA-NeXT-7B

Visual token retention ratioMethodFLOPs Avg. (%)GQATextVQAPOPEMMEMMBench ENSEED-imgMMStarAvg. (%)
22.2%ShortV (N=19)12.661.852.787.71481.066.167.534.696.0
22.2%Fixed-Budget AdaVSkip (K=16)12.662.051.987.71497.366.868.936.897.3
22.2%AdaVSkip (K=16)11.263.357.588.21498.666.768.536.798.9
11.1%ShortV (N=19)7.959.852.187.31448.263.865.933.493.7
11.1%Fixed-Budget AdaVSkip (K=16)7.960.051.287.81452.564.366.836.095.0
11.1%AdaVSkip (K=16)6.962.456.288.81473.965.967.635.397.3

For the hybrid configurations reported in the original paper, each hybrid combines token pruning at twice the standalone retention ratio with AdaVSkip at a nominal 50% skipping budget. For LLaVA, the 22.2%/11.1%/5.6% equivalent budgets refer to 44.4%/22.2%/11.1% token retention combined with AdaVSkip (K=16). For Qwen3-VL, the 40%/20%/10% equivalent budgets similarly combine 80%/40%/20% token retention with AdaVSkip (K=18).

Question 2: What hardware, batch size and generation length does Table 6 use, and how do CDPruner alone and ShortV compare under that setup, including batched inference where samples take different routing paths?

Response: Thank you for raising this question. The latency results in Table 6 were measured on MME using a single NVIDIA RTX 5090 GPU with a batch size of 1. Each response consists of two generated tokens: an answer token (“yes” or “no”) followed by an end-of-sequence token. We compare AdaVSkip with ShortV and evaluate the effect of adding AdaVSkip to CDPruner. Compared with ShortV, AdaVSkip reduces end-to-end latency by 14.3 ms and improves the MME score by 9.4 points. Compared with CDPruner alone, CDPruner + AdaVSkip improves the MME score by 48.6 points at comparable end-to-end latency.

MethodEnd-to-end latency (ms) ↓MME ↑
ShortV (N=19)123.6 (1.57×)1526.0
AdaVSkip(K=16)109.3 (1.78×)1535.4
CDPruner63.5 (3.06×)1425.3
CDPruner + AdaVSkip(K=16)64.6 (3.01×)1473.9

During batched inference, each sample makes independent attention and MLP routing decisions at every layer. For attention, we form up to two sub-batches: samples retaining visual attention process both text and visual tokens, while those skipping it process only text tokens. Different sequence lengths are accommodated by padding within each sub-batch. For the MLP, we regroup samples according to their MLP routing decisions. The keep group processes both text and visual tokens, while the skip group processes only text tokens. We then write the updates back to their original positions, leaving the skip group’s visual states unchanged. We report end-to-end speedups relative to vanilla LLaVA-NeXT-7B at the same batch size. At B=8, AdaVSkip achieves a 2.10× speedup compared with 1.78× for ShortV. CDPruner + AdaVSkip reaches 6.6× compared with 6.17× for CDPruner alone. As batch size increases, AdaVSkip achieves greater end-to-end speedups, highlighting its potential for efficient batched inference.

MethodB=1B=2B=4B=8
ShortV (N=19)1.57×1.70×1.77×1.78×
AdaVSkip(K=16)1.78×1.94×2.05×2.10×
CDPruner3.06×4.65×5.76×6.17×
CDPruner + AdaVSkip(K=16)3.01×4.59×5.84×6.60×

Question 3: How much do results vary across seeds for both training stages, particularly for the sub-point steps of the training-component ablation?

Response: Thank you for raising this question. We repeated the Stage I and Stage II training-component ablations in Table 4 using three random seeds for each configuration. In Stage I, adding the downstream CE loss to module-wise routing with necessity targets increased mean relative performance from 98.1% to 98.4%. Adaptive keep counts raised it further to 99.2%, which was 0.5 percentage points above the full layer-wise baseline of 98.7%. In Stage II, RL refinement raised mean relative performance from 99.2% to 99.9% without the skip-consistency reward (SCR) and to 99.8% with SCR. Adding SCR reduced mean retained FLOPs from 53.4% to 52.1%, a saving of 1.3 percentage points, with a 0.1-point decrease in mean relative performance.

MethodPaper Avg. (%)Seed 1 Avg. (%)Seed 2 Avg. (%)Seed 3 Avg. (%)Avg. (%), mean ± SDRetained FLOPs (%), mean ± SD
Layer-wise Routing (Full Stage I)98.998.998.598.698.7 ± 0.252.5 ± 0.5
Module-wise + Necessity Targets98.298.198.198.298.1 ± 0.154.2 ± 0.3
+ Downstream CE Loss98.598.398.598.498.4 ± 0.153.8 ± 0.6
+ Adaptive Keep Counts99.399.499.099.399.2 ± 0.251.7 ± 0.3
+ RL Refinement w/o SCR100.0100.299.699.999.9 ± 0.353.4 ± 0.2
+ Skip-Consistency Reward100.0100.099.699.899.8 ± 0.252.1 ± 0.1

Question 4: What does AdaVSkip add beyond published methods that learn layer or module skipping for visual tokens in MLLMs, such as gamma-MoD (Luo et al.), p-MoD (Zhang et al.), Skip-Vision (Zeng et al.) and AdaLLaVA (Xu et al.)? Most fine-tune the backbone, so a discussion in Section 2.3 would suffice.

Response: Thank you for highlighting these closely related works. AdaVSkip differs from them in both what it routes and how it learns the routing policy. (1) γ-MoD and p-MoD route individual tokens through decoder blocks. Skip-Vision bypasses FFNs for selected visual tokens, while AdaLLaVA chooses a latency-constrained execution plan shared by image and text. AdaVSkip targets vertical redundancy by adaptively determining how many modules process visual tokens and which ones do so for each input, with separate routing decisions for self-attention and MLP. This design helps preserve task performance while skipping visual computation and can be directly combined with other token compression methods that target horizontal redundancy.(2) p-MoD and AdaLLaVA learn routing through answer prediction. γ-MoD adds auxiliary token-selection supervision, while Skip-Vision fine-tunes under a prescribed skipping rule. AdaVSkip trains its routers in two stages. In Stage I, we measure how skipping visual computation in individual modules changes the model’s answer-token distributions, yielding fine-grained, input-specific routing supervision that is combined with the language-modeling loss. Given that routing decisions jointly affect the final answer, we refine the policy in Stage II through reinforcement learning on complete sampled paths, with rewards for answer correctness and consistency with each input’s Stage-I retention rates. We have revised Section 2.3 to discuss all four methods and clarify these distinctions.

Weakness 2: Because routing acts only during prefill, decode cost depends on whether generated tokens still attend to visual keys and values in skipped layers, which the paper leaves open; VisionZip and MMTok also show identical prefill but clearly different end-to-end times, without explanation.

Response: Thank you for raising this question. We would like to clarify that when AdaVSkip skips visual computation in a self-attention layer during prefill, the visual tokens bypass the module entirely and their hidden states pass through unchanged. The layer computes no visual keys or values, so prompt text tokens cannot attend to visual tokens at that layer. At the end of prefill, the layer’s cache contains only prompt-text K/V. During decoding, only text K/V are appended to this cache, so generated tokens cannot attend to visual tokens at that layer either. Generated tokens can still attend to visual tokens at layers where visual K/V were cached during prefill.

We expand on Table 6 below to break down end-to-end latency into vision encoding, token compression, LLM prefill, decoding, and other overhead. Although VisionZip and MMTok have the same LLM prefill latency of 25.0 ms, their end-to-end latencies differ because of the time spent on other stages. In particular, MMTok incurs substantially higher token compression overhead: 16.9 ms compared with 1.9 ms for VisionZip.

MethodViTCompressionPrefillDecodingOtherE2EKV cache (MB)
Vanilla13.10.0161.116.63.5194.31087.7
AdaVSkip-only13.10.077.115.63.5109.3517.5
VisionZip18.31.925.014.63.563.3112.2
MMTok13.116.925.014.83.573.3112.5
CDPruner + AdaVSkip13.17.126.414.53.564.6103.7
Reviewer_Z15FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaVSkip, an input-adaptive framework for reducing visual-token computation in multimodal large language models. Motivated by the observation that visual computation requirements vary across inputs and between attention and MLP modules, the method introduces two independent routers per layer to determine whether visual tokens participate in each module. The routers are trained with necessity-guided supervision followed by reinforcement learning, while the backbone remains frozen. Experiments across three backbones demonstrate favorable performance–computation trade-offs, particularly under aggressive skipping budgets, and show that the method can complement visual-token compression.

Critical strengths
  1. Well-supported motivation. The paper provides useful empirical evidence that visual-token redundancy depends on the input and differs between attention and MLP modules. These observations clearly motivate moving beyond fixed skipping patterns and treating the two module types separately.

  2. Innovative and coherent design. Input-dependent, module-wise routing offers a flexible way to allocate visual computation. Allowing visual tokens to bypass selected modules and resume computation later is appealing. The two-stage training framework also addresses a meaningful limitation of isolated necessity scores by subsequently evaluating complete routing paths.

  3. Substantial ablation evidence. The experiments investigate routing granularity, adaptive computation budgets, module selection, router inputs, and the contributions of the training components. These ablations help explain the design choices and support the effectiveness of the overall framework.

Critical weaknesses
  1. The autoregressive decoding configuration is insufficiently clear. Section 3.2 states that routing is applied only during prefill and that visual tokens do not participate in skipped attention modules. However, it remains unclear whether generated tokens can attend to visual tokens at those layers during autoregressive decoding. Table 6 reports approximately 33.2 ms outside prefill for the vanilla model and 32.2 ms for AdaVSkip. These similar times suggest that decoding costs may remain largely unchanged, but they do not establish the attention or KV-cache configuration. My tentative interpretation is that visual tokens may still be accessible during decoding; the authors should explicitly confirm or correct this interpretation. If they remain accessible, the paper should explain how their K/V entries are obtained and why this decoding configuration is adopted. This clarification is necessary to understand both the method’s behavior and its efficiency claims.

  2. The practical acceleration benefits are insufficiently characterized for increasingly important agentic workloads. Reducing visual prefill computation can plausibly provide substantial benefits for high-resolution inputs with short text prompts and short outputs. However, as agentic applications become increasingly important, the method’s effectiveness with long prompts and long generated outputs becomes a central practical question. Such applications can involve substantial accumulated context and extended generation, changing the balance between visual prefill, textual prefill, and decoding costs. The paper does not clearly establish how end-to-end speedups vary with the relative numbers of visual tokens, prompt tokens, and generated tokens. Consequently, the current latency results provide limited evidence about the method’s benefits in these increasingly relevant settings. A systematic evaluation across these dimensions is needed to establish where substantial acceleration persists and where its benefits diminish.

Questions for authors
  1. In layers where visual attention computation is skipped during prefill, can generated tokens attend to visual tokens during decoding? Please clarify the corresponding KV-cache construction and attention configuration, explain the rationale for this choice, and specify what is included in the reported non-prefill latency.

  2. Could you report end-to-end latency, with a prefill/decoding breakdown, across different visual-token counts, prompt lengths, and output lengths? Given the growing importance of agentic applications, please explicitly include long-prompt and long-output settings and characterize the conditions under which the method continues to provide substantial acceleration.

4 comments

Thank you for your careful review and constructive feedback.

Question 1: In layers where visual attention computation is skipped during prefill, can generated tokens attend to visual tokens during decoding? Please clarify the corresponding KV-cache construction and attention configuration, explain the rationale for this choice, and specify what is included in the reported non-prefill latency.

Response: Thank you for raising this question. We would like to clarify that when AdaVSkip skips visual computation in a self-attention layer during prefill, the visual tokens bypass the module entirely and their hidden states pass through unchanged. The layer computes no visual keys or values, so prompt text tokens cannot attend to visual tokens at that layer. At the end of prefill, the layer’s cache contains only prompt-text K/V. During decoding, only text K/V are appended to this cache, so generated tokens cannot attend to visual tokens at that layer either. Generated tokens can still attend to visual tokens at layers where visual K/V were cached during prefill. We chose this design to reduce the size of the KV cache. At the same time, even without visual K/V at the skipped attention layers, AdaVSkip with (K=16) matches or slightly exceeds vanilla performance on average across eight benchmarks, reaching 100.0% of the vanilla baseline on LLaVA-1.5-7B and 100.4% on LLaVA-NeXT-7B (Tables 1 and 2).

We expand on Table 6 below to report each method’s latency breakdown and KV-cache size. Our end-to-end (E2E) latency includes ViT encoding, visual token compression (when applicable), LLM prefill, autoregressive decoding, and “Other” overhead, including multimodal projection and per-step stopping checks. The reviewer’s 33.2 ms for Vanilla and 32.2 ms for AdaVSkip-only equal E2E latency minus LLM prefill time. Both include 13.1 ms for ViT encoding and 3.5 ms for other overhead, leaving 16.6 and 15.6 ms per request for decoding, respectively (a 1.06× speedup). Although the decoding speedup is modest, omitting visual K/V at skipped attention layers reduces the KV cache from 1,087.7 MB to 517.5 MB for AdaVSkip-only, a 52.4% reduction. Adding CDPruner further reduces the cache to 103.7 MB and decoding latency to 14.5 ms, a 1.14× speedup over Vanilla. VisionZip and MMTok achieve similar decoding speedups of 1.14× and 1.12×, respectively.

MethodViTCompressionPrefillDecodingOtherE2EKV cache (MB)
Vanilla13.10.0161.116.63.5194.31087.7
AdaVSkip-only13.10.077.115.63.5109.3517.5
VisionZip18.31.925.014.63.563.3112.2
MMTok13.116.925.014.83.573.3112.5
CDPruner + AdaVSkip13.17.126.414.53.564.6103.7

To explain why halving the KV-cache size yields only a 1.06× decoding speedup, we break down autoregressive decoding latency by operation. During decoding, AdaVSkip’s shorter KV cache reduces the time spent on cache updates from 2.1 to 1.0 ms and on attention from 0.7 to 0.5 ms. Together, however, these operations account for only 17% of Vanilla’s decoding time. Other operations, including the Q/K/V/O projections and MLPs, process only the current token, so their cost is independent of cache length. As a result, total decoding latency falls from 16.6 to 15.6 ms per request, a 1.06× speedup.

MethodQ/K/V/O projectionsKV cache updateAttentionMLPOther GPU kernelsOther timeDecoding total
Vanilla3.02.10.75.71.33.816.6
AdaVSkip-only3.01.00.55.71.34.115.6
CDPruner + AdaVSkip3.00.30.35.71.33.914.5

Taken together, these results show that AdaVSkip delivers its largest speedup during prefill by skipping visual-token computation in selected modules. Omitting visual K/V at skipped attention layers also reduces the cache size, making cache updates and attention faster during decoding. Each newly generated text token still requires the Q/K/V/O projections and MLP computations, so the overall decoding speedup remains modest. VisionZip and MMTok show a similar pattern: token compression reduces visual K/V entries but leaves these per-token computations unchanged, resulting in similarly modest decoding speedups. We believe that both prefill and decoding are important optimization targets. Faster prefill reduces time to first token, while faster decoding accelerates subsequent generation. Their distinct computational bottlenecks make it difficult for a single method to accelerate both equally well. Which stage matters more for end-to-end latency depends on the task. Decoding speed becomes increasingly important for long responses. For tasks with many visual tokens and short answers, AdaVSkip is particularly valuable because it delivers substantial prefill speedups while also reducing KV-cache memory usage.

Question 2: Could you report end-to-end latency, with a prefill/decoding breakdown, across different visual-token counts, prompt lengths, and output lengths? Given the growing importance of agentic applications, please explicitly include long-prompt and long-output settings and characterize the conditions under which the method continues to provide substantial acceleration.

Response to Question 2 (Part 1/2): Thank you for raising this question. We report end-to-end generation latency on LLaVA-NeXT-7B across a range of visual-token counts, text prompt lengths, and output lengths, with separate timings for prefill and decoding. All latencies are reported in milliseconds. Entries in the Prefill, Decoding, and E2E columns show Vanilla / AdaVSkip, with the corresponding speedup in parentheses.

1. Visual-token count. We evaluate LLaVA-NeXT-7B on MMBench and divide the examples into three groups based on the number of visual tokens produced by native AnyRes. Both methods generate exactly two tokens per example, including EOS. From the lowest to the highest visual-token group, prefill speedup increases from 1.82× to 2.25× and end-to-end speedup from 1.60× to 1.98×. Decoding gains are smaller, ranging from 1.03× to 1.15×.

Visual-token countMean visual tokensPrefillDecodingE2E
n ≤ 1,5001,207.6121.3 / 66.5 (1.82×)14.7 / 14.3 (1.03×)147.4 / 92.3 (1.60×)
1,500 < n < 2,5002,063.7174.3 / 84.1 (2.07×)15.8 / 14.3 (1.10×)206.0 / 114.4 (1.80×)
n ≥ 2,5002,802.5243.5 / 108.4 (2.25×)16.8 / 14.6 (1.15×)277.8 / 140.6 (1.98×)

2. Prompt length. We group examples from MMBench, TextVQA, and MME by text prompt length with 2,880 visual tokens per example across all three groups. Because AdaVSkip skips only visual-token computation, a higher proportion of text tokens reduces the fraction of prefill work it can skip. Across the three groups, prefill speedup falls from 2.14× to 1.79× and E2E speedup from 1.85× to 1.62×.

Prompt-token rangeMean prompt tokensPrefillDecodingE2E
0–10080.6255.1 / 119.5 (2.14×)17.1 / 15.0 (1.14×)300.2 / 162.5 (1.85×)
101–200136.0255.8 / 124.3 (2.06×)17.2 / 15.1 (1.14×)301.9 / 168.3 (1.79×)
>400542.9272.2 / 151.8 (1.79×)17.7 / 15.6 (1.14×)320.8 / 198.2 (1.62×)

3. Generated-output length. We group HallusionBench examples by output length, with 2,880 visual tokens per example across all four groups. On the original inference backend, AdaVSkip achieves a consistent 2.14–2.16× prefill speedup across groups. As the mean output length increases from 2.9 to 320.3 tokens, decoding accounts for a larger share of generation time, and the E2E speedup falls from 1.80× to 1.12×.

Output-token rangeMean output tokensPrefillDecodingE2E
2–52.9250.3 / 116.1 (2.16×)30.0 / 26.3 (1.14×)309.7 / 171.8 (1.80×)
6–5028.3249.7 / 116.1 (2.15×)412.8 / 377.8 (1.09×)690.8 / 522.1 (1.32×)
51–10067.7249.4 / 115.9 (2.15×)1,005.5 / 914.5 (1.10×)1,283.9 / 1,059.3 (1.21×)
>200320.3246.7 / 115.4 (2.14×)4,853.2 / 4,417.3 (1.10×)5,129.2 / 4,562.2 (1.12×)

We also report results using vLLM for the same four output-length groups in the table below. Using vLLM reduces decoding latency for both methods and increases AdaVSkip’s decoding speedup over Vanilla in all groups. For outputs longer than 200 tokens, decoding latency drops from 4,853.2 to 3,969.8 ms for Vanilla and from 4,417.3 to 3,404.1 ms for AdaVSkip, raising the speedup from 1.10× to 1.17×. E2E speedup remains nearly unchanged for short outputs (1.80× vs. 1.78×), but increases from 1.32× to 1.38× for 6–50 tokens, from 1.21× to 1.28× for 51–100 tokens, and from 1.12× to 1.19× for outputs longer than 200 tokens. These results show that pairing AdaVSkip with a more efficient decoding backend such as vLLM can unlock more of its potential for end-to-end acceleration.

Output-token rangeMean output tokensPrefillDecodingE2E
2–52.9215.1 / 101.1 (2.13×)23.3 / 19.9 (1.17×)267.7 / 150.3 (1.78×)
6–5028.3214.6 / 100.8 (2.13×)335.7 / 290.3 (1.16×)578.5 / 419.3 (1.38×)
51–10067.7214.7 / 101.1 (2.12×)821.1 / 703.7 (1.17×)1,064.7 / 833.7 (1.28×)
>200320.3213.5 / 100.5 (2.12×)3,969.8 / 3,404.1 (1.17×)4,212.7 / 3,533.9 (1.19×)

Response to Question 2 (Part 2/2): Thank you for raising this question. To address the question about agentic applications, we extend our evaluation to AndroidControl [1] using Qwen3-VL-8B-Instruct. AndroidControl is a mobile GUI benchmark with 15,283 human demonstrations across 833 Android apps. At each step, the agent predicts its next action given the current screen and either a high-level task goal or a step-specific instruction. With an average of 1,020.1 input text tokens per model call, this evaluation covers an agentic workload with long textual inputs. Compared with Vanilla, AdaVSkip achieves a 1.89× prefill speedup and a 1.19× E2E speedup, with overall action exact match (EM) decreasing from 73.9% to 71.6% (2.3 percentage points). Compared with ShortV, AdaVSkip improves overall EM by 2.1 percentage points and reduces mean end-to-end latency from 2,075.7 to 1,921.1 ms per action.

MethodOverall EM (%)High-level acc. (%)Low-level acc. (%)Prefill (ms)Decoding (ms)E2E (ms)
Vanilla73.965.582.3445.9 (1.00×)1,433.0 (1.00×)2,295.0 (1.00×)
ShortV (N=19)69.561.377.6253.3 (1.76×)1,404.9 (1.02×)2,075.7 (1.11×)
AdaVSkip (K=18)71.662.780.5235.4 (1.89×)1,268.7 (1.13×)1,921.1 (1.19×)

Overall, across the evaluated settings, AdaVSkip delivers larger end-to-end speedups in settings with many visual tokens, short text prompts, and short outputs, while its gains diminish as prompts or outputs become longer. The AndroidControl results further demonstrate AdaVSkip’s ability to accelerate agentic workloads.

[1] Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. “On the Effects of Data Scale on UI Control Agents.” NeurIPS, 2024.

Thank you again for your thoughtful questions. As the discussion deadline approaches, we would like to follow up on the two points you raised.

  1. Clarifying KV-cache construction

    When visual tokens skip an attention layer during prefill, AdaVSkip caches only text K/V at that layer. During decoding, generated tokens can therefore attend to visual tokens only at layers that retained visual K/V.

    To understand why decoding benefits less than prefill, we profiled individual decoding operations in the setting reported in Table 6. A shorter cache speeds up cache updates and attention, which together account for only 17% of baseline decoding time, while other operations such as Q/K/V/O projections and MLPs process only the current text token at a cost independent of cache length. As a result, AdaVSkip’s 52.4% reduction in KV-cache memory usage yields only a modest decoding speedup.

  2. End-to-end speedup across workloads

    We added prefill and decoding measurements across different visual-token counts, prompt lengths, and output lengths. End-to-end gains were larger with more visual tokens and smaller as prompts or outputs grew longer.

    On AndroidControl, AdaVSkip achieved 1.89× prefill and 1.19× end-to-end speedups over Vanilla, with a 2.3-point drop in action exact match. Compared with ShortV, AdaVSkip improved action exact match by 2.1 percentage points and reduced latency per action by 7.4%. Together, these results show that AdaVSkip can speed up action prediction in a realistic mobile agent task.

We would be grateful if you could let us know whether our responses address your concerns. If any questions remain, we would be happy to discuss them further. Thank you for your time and consideration.

Reviewer_ZUywOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AdaVSkip introduces two lightweight routers per layer to decide whether visual tokens participate in attention and MLP computation for each input. The routers are trained in two stages, first with KL-based necessity targets and language modeling supervision, then with RL using correctness and skip-consistency rewards. The backbone remains frozen throughout. Experiments on three MLLM backbones show improvements over the evaluated layer-skipping baselines, and the method can be combined with token pruning for further FLOPs reduction.

Critical strengths

The results are encouraging. On LLaVA-NeXT-7B, AdaVSkip achieves 100.4% average relative performance at 46.8% of the original prefill FLOPs, and 97.2% at 6.9% FLOPs when combined with CDPruner. The ablations examine module-wise routing, adaptive keep counts, and both training stages. In particular, the skip-consistency reward reduces retained FLOPs from 53.5% to 52.0% without sacrificing average performance in Table 4.

Critical weaknesses

SkipGPT [1] already explores token-aware routing with separate attention and MLP decisions and a two-stage optimization procedure. The routing granularity and training objectives differ: AdaVSkip makes a shared decision for all visual tokens in each module and uses necessity-guided supervision followed by RL. However, the paper does not clearly establish the contribution beyond adapting these ideas to visual-token computation. Visual-Skip [2] also uses KL-based profiling to obtain task-calibrated visual attention skipping patterns without training additional routers. Given its timing, I do not consider the missing citation itself a problem, but a matched-FLOPs comparison would help establish how much AdaVSkip gains from learned per-input routing and MLP skipping.

The practical efficiency evaluation is also incomplete. Table 8 reports router training times but excludes the necessity profiling required for Stage I. Table 13 gives profiling times without specifying the sample count or hardware, making the total offline cost difficult to assess. Table 6 reports latency on LLaVA-NeXT-7B using MME, but does not specify the batch size. Since routing decisions are input-dependent, different samples within a batch may follow different computation paths. It is unclear how this is handled or whether the reported 2.09× prefill speedup holds for batched inference. These costs matter when evaluating whether the additional training and routing complexity is worthwhile.

[1] Zhao et al. "SkipGPT: Each Token is One of a Kind." ICML, 2025. https://proceedings.mlr.press/v267/zhao25ab.html

[2] Ma et al. "Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs." arXiv:2606.08511, 2026. https://arxiv.org/abs/2606.08511

Questions for authors
  1. Beyond the routing granularity and visual-token setting, what are the main technical differences from SkipGPT [1]? Could the authors provide a SkipGPT-style baseline and a matched-FLOPs comparison with Visual-Skip [2]?

  2. What batch size and hardware were used for Table 6? How are different routing paths handled within a batch, and do the reported speedups hold at batch sizes greater than one?

  3. What is the total offline cost of the method, including necessity profiling? Please specify the number of profiled samples, hardware, and wall-clock time.

  4. For LLaVA-NeXT with AnyRes inputs, how is NvN_v determined in the FLOPs calculation? What token retention ratios and skipping budgets are used for the hybrid configurations? In particular, please clarify the configurations behind the 7.7% and 6.9% FLOPs reported in Table 2.

  5. Does the OneThinker subset used for Stage II contain examples from the training splits of ChartQA, DocVQA, InfoVQA, TextVQA, or GQA? Have the authors checked for image-level overlap with the evaluation sets?

5 comments

Thank you for your careful review and constructive feedback.

Question 1: Beyond the routing granularity and visual-token setting, what are the main technical differences from SkipGPT? Could the authors provide a SkipGPT-style baseline and a matched-FLOPs comparison with Visual-Skip?

Response: Thank you for your question. Beyond routing granularity and the visual token setting, the main difference is how we train the routers. SkipGPT trains its routers with a language-modeling loss and a global sparsity penalty. These objectives do not provide sufficiently fine-grained supervision for individual routing decisions. We therefore adopt a two-stage training strategy. In Stage I, we pair the language-modeling loss with per-input gate targets derived by measuring how skipping visual computation in one module at a time changes the model’s output distributions. Since these targets only assess skips in isolation, Stage II rewards sampled routing paths that produce correct answers and penalizes deviations from each input’s Stage I keep ratios for attention and MLP modules.

Since SkipGPT’s second stage fine-tunes the backbone via LoRA using only the language-modeling loss, whereas AdaVSkip keeps the backbone frozen throughout, we focus on a controlled comparison of router-training objectives. Specifically, we compare AdaVSkip-S1 with a baseline trained using SkipGPT’s router-tuning objective. Both use the same Stage I training data, router architecture, and skipping operators, and update only the routers.

BackboneMethodFLOPs (%) ↓VQAv2GQATextVQAPOPEMMEMMBSEEDMMStarAvg. (%) ↑
LLaVA-1.5-7BVanilla100.078.561.958.285.91510.764.366.133.6100.0
SkipGPT-style router tuning54.271.656.552.084.81248.560.564.434.493.4
AdaVSkip-S1 (K=16)51.877.661.156.886.31516.364.066.233.399.3
AdaVSkip (K=16)52.078.361.857.986.41512.864.965.933.4100.0
LLaVA-NeXT-7BVanilla100.081.864.261.386.51519.067.470.235.8100.0
SkipGPT-style router tuning50.778.061.656.183.61432.257.060.536.593.3
AdaVSkip-S1 (K=16)46.381.063.657.387.31509.267.069.437.699.4
AdaVSkip (K=16)46.881.364.159.487.31535.466.470.038.3100.4

AdaVSkip-S1 raises average performance retention from 93.4% to 99.3% on LLaVA-1.5 and from 93.3% to 99.4% on LLaVA-NeXT, using fewer prefill FLOPs than the SkipGPT-style baseline on both backbones. Stage II further raises retention to 100.0% and 100.4%, respectively. These results demonstrate the effectiveness of our training approach in preserving model performance while reducing computational cost.

Regarding Visual-Skip, we would like to clarify two key differences between our approach and theirs. (1) Visual-Skip uses 20 calibration pairs per task to select a fixed set of layers for skipping visual attention. AdaVSkip’s routers adapt both the number and location of skips to each image–question pair. (2) Visual-Skip bypasses only the attention update for visual queries. Text queries still attend to visual keys and values, and visual tokens do not skip the MLP. AdaVSkip makes independent skipping decisions for the attention and MLP modules in each layer. When a module is selected for skipping, visual tokens do not participate in any computation within that module and are passed through with their representations unchanged. We also compare AdaVSkip with Visual-Skip on LLaVA-1.5-7B and LLaVA-NeXT-7B below.

BackboneMethodPrefill FLOPs retained (%) ↓GQAPOPEMMBSQAAI2DVizWizAvg. retention (%) ↑
LLaVA-1.5-7BVisual-Skip (N=19)90.461.885.565.069.355.454.5100.3
Visual-Skip (N=24)87.861.385.164.769.155.054.699.9
Visual-Skip (N=32)83.854.578.760.768.852.254.894.9
AdaVSkip (K=16)52.661.886.464.969.054.355.6100.4
LLaVA-NeXT-7BVisual-Skip (N=19)86.663.486.366.870.065.259.399.2
AdaVSkip (K=16)46.864.187.366.469.364.961.399.8

Because Visual-Skip bypasses only visual-token updates in self-attention, it still requires 83.8% of the original prefill FLOPs on LLaVA-1.5-7B even when applied to all 32 layers. Thus, increasing (N) cannot match AdaVSkip’s 52.6% prefill FLOPs at (K=16). Compared with Visual-Skip at (N=19), (24), and (32), AdaVSkip improves average performance retention by 0.1, 0.5, and 5.5 percentage points, respectively, while using 41.8%, 40.1%, and 37.2% fewer prefill FLOPs. On LLaVA-NeXT-7B, AdaVSkip achieves 99.8% retention versus 99.2% for Visual-Skip at (N=19), with 46.0% fewer prefill FLOPs.

Question 2: What batch size and hardware were used for Table 6? How are different routing paths handled within a batch, and do the reported speedups hold at batch sizes greater than one?

Response: Thank you for your question. The latency results in Table 6 were measured with batch size 1 on a single NVIDIA RTX 5090. In our batched implementation, each sample makes independent attention and MLP routing decisions at every layer. For attention, we form up to two sub-batches: samples retaining visual attention process both text and visual tokens, while those skipping it process only text tokens. Different sequence lengths are accommodated by padding within each sub-batch. For the MLP, we regroup samples according to their MLP routing decisions. The keep group processes both text and visual tokens, while the skip group processes only text tokens. We then write the updates back to their original positions, leaving the skip group’s visual states unchanged.

We further evaluate router-only AdaVSkip (K=16) on MME at batch sizes of 1, 2, 4, and 8. Each table entry reports the speedup during prompt processing (prefill) followed by the end-to-end (E2E) speedup, both relative to the vanilla model at the same batch size. As batch size increases from 1 to 8, prefill speedup rises from 1.63× to 2.01× on LLaVA-1.5-7B and from 2.09× to 2.25× on LLaVA-NeXT-7B. The corresponding E2E speedups rise from 1.42× to 1.84× and from 1.78× to 2.10×, respectively.

BackboneB=1B=2B=4B=8
LLaVA-1.5-7B1.63× / 1.42×1.65× / 1.46×1.89× / 1.70×2.01× / 1.84×
LLaVA-NeXT-7B2.09× / 1.78×2.13× / 1.94×2.23× / 2.05×2.25× / 2.10×

Question 3: What is the total offline cost of the method, including necessity profiling? Please specify the number of profiled samples, hardware, and wall-clock time.

Response: Thank you for your question. To generate the routing targets for Stage I, we profile 5,500 samples per LLaVA backbone on four NVIDIA RTX 5090 GPUs, matching the Stage I training set size. Appendix C provides two profiling options. On LLaVA-1.5, switching from full profiling, used for our main results, to accelerated profiling cuts the time from 0.66 to 0.25 h (2.64× speedup) with no drop in Stage I average relative performance. On LLaVA-NeXT, profiling time drops from 2.05 to 0.68 h (3.01× speedup), with a slight decrease in performance from 99.4% to 99.2% (Tables 13 and 15). Preparing the Stage II training data takes an additional 0.06 h for LLaVA-1.5 and 0.19 h for LLaVA-NeXT, also using four NVIDIA RTX 5090 GPUs. Adding these data-preparation times to the profiling times in Table 13 and the training times in Table 8 gives the total offline wall-clock times below. With full profiling, the totals are 1.60 h for LLaVA-1.5 and 4.04 h for LLaVA-NeXT. Accelerated profiling reduces these totals to 1.19 h and 2.67 h.

BackboneProfiling: Full / Accel. (h)Stage I (h)Stage II data prep. (h)Stage II (h)Total: Full / Accel. (h)
LLaVA-1.5-7B0.66 / 0.250.130.060.751.60 / 1.19
LLaVA-NeXT-7B2.05 / 0.680.350.191.454.04 / 2.67

Notably, AdaVSkip provides flexible profiling and training options that can be combined to suit different offline time budgets and accuracy requirements. To minimize offline time, accelerated profiling can be paired with Stage I training alone, achieving 99.3% average relative performance on LLaVA-1.5 in 0.38 h and 99.2% on LLaVA-NeXT in 1.03 h. When higher accuracy is the priority, full profiling followed by Stage I training and Stage II RL refinement raises these results to 100.0% and 100.4%, with total offline times of 1.60 h and 4.04 h, respectively.

Question 4: For LLaVA-NeXT with AnyRes inputs, how is (N_v) determined in the FLOPs calculation? What token retention ratios and skipping budgets are used for the hybrid configurations? In particular, please clarify the configurations behind the 7.7% and 6.9% FLOPs reported in Table 2.

Response: Thank you for your question. As described in Appendix A.7, N_v is the actual number of visual tokens in each sample after AnyRes processing. We compute each sample’s prefill FLOPs using its actual token counts and routing decisions, normalize by the original model’s prefill FLOPs for the same input, and report the mean of these per-sample ratios. All hybrid configurations use twice the standalone token budget, combined with approximately 50% skipping K=16 for LLaVA-1.5/NeXT and K=18 for Qwen3-VL. In Table 2, standalone CDPruner uses a visual token budget of 5.6% and retains 7.7% of the original prefill FLOPs. CDPruner + AdaVSkip doubles that budget to 11.1% and applies routing at K=16, retaining 6.9% of the original prefill FLOPs.

Question 5: Does the OneThinker subset used for Stage II contain examples from the training splits of ChartQA, DocVQA, InfoVQA, TextVQA, or GQA? Have the authors checked for image-level overlap with the evaluation sets?

Response: Thank you for your question. According to OneThinker’s source annotations, our Stage II subsets include examples from the ChartQA and TextVQA training splits. None of the selected examples is explicitly attributed to DocVQA, InfoVQA, or GQA. For each backbone, we checked all 3,000 Stage II training examples against the evaluation splits used in our experiments and found no image-level overlap.

Comments on the paper

1 comment

Dear Reviewers,

As we are approaching the end of reviewer-author discussions, please go through authors' responses and raise questions that you would like authors to reply.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.