acceptodds
Desk rejected at ICLR 2027

SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Structured pruning reduces the parameter footprint of vision-language models (VLMs), but its effectiveness depends on how multimodal inputs are used to estimate component importance. Visual tokens often outnumber text tokens, and including redundant visual content in calibration can impair pruning decisions. We propose SlimVLM, a structured pruning framework that combines adaptive visual token selection with sensitivity-aware allocation of pruning ratios. At each decoder layer selected for pruning, SlimVLM uses average text-to-visual attention to select visual tokens for importance estimation while retaining all text tokens. It then adjusts the pruning ratio of each attention or MLP module according to the correlation between its pruned and original outputs, removes attention heads and MLP channels, and compensates for output deviations through least-squares fitting. Token selection operates during calibration; the deployed model uses the reduced parameter structure. Experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL demonstrate the effectiveness of this approach across eight multimodal benchmarks. At a 20% pruning ratio on LLaVA-1.5-7B, SlimVLM retains 93.80% of the unpruned model's average relative performance and leads the evaluated structured-pruning baselines on eight of nine reported task settings. Ablations further show that selecting visual tokens for calibration improves post-pruning performance in both SlimVLM and FLAP.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Structured Pruning, Model Compression, Adaptive Visual Token Selection, Sensitivity-Aware Pruning, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KsQzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SlimVLM, a structured pruning framework for vision-language models. It selects visual calibration tokens using text-to-visual attention while retaining all text tokens, and adjusts module-wise pruning ratios using the correlation between original and pruned outputs. The framework removes attention heads and MLP channels and applies least-squares output correction. Token selection is used during calibration, while the deployed model processes the full input sequence. Experiments evaluate performance retention on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, including different model sizes, pruning ratios, and optional LoRA recovery.

Critical strengths
  1. The paper provides direct evidence that selecting visual calibration tokens improves structured pruning. In Table 4, adaptive selection improves average relative performance from 73.30% to 93.80% for SlimVLM and from 76.26% to 89.37% for FLAP. The improvement on FLAP makes this observation useful beyond the proposed pruning framework. Table 5 further controls the retained token count: selecting the top 20% by attention achieves 92.42%, compared with 89.34% and 84.82% for the middle and bottom selections. These results support the importance of which visual tokens contribute to calibration.
  2. The complete framework shows improvements across several LLaVA configurations without requiring LoRA recovery. At 20% pruning on LLaVA-1.5-7B, SlimVLM retains 93.80% average relative performance, compared with 89.37% for FLAP, at 40%, the corresponding results are 75.16% and 59.32%. The main LLaVA comparisons use the same visual token selection across methods. Results on LLaVA-NeXT and LLaVA-1.5-13B extend the evaluation to higher-resolution inputs and a larger model. The Qwen2.5-VL results additionally demonstrate applicability to grouped-query attention.
Critical weaknesses
  1. The benefit of adapting the retained token count is less clearly isolated than the benefit of attention-based selection. Table 4 compares the adaptive rule with a fixed top-kk policy whose kk equals the text-token count, obtaining 93.80% versus 91.63%. Table 5 also reports 92.42% for retaining the top 20% of visual tokens. These results favor the proposed rule over the tested alternatives, but the policies do not explicitly match the adaptive rule’s average token retention. Please report the number of tokens retained across layers and inputs. Also compare with a top-kk baseline that retains a similar number of tokens on average. This would help clarify whether the gains come from adapting the token count or simply choosing a better average count.

  2. The comparison with SlimLLM leaves the incremental contribution of the revised allocation rule unclear. Table 11 provides a useful comparison, with SlimVLM achieving 93.80% average relative performance versus 90.52% for SlimLLM. However, it does not specify whether SlimLLM uses the same calibration samples and visual token selection. The shared selection policy in Section 4.1 explicitly covers the three main methods. Table 6 reports results with and without dynamic pruning ratios, but does not identify a direct comparison with SlimLLM’s allocation rule. Please clarify the settings used for SlimLLM, a comparison that changes only the allocation rule while holding the remaining pruning implementation, calibration inputs, and actual parameter budget fixed would make its contribution clearer.

Questions for authors

As a related reporting clarification, please explain how r0r_0 corresponds to the reported pruning ratios and provide the parameter counts of the two configurations in Table 6.

1 comment

Response to Reviewer KsQz

Thank you for asking how the adaptive token budget and the pruning-ratio allocation affect the results.

W1. Retained calibration tokens and a mean-matched fixed-budget comparison

On LLaVA-1.5-7B with 24 calibration examples and β=0.2\beta=0.2, adaptive selection retains 283.2 visual tokens per example on average, or approximately 49.2% of the 576 available tokens. Across the prunable decoder layers (3–30), the per-example layer means range from 214.3 to 348.0.

Retained visual tokens across the 28 prunable layers. The standard deviations are across layers.

StatisticLayer 3Layer 14 (max)Layer 18 (min)Layer 30Mean ± SD
Total over 24 calibration examples6,4038,3535,1436,3136,796.8 ± 836.1
Mean per example266.8348.0214.3263.0283.2 ± 34.8

We also tested a fixed top-kk rule at k=283k=283, the rounded adaptive mean. It improves the four-task average over Fixed Top 20% (92.43% vs. 91.74%), while Fixed Top-kk with k=Ltk=L_t, where LtL_t is the text-token count, reaches 92.52%. Adaptive selection achieves 93.02%, a gain of 0.59 percentage points over the mean-matched baseline, and improves all four individual scores.

A fixed count can admit low-attention tokens when attention is concentrated and omit relevant tokens when attention is spread over more of the image. The adaptive threshold varies the budget with the attention distribution for each layer and input. The mean-matched comparison supports this flexibility.

Token-retention policies on LLaVA-1.5-7B at 20% pruning. The average is normalized to the unpruned model.

PolicyVisual tokens per exampleGQAMMBMMESQA-I4-task avg. (%)
Unpruned (Table 1)—61.9462.021780.6465.99100.00
Fixed Top-kk (Table 4)k=Ltk=L_t57.5256.011683.8060.9392.52
Fixed Top 20% (Table 5)115 (≈20%)57.3755.311658.7360.7191.74
Mean-matched Fixed Top-kk283 (≈49.2%)57.4856.181675.3460.8892.43
Adaptive selection (ours)283.2 on average58.0056.441682.9661.3393.02

W2. Controlled comparison of SlimLLM and SlimVLM allocation rules

In Table 11, the SlimLLM allocation rule uses the same 24 MMBench development examples, adaptive visual token selection (β=0.2\beta=0.2), head and channel scoring (Eqs. 8–10), and affine output compensation (Eq. 7) as SlimVLM. Together with the uniform allocation in Table 6, this comparison varies the ratio allocation rule while holding those settings fixed.

SlimLLM's layer-level rule uses pre-pruning input–output similarity and improves the nine-task average from 90.16% to 90.52% (+0.36 percentage points). SlimVLM's module-level rule uses output correlation before and after trial pruning and reaches 93.80% (+3.28 points over SlimLLM and +3.64 points over uniform allocation). SlimVLM improves all nine scores over SlimLLM, including +71.58 on MME, +3.87 on MMB-CN, and +3.48 on VizWiz.

SlimLLM assigns a common ratio to attention and MLP within each decoder layer. SlimVLM assigns their ratios separately. In Figure 4, the allocation removes up to about 31.25% of attention heads in layers 25–30, while allocating roughly 13–15% pruning to their MLP channels. This pattern is consistent with attention becoming more redundant after visual information is integrated into text-token representations while the deep MLPs remain sensitive.

Ratio allocation on LLaVA-1.5-7B at 20% pruning.

Allocation ruleGQAMMBMMB-CNMMEPOPESQA-IVQAv2TextVQAVizWiz9-task avg. (%)
Uniform (Table 6)56.7754.7342.781677.9385.6659.9966.6040.1044.8990.16
SlimLLM (Table 11)57.2154.7842.011611.3884.7259.1764.5740.8851.0790.52
SlimVLM (ours)58.0056.4445.881682.9685.4961.3365.9041.5654.5593.80

Q1. Target ratio and parameter counts in Table 6

We set r0r_0 to the reported full-decoder target, such as r0=0.20r_0=0.20 for the 20% setting. The denominator covers all decoder blocks. We keep the four boundary blocks (layers 0–2 and 31), vision encoder, projector, and embeddings intact. We rescale and round the module ratios from Eq. 6 over the 28 prunable blocks against the full-decoder parameter target. Both configurations in Table 6 retain 5.67B parameters at the reported precision. Under this shared budget, sensitivity-aware allocation improves the nine-task average by 3.64 percentage points.

Reviewer_cqMVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SlimVLM, which combines adaptive visual token selection with sensitivity-aware structured pruning. By selecting visual tokens during calibration and removing entire attention heads and MLP channels, SlimVLM compresses the model while avoiding additional token selection overhead during inference. Experiments show competitive results on LLaVA-1.5 and LLaVA-NeXT. However, the novelty of its token selection strategy compared with existing attention-based methods and its suitability for structured pruning calibration remain unclear. The paper also lacks systematic comparisons with recent state-of-the-art pruning methods, and its performance and efficiency still lag behind leading vision token pruning methods. Furthermore, the degradation of fine-grained visual tasks under high pruning ratios lacks in-depth analysis, and the method’s generalizability remains uncertain because adaptation occurs only during offline calibration.

Critical strengths

S1. Reasonable Research Motivation

The paper investigates how visual token redundancy affects importance estimation in structured parameter pruning. By integrating visual token selection with parameter pruning, it explores a practically valuable direction for multimodal model compression.

S2. Competitive Experimental Results

On LLaVA-1.5 and LLaVA-NeXT, SlimVLM outperforms the main baselines, FLAP and Wanda-sp, at different pruning ratios. Ablation studies also demonstrate that both Adaptive Visual Token Selection and Dynamic Pruning Ratios contribute to performance improvements.

Critical weaknesses

W1. Novelty and Design Rationale of Adaptive Visual Token Selection

Selecting important visual tokens based on text-to-visual attention has been widely explored in prior works such as FastV and VisPruner. VisPruner also shows that attention scores do not always accurately reflect visual token importance. Although this paper applies the strategy during structured pruning calibration, it does not sufficiently explain why high-attention tokens are more suitable for estimating the importance of attention heads and MLP channels. Moreover, Table 4 shows that random token selection also substantially outperforms using all tokens, suggesting that the gains may partly result from reducing the number of calibration tokens rather than the proposed selection criterion. Comparisons with alternative calibration strategies, such as visual attention and attention combined with diversity, would help validate the method’s distinct contribution.

W2. Insufficient Comparisons, Particularly on Newer Models

The paper mainly compares SlimVLM with earlier methods such as FLAP and Wanda-sp. Although the appendix includes SlimLLM and Puzzle, a systematic comparison with more recent state-of-the-art structured pruning methods is still lacking. More importantly, Table 3 reports results for SlimVLM on Qwen2.5-VL-7B without baselines from other methods under the same model and pruning ratios. Therefore, its relative advantages on newer VLM architectures remain unclear. The evaluation should include recent pruning methods and fair comparisons on newer architectures such as Qwen3-VL.

W3. Insufficient Analysis of Visual Capability Degradation at High Pruning Ratios

At a 40% pruning ratio, SlimVLM’s overall performance drops substantially. In particular, TextVQA performance on LLaVA-1.5-7B decreases from 48.07 to 33.17, indicating limited robustness on certain fine-grained visual tasks. However, the paper focuses mainly on average relative performance and lacks an in-depth analysis of degradation in OCR, fine-grained recognition, and complex visual reasoning. Targeted task-specific analyses are needed to clarify the method’s limitations and applicability.

Questions for authors

Q1. Effectiveness and Novelty of Adaptive Visual Token Selection

Why are visual tokens with high Text-to-Visual Attention more suitable for estimating the importance of Attention Heads and MLP Channels? The paper does not provide sufficient justification for this design. Could the authors compare the proposed strategy with alternatives such as Attention + Diversity under the same calibration token budget to verify that the gains come from the selection criterion itself rather than simply reducing the number of calibration tokens?

Q2. Consistency of the Dynamic Pruning Ratio Formula

The paper defines the pruning ratio as r=r0exp⁡(γ(L/T−1))r=r_0\exp(\gamma(L/T-1)), where LL is the Pearson correlation coefficient and the default threshold is T=1T=1. Since L≤1L\leq1, the formula yields r≤r0r\leq r_0 when γ>0\gamma>0, with equality only when L=1L=1. This seems inconsistent with the claim that modules with higher correlation should undergo more pruning. Could the authors clarify the rationale behind this formulation and whether the equation, default parameters, and actual implementation are consistent?

Q3. Fair Comparison with Recent Structured Pruning Methods

The paper lacks systematic comparisons with recent state-of-the-art structured pruning methods under the same model architectures and pruning ratios. In particular, Table 3 reports results for SlimVLM on Qwen2.5-VL-7B without baselines from competing methods under identical settings, making it difficult to assess its relative advantages on newer VLMs. Could the authors include recent methods and provide fair comparisons on architectures such as Qwen2.5-VL-7B or Qwen3-VL?

Q4. Degradation of Visual Capabilities at High Pruning Ratios

The paper reports substantial performance degradation on some fine-grained visual tasks at high pruning ratios, but provides limited analysis of the underlying causes. Could the authors evaluate OCR, fine-grained recognition, and complex visual reasoning separately, and investigate how pruning different components, particularly Attention Heads and MLP Channels, affects these capabilities? Such analysis would help identify the main sources of degradation and clarify the method’s limitations.

Q5. Pruning Budget, Implementation Details, and Experimental Stability

Several aspects of the implementation and evaluation remain unclear:

1. How is the target pruning budget enforced? Since different modules receive different pruning ratios based on their sensitivity, the overall pruning ratio may deviate from the target of 20% or 40%. Does the method normalize module-wise ratios or reallocate the budget? What parameters are included in the denominator when calculating the pruning ratio?

2. How is the dynamic pruning ratio actually determined? The correlation used in the formula depends on the pruned module outputs, which in turn depend on the pruning ratio. Does the algorithm first prune using an initial ratio and then iteratively adjust it, or does it use another search procedure? If iterative updates are involved, what are the update rules and stopping criteria?

3. How many visual tokens are actually retained by adaptive selection? Although the paper introduces the threshold coefficient β\beta, it does not report the number or proportion of retained visual tokens across layers and samples. Could the authors provide these statistics and analyze how different token retention ratios affect pruning performance?

4. Are the reported improvements stable across different calibration samples and random seeds? The paper does not report performance variance or confidence intervals across repeated experiments. Repeated evaluations would help establish the reliability of the reported gains, particularly on tasks where the performance differences between methods are small.

5 comments

Response to Reviewer cqMV — Part (1/3)

W1 / Q1. Why text-to-visual attention is useful for pruning calibration

FastV and VisPruner select visual tokens during inference. VisPruner combines visual attention with similarity-based deduplication to maintain scene coverage when attention concentrates on similar patches. SlimVLM selects visual tokens during calibration to estimate the statistics used for structured weight pruning. Inference on LLaVA-1.5-7B uses all 576 visual tokens.

The calibration subset supplies observations for cross-head output correlations in MHA (Eq. 8), output PCA in the MLP (Eqs. 9–10), and least-squares affine compensation after pruning (Eq. 7). The large number of homogeneous background tokens can give prompt-irrelevant features disproportionate weight in these estimates. Text-to-visual attention favors tokens whose activations are relevant to the current prompt. A diversity criterion can also retain visually distinct background tokens with little relevance to the prompt, which may affect the estimated subspaces and regression targets.

Random selection improves over using all visual tokens, supporting the benefit of reducing the imbalance between 576 visual tokens and LtL_t text tokens. At the same per-sample budget of k=Ltk=L_t, ranking tokens by text-to-visual attention (Fixed Top-kk) improves the four-task average by 3.32 percentage points and MME by 195.21 over Random. At 20% visual token retention, Top 20% improves the four-task average by 7.04 points over Bottom 20% (Table 5).

We also compare prompt-agnostic ViT [CLS] attention and a VisPruner-style attention-plus-diversity criterion with our method at an average budget of approximately 283 visual tokens per sample. ViT [CLS] attention reaches 89.27% on four tasks. For reference, Random reaches 89.20% at its original k=Ltk=L_t budget. Adaptive text-to-visual attention reaches 93.02%, 2.05 points above attention plus diversity, with gains on all four tasks, including 82.62 on MME.

Calibration selection on LLaVA-1.5-7B at 20% pruning. We normalize each task score to the unpruned model's score and then average across tasks.

Selection criterionVisual tokens per sampleGQAMMBMMESQA-I4-task avg. (%)
Unpruned model (Table 1)—61.9462.021780.6465.99100.00
None (Table 4)57647.0935.571065.5152.7068.27
Random (Table 4)LtL_t58.1755.501488.5959.2589.20
Fixed Top-kk (Table 4)LtL_t57.5256.011683.8060.9392.52
Visual attention (ViT [CLS])28356.2855.801528.7659.6689.27
Attention + diversity (VisPruner-style)28357.1556.071600.3460.2890.97
Adaptive text-to-visual attention (ours)283.2 avg.58.0056.441682.9661.3393.02

Q2. Monotonicity of Eq. 6 and the final pruning ratios

Equation 6 defines the raw ratio rlraw=r0exp⁡(γ(Ll/T−1))r_l^{\mathrm{raw}}=r_0\exp(\gamma(L_l/T-1)), where LlL_l is the Pearson correlation between the original and trial-pruned outputs. Under the default T=1T=1 and γ>0\gamma>0, the raw ratio increases strictly with LlL_l and reaches its upper bound of r0r_0 at Ll=1L_l=1. Higher correlation therefore gives a larger raw pruning weight.

Before structural removal, we multiply all raw ratios for prunable modules by a common positive factor α\alpha, giving r^l=αrlraw\hat{r}_l=\alpha r_l^{\mathrm{raw}}. This preserves their ordering before rounding. The scale is set by the full-decoder budget, which includes layers kept intact. Individual module ratios can therefore exceed r0r_0 after normalization. In Figure 4, under the 20% decoder budget, some deep attention modules reach about 31.25% pruning, while the corresponding deep MLP modules have pruning ratios of about 13–15%.

Response to Reviewer cqMV — Part (2/3)

W2 / Q3. Structured pruning baselines on Qwen2.5-VL-7B

Qwen2.5-VL uses grouped-query attention (GQA). Multiple query heads share each key/value head, so pruning a shared key/value head affects its whole query group. Porting MHA-oriented pruning code that uses a shared one-to-one head mask across the Q, K, and V projections, such as FLAP's implementation, requires accounting for these dependencies when slicing the projections and matching the parameter budget. SlimVLM's GQA procedure prunes query heads and permits different retained query-head counts across key/value groups (Section 3.3). Table 3 evaluates that procedure.

For the rebuttal, we adapted Wanda-sp and FLAP to GQA and evaluated all three methods with the same 24 calibration samples and matched parameter budgets at 20% pruning. SlimVLM reaches 92.35% average relative performance, 3.35 percentage points above Wanda-sp and 2.28 points above FLAP. It achieves the highest score on all four benchmarks, including MME gains of 141.42 over Wanda-sp and 86.99 over FLAP.

GQA-compatible structured pruning on Qwen2.5-VL-7B at 20% pruning.

MethodGQAMMBMMESQA-I4-task avg. (%)
Unpruned model (Table 3)58.2880.672218.0680.61100.00
Wanda-sp52.9871.861892.3473.1089.00
FLAP53.2073.791946.7772.3690.07
SlimVLM (ours)54.4775.232033.7673.3492.35

W3 / Q4. Task-specific degradation at 40% pruning

At 40% training-free structured pruning, LLaVA-1.5-7B's TextVQA score falls from 48.07 to 25.31, retaining 52.65% of the unpruned score. TextVQA is particularly sensitive to this setting compared with tasks such as the binary hallucination checks in POPE. Table 1 covers OCR (TextVQA), fine-grained recognition (GQA and VizWiz), reasoning (MMB and SQA-I), and hallucination checks (POPE). Under the 40% budget, SlimVLM retains 75.16% average relative performance, compared with 60.25% for Wanda-sp and 59.32% for FLAP. The following mechanisms are possible explanations for these task-level differences.

  • OCR. Reading requires spatial grounding of small text regions and accurate transcription. There is limited redundancy across characters, and small changes in transcription can alter the answer and reduce its score.
  • Fine-grained recognition and reasoning. Relational perception and multi-step reasoning can depend on repeated cross-modal processing. Approximation errors may accumulate across the 28 pruned decoder layers, making these tasks sensitive to a high pruning ratio.
  • Attention heads and MLP channels. Figure 4 shows different pruning ratios for attention and MLP modules at 20% pruning. Specialized attention heads in early and middle layers may help ground small text regions. As the pruning ratio increases, removal can reach some of these heads as well as the more redundant deep heads. Deep MLP channels can support lexical decoding and multi-step composition and remain sensitive to width reduction. The affine least-squares correction in Eq. 7 reduces output discrepancies without gradient updates. Its linear form has limited capacity to recover nonlinear activation patterns that support fine-grained predictions.

Although this degradation limits direct training-free deployment on OCR-intensive tasks at high pruning ratios, we believe maintaining a strictly gradient-free, minute-scale pipeline provides strong practical value: it supports plug-and-play compression for general multimodal deployment while offering a stronger structural initialization than competing pruners whenever downstream applications opt for lightweight recovery such as LoRA.

Response to Reviewer cqMV — Part (3/3)

Q5-1 / Q5-2. Decoder budget and computation of dynamic ratios

The budget denominator is the parameter count of all LLM decoder blocks, including the unpruned boundary layers. We keep the vision encoder, projector, and embeddings intact. We determine the module pruning ratios in two passes.

  1. Sensitivity probing. We temporarily apply the same trial pruning ratio to each prunable module, using the global target r0r_0. We measure the correlation between the original and trial-pruned outputs (Eq. 5) while preserving the original dense weights for final pruning.
  2. Budget reconciliation and pruning. We turn the correlations into raw candidate ratios using Eq. 6 and rescale them globally across the prunable layers against the full-decoder parameter target. Rounding to whole heads and channels produces the discrete ratios in Figure 4. Budget reconciliation completes the ratio allocation. We then select heads and channels as described in Section 3.3, including the iterative head exchanges, and fit the affine correction in Eq. 7.

Q5-3. Retained calibration tokens and the mean-matched Fixed Top-kk baseline

On LLaVA-1.5-7B with β=0.2\beta=0.2, adaptive selection retains 283.2 of the 576 visual tokens per sample on average. The layer-level means range from 214.3 to 348.0 visual tokens per sample across layers 3–30.

Retained visual tokens across prunable layers on LLaVA-1.5-7B (24 calibration samples). The last column reports the mean and standard deviation across layers.

MetricLayer 3Layer 14 (max)Layer 18 (min)Layer 30Mean ± SD
Total across 24 samples6,4038,3535,1436,3136,796.8 ± 836.1
Mean per sample266.8348.0214.3263.0283.2 ± 34.8

Increasing the fixed budget to k=283k=283 improves the four-task average over Fixed Top 20% (92.43% vs. 91.74%). Fixed Top-kk with k=Ltk=L_t reaches 92.52%. Adaptive selection reaches 93.02% and outperforms the mean-matched fixed budget on all four benchmarks. These results support adapting the count to the attention distribution in each layer and sample. A fixed count can include low-attention tokens when attention is concentrated or omit relevant visual cues when attention is spread over more tokens.

Calibration retention policies on LLaVA-1.5-7B at 20% pruning.

Retention policyVisual tokens per sampleGQAMMBMMESQA-I4-task avg. (%)
Unpruned model (Table 1)—61.9462.021780.6465.99100.00
Fixed Top-kk (Table 4)k=Ltk=L_t57.5256.011683.8060.9392.52
Fixed Top 20% (Table 5)115 (≈20%)57.3755.311658.7360.7191.74
Mean-matched Fixed Top-kk283 (≈49.2%)57.4856.181675.3460.8892.43
Adaptive selection (ours)283.2 avg.58.0056.441682.9661.3393.02

Q5-4. For a specific response, please refer to Reviewer YkWq Part (1/3) W2-1.

Reviewer_YkWqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Calibration-based structured pruning (attention heads + MLP channels) of the VLM language decoder; the deployed model still processes all tokens. Key observation: visual tokens dominate calibration statistics and skew importance estimates. SlimVLM calibrates on all text tokens plus only the visual tokens that receive enough text attention, and adds module-wise pruning ratios based on output correlation. Scoring and output correction follow SlimLLM. Evaluated on LLaVA-1.5 (7B/13B), LLaVA-NeXT and Qwen2.5-VL; clearly ahead of FLAP / Wanda-sp, especially at higher pruning ratios.

Critical strengths

1. Simple, transferable observation.

  • Changing only the calibration tokens gives large gains, for SlimVLM and for FLAP alike, so it isn't tied to one scorer.
  • Zero inference cost; could be added to most calibration-based pruners.

2. Clear gains at higher pruning ratios on the LLaVA backbones.

  • Large margins over FLAP / Wanda-sp at the more aggressive ratio, not driven by a single benchmark.
  • Also runs on a GQA decoder, plus a 13B model, LoRA recovery, and composition with token pruning.
Critical weaknesses

1. Limited novelty; closest prior work missing.

  • Head/channel scoring and output correction are from SlimLLM; the ratio allocation is a small variant.
  • Attention-guided calibration-token selection with adaptive layer-wise sparsity already exists (TAMP, unstructured), and isn't discussed. Other VLM pruning work (ECoFLaP, MULTIFLOW) also missing.

2. Evidence for the gains is weaker than it looks.

  • Calibration data and hyperparameters come from the evaluation benchmarks; single run with a tiny calibration set.
  • One noisy benchmark (VizWiz) accounts for a large part of the ablation margins.
  • Random visual-token selection already recovers most of the gain, so the effect may mostly be rebalancing visual vs. text tokens rather than "text-guided" selection.
  • Qwen2.5-VL is SlimVLM-only, no FLAP or SlimLLM. Shows it runs, not that it wins. And at the higher ratio it drops quite a lot, with a few benchmarks degrading badly; hard to judge without baselines.

3. Budget accounting unclear.

  • The ratio formula, with the reported settings, can't produce the reported per-module ratios; there must be an undescribed step.
  • Unclear whether methods are compared at the same parameter count; the speed advantage at "equal" ratios may partly reflect this.
Questions for authors
  1. How is the global budget enforced? Actual parameter counts for all pruned models and methods?
  2. Results with calibration data disjoint from evaluation, over several seeds?
  3. Text-only calibration, reweighted-all-tokens, and random-at-same-count baselines: is it text guidance or rebalancing?
  4. Positioning and head-to-head against TAMP's token selection within the same structured pruner?
  5. Baselines (FLAP, SlimLLM) on Qwen2.5-VL?
3 comments

Response to Reviewer YkWq — Part (1/3)

W1. Relationship to prior VLM pruning and SlimLLM

Thank you for pointing us to TAMP, ECoFLaP, and MULTIFLOW and for asking us to clarify the connection to SlimLLM.

VLM parameter pruning. ECoFLaP estimates layer importance globally and then performs unstructured weight pruning within each layer. MULTIFLOW scores individual weights using information flow and allocates sparsity using the multimodal distribution of the parameters. SlimVLM addresses the effect of abundant visual background tokens on the statistics used to remove entire attention heads and MLP channels. Removing these structures reduces dense tensor dimensions. At 20% pruning, SlimVLM runs at 1.25× the throughput of unpruned LLaVA-1.5-7B and uses less memory with standard dense kernels.

Relationship to TAMP. TAMP uses attention to select representative multimodal calibration tokens for activation-based weight scoring. Its main method uses modality-specific output-token diversity to allocate layer-wise sparsity, and it also evaluates block pruning. SlimVLM retains every text token and uses causal attention from text queries to visual-token positions to select visual tokens relevant to the prompt. We use the selected calibration tokens to score each attention head by computing the Pearson correlation between the full MHA output and the output after removing that head (Eq. 8), and to score MLP channels using output PCA (Eqs. 9–10). SlimVLM removes whole heads and channels, with pruning ratios allocated separately to attention and MLP modules.

Relationship to SlimLLM. SlimVLM builds on SlimLLM's head and channel scoring and affine compensation (Sections 2, 3.2, and 3.3). SlimLLM allocates a shared ratio to each decoder block from its pre-pruning input–output similarity. At 20% pruning, Figure 4 shows approximately 31.25% attention-head pruning in layers 26–30 and approximately 13–15% MLP-channel pruning in layers 28–30. This pattern is consistent with attention becoming more redundant after visual features are integrated into text-token representations. SlimVLM uses the Pearson correlation between each module's original and trial-pruned outputs (Eqs. 5–6) to assign separate ratios. It also supports retaining different numbers of query heads across key/value groups in Qwen2.5-VL.

Empirical impact. Under matched calibration and pruning operators in Table 11, SlimVLM's allocation rule raises average retention from 90.52% to 93.80% (+3.28 percentage points), including gains of 71.58 on MME, 3.87 on MMB-CN, and 3.48 on VizWiz. In Table 4, using all visual tokens for SlimVLM's calibration yields 73.30% average retention. Applying our token selection to FLAP raises its average retention from 76.26% to 89.37% (+13.11 percentage points).

W2-1. Calibration split, token-level statistics, and seed stability

We draw 24 unlabeled calibration inputs (12 English and 12 Chinese) from the MMBench development splits (Section 4.1). Following standard LLaVA local evaluation protocols—where MMBench test labels are withheld for server submission—local MMBench accuracy is evaluated on the full development splits, while the remaining seven benchmarks (including GQA, MME, SQAI\text{SQA}^\text{I}, TextVQA, POPE, and VizWiz) are completely disjoint datasets. Although the 12 sampled inputs per language appear within the >4,300-example development splits (<0.3% of evaluated items), calibration uses only raw image–question inputs without ground-truth labels or gradient updates, and all compared methods share the exact same 24 calibration examples.

Calibration estimates module-output correlations, MLP output principal components, and affine regressions from label-free forward activations, without gradient updates. Each row of the MLP output matrix OMLP∈RN×DO^{\mathrm{MLP}} \in \mathbb{R}^{N \times D} is a token-level observation. The 24 multimodal inputs yield thousands of activation vectors per layer for these estimates. Table 8 shows that increasing the sample count from 12 to 36 raises GPU memory use from 17.74 GB to 26.11 GB and produces mixed changes in benchmark scores. Repeating calibration across independent random seeds (Table-R1) further confirms stable performance on both MMBench and disjoint benchmarks.

Table-R1: Calibration seed results for LLaVA-1.5-7B at 20% pruning with 24 samples per run.

SeedGQAMMBMMESQAI\text{SQA}^\text{I}
058.3456.741672.6861.15
4258.0056.441682.9661.33
202757.6657.011677.3761.48
Mean ± SD58.00 ± 0.3456.73 ± 0.291677.67 ± 5.1561.32 ± 0.17

The scores vary little across the three randomly drawn calibration subsets.

Response to Reviewer YkWq — Part (2/3)

W2-2. Ablation results excluding VizWiz

We recomputed average relative performance for Tables 4–6 across the eight remaining task settings after excluding VizWiz. The ordering of variants is unchanged in each ablation.

Manuscript table and settingVariant9-task avg. (%)8-task avg. excluding VizWiz (%)
Table 4, FLAPNone76.2673.66
Table 4, FLAPRandom82.0081.52
Table 4, FLAPAdaptive (ours)89.3789.37
Table 4, SlimVLMNone73.3071.72
Table 4, SlimVLMRandom89.5190.04
Table 4, SlimVLMFixed91.6392.08
Table 4, SlimVLMAdaptive (ours)93.8093.01
Table 5, attention rankingBottom 20%84.8285.65
Table 5, attention rankingMiddle 20%89.3489.00
Table 5, attention rankingTop 20%92.4291.63
Table 6, pruning allocationUniform90.1691.13
Table 6, pruning allocationDynamic (ours)93.8093.01
  • Table 4. On SlimVLM, adaptive selection outperforms None, Random, and Fixed by 21.29, 2.97, and 0.93 percentage points, respectively, on the eight-task average. Adaptive selection improves seven of eight task settings over Fixed, including MMB-CN (+1.89) and TextVQA (+0.55). On FLAP, the average gain over None increases from 13.11 to 15.71 percentage points when VizWiz is excluded.
  • Table 5. Top 20% outperforms Bottom 20% by 5.98 percentage points on the eight-task average.
  • Table 6. Dynamic allocation improves the eight-task average by 1.88 percentage points and improves six task settings. MMB-CN rises from 42.78 to 45.88, MMB from 54.73 to 56.44, TextVQA from 40.10 to 41.56, SQA-I from 59.99 to 61.33, GQA from 56.77 to 58.00, and MME from 1677.93 to 1682.96.

VizWiz images can contain blur and poor framing, which may leave relatively few patches informative about the question. This could make prompt-guided filtering of background tokens and retention of deep MLP channels especially helpful on VizWiz.

W2-3 / Q3. Token-count rebalancing and text-guided selection

As discussed in Sections 1 and 3.1, an unfiltered LLaVA-1.5-7B calibration sequence has 576 visual tokens, typically far more than the number of text tokens. Random visual token selection increases average retention from 73.30% to 89.51% for SlimVLM and from 76.26% to 82.00% for FLAP. These gains support rebalancing the modalities during calibration. Two comparisons then hold the visual token budget fixed and test which tokens to retain.

  1. Fixed 20% budget (Table 5). Each variant retains about 115 visual tokens per sample. The nine-task average rises from 84.82% to 89.34% to 92.42% as we move from the bottom to the middle to the top 20% by text-to-visual attention. The top-to-bottom gap is 7.60 percentage points.
  2. Budget matched to the text length (Table 4). Random and Fixed both retain k=Ltk=L_t visual tokens, where LtL_t is the number of text tokens. Selecting the top-ranked tokens at this budget improves SlimVLM's nine-task average by 2.12 percentage points (89.51% to 91.63%) and MME by 195.21 (1488.59 to 1683.80).

The adaptive attention threshold changes both the set and the number of retained visual tokens. It adds 2.17 percentage points over Fixed on SlimVLM. On FLAP, switching from Random at k=Ltk=L_t to adaptive selection improves the average by 7.37 percentage points, MME by 359.80, and VQAv2 by 11.57.

Selected benchmark results from the token-selection ablations.

Model and visual token budgetComparisonMMEMMBMMB-CNSQA-I9-task avg. (%)
SlimVLM, 20% (Table 5)Bottom → Top 20%1530.37 → 1658.7349.91 → 55.3140.77 → 44.0855.38 → 60.7184.82 → 92.42
SlimVLM, k=Ltk=L_t (Table 4)Random → Fixed Top-kk1488.59 → 1683.8055.50 → 56.0142.61 → 43.9959.25 → 60.9389.51 → 91.63
FLAP, k=Ltk=L_t vs. adaptive (Table 4)Random → Adaptive1286.12 → 1645.9250.10 → 52.5837.68 → 41.2457.30 → 59.8982.00 → 89.37

Response to Reviewer YkWq — Part (3/3)

W2-4 / Q5. Structured pruning baselines on Qwen2.5-VL

Qwen2.5-VL uses grouped-query attention (GQA), in which several query heads share each key/value head. The official FLAP implementation uses one head mask across the Q, K, and V projections, assuming a one-to-one correspondence. In GQA, removing a shared key/value head affects every query head in its group. Adapting the baselines therefore requires tracking query–key/value dependencies, slicing the corresponding projections, and accounting for the parameters removed. Table 3 evaluates SlimVLM's native GQA support, which allows different numbers of query heads to be retained in different key/value groups (Section 3.3).

For the rebuttal, we evaluated GQA-adapted Wanda-sp and FLAP on Qwen2.5-VL-7B at the nominal 20% pruning setting. SlimVLM achieves 92.35% average relative performance, 3.35 percentage points above Wanda-sp and 2.28 above FLAP. SlimVLM scores higher on all four benchmarks, including MME gains of 141.42 over Wanda-sp and 86.99 over FLAP.

Structured pruning on Qwen2.5-VL-7B with 20% pruning and 24 calibration inputs.

MethodGQAMMBMMESQA-I4-task avg. (%)
Unpruned (Table 3)58.2880.672218.0680.61100.00
Wanda-sp52.9871.861892.3473.1089.00
FLAP53.2073.791946.7772.3690.07
SlimVLM (ours)54.4775.232033.7673.3492.35

W3 / Q1. Accounting for the global decoder pruning budget

SlimVLM applies a deterministic two-pass procedure to allocate the decoder-wide pruning budget. The budget is defined over all decoder layers, including the unpruned boundary layers.

  1. Sensitivity estimation. We temporarily apply the target ratio to each prunable module and measure the Pearson correlation between its original and trial-pruned outputs (Eq. 5). These correlations determine the candidate pruning ratios (Eq. 6). We preserve the dense weights during the trial.
  2. Budget reconciliation and execution. We globally rescale the candidates across the prunable layers against the full-decoder parameter target. We round the removal counts to whole heads and channels, producing the discrete steps in Figure 4, then apply structural pruning and closed-form compensation once to the original weights.

Parameter counts, throughput, and performance for LLaVA-1.5-7B.

SettingMethodParametersThroughput (tok/s)Relative avg. (%)
UnprunedVanilla7.06B85.39100.00
20% pruningWanda-sp5.62B94.7685.31
20% pruningFLAP5.72B93.3389.37
20% pruningSlimVLM (ours)5.67B106.6993.80

At both pruning levels, SlimVLM attains higher average performance than FLAP with fewer retained parameters. At 20% pruning, SlimVLM retains slightly more parameters than Wanda-sp and runs at 106.69 tok/s versus 94.76 tok/s.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.