- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SlimVLM, a structured pruning framework for vision-language models. It selects visual calibration tokens using text-to-visual attention while retaining all text tokens, and adjusts module-wise pruning ratios using the correlation between original and pruned outputs. The framework removes attention heads and MLP channels and applies least-squares output correction. Token selection is used during calibration, while the deployed model processes the full input sequence. Experiments evaluate performance retention on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, including different model sizes, pruning ratios, and optional LoRA recovery.
Critical strengths
- The paper provides direct evidence that selecting visual calibration tokens improves structured pruning. In Table 4, adaptive selection improves average relative performance from 73.30% to 93.80% for SlimVLM and from 76.26% to 89.37% for FLAP. The improvement on FLAP makes this observation useful beyond the proposed pruning framework. Table 5 further controls the retained token count: selecting the top 20% by attention achieves 92.42%, compared with 89.34% and 84.82% for the middle and bottom selections. These results support the importance of which visual tokens contribute to calibration.
- The complete framework shows improvements across several LLaVA configurations without requiring LoRA recovery. At 20% pruning on LLaVA-1.5-7B, SlimVLM retains 93.80% average relative performance, compared with 89.37% for FLAP, at 40%, the corresponding results are 75.16% and 59.32%. The main LLaVA comparisons use the same visual token selection across methods. Results on LLaVA-NeXT and LLaVA-1.5-13B extend the evaluation to higher-resolution inputs and a larger model. The Qwen2.5-VL results additionally demonstrate applicability to grouped-query attention.
Critical weaknesses
-
The benefit of adapting the retained token count is less clearly isolated than the benefit of attention-based selection. Table 4 compares the adaptive rule with a fixed top- policy whose equals the text-token count, obtaining 93.80% versus 91.63%. Table 5 also reports 92.42% for retaining the top 20% of visual tokens. These results favor the proposed rule over the tested alternatives, but the policies do not explicitly match the adaptive rule’s average token retention. Please report the number of tokens retained across layers and inputs. Also compare with a top- baseline that retains a similar number of tokens on average. This would help clarify whether the gains come from adapting the token count or simply choosing a better average count.
-
The comparison with SlimLLM leaves the incremental contribution of the revised allocation rule unclear. Table 11 provides a useful comparison, with SlimVLM achieving 93.80% average relative performance versus 90.52% for SlimLLM. However, it does not specify whether SlimLLM uses the same calibration samples and visual token selection. The shared selection policy in Section 4.1 explicitly covers the three main methods. Table 6 reports results with and without dynamic pruning ratios, but does not identify a direct comparison with SlimLLM’s allocation rule. Please clarify the settings used for SlimLLM, a comparison that changes only the allocation rule while holding the remaining pruning implementation, calibration inputs, and actual parameter budget fixed would make its contribution clearer.
Questions for authors
As a related reporting clarification, please explain how corresponds to the reported pruning ratios and provide the parameter counts of the two configurations in Table 6.
Response to Reviewer KsQz
Thank you for asking how the adaptive token budget and the pruning-ratio allocation affect the results.
W1. Retained calibration tokens and a mean-matched fixed-budget comparison
On LLaVA-1.5-7B with 24 calibration examples and β=0.2, adaptive selection retains 283.2 visual tokens per example on average, or approximately 49.2% of the 576 available tokens. Across the prunable decoder layers (3–30), the per-example layer means range from 214.3 to 348.0.
Retained visual tokens across the 28 prunable layers. The standard deviations are across layers.
We also tested a fixed top-k rule at k=283, the rounded adaptive mean. It improves the four-task average over Fixed Top 20% (92.43% vs. 91.74%), while Fixed Top-k with k=Lt, where Lt is the text-token count, reaches 92.52%. Adaptive selection achieves 93.02%, a gain of 0.59 percentage points over the mean-matched baseline, and improves all four individual scores.
A fixed count can admit low-attention tokens when attention is concentrated and omit relevant tokens when attention is spread over more of the image. The adaptive threshold varies the budget with the attention distribution for each layer and input. The mean-matched comparison supports this flexibility.
Token-retention policies on LLaVA-1.5-7B at 20% pruning. The average is normalized to the unpruned model.
W2. Controlled comparison of SlimLLM and SlimVLM allocation rules
In Table 11, the SlimLLM allocation rule uses the same 24 MMBench development examples, adaptive visual token selection (β=0.2), head and channel scoring (Eqs. 8–10), and affine output compensation (Eq. 7) as SlimVLM. Together with the uniform allocation in Table 6, this comparison varies the ratio allocation rule while holding those settings fixed.
SlimLLM's layer-level rule uses pre-pruning input–output similarity and improves the nine-task average from 90.16% to 90.52% (+0.36 percentage points). SlimVLM's module-level rule uses output correlation before and after trial pruning and reaches 93.80% (+3.28 points over SlimLLM and +3.64 points over uniform allocation). SlimVLM improves all nine scores over SlimLLM, including +71.58 on MME, +3.87 on MMB-CN, and +3.48 on VizWiz.
SlimLLM assigns a common ratio to attention and MLP within each decoder layer. SlimVLM assigns their ratios separately. In Figure 4, the allocation removes up to about 31.25% of attention heads in layers 25–30, while allocating roughly 13–15% pruning to their MLP channels. This pattern is consistent with attention becoming more redundant after visual information is integrated into text-token representations while the deep MLPs remain sensitive.
Ratio allocation on LLaVA-1.5-7B at 20% pruning.
Q1. Target ratio and parameter counts in Table 6
We set r0 to the reported full-decoder target, such as r0=0.20 for the 20% setting. The denominator covers all decoder blocks. We keep the four boundary blocks (layers 0–2 and 31), vision encoder, projector, and embeddings intact. We rescale and round the module ratios from Eq. 6 over the 28 prunable blocks against the full-decoder parameter target. Both configurations in Table 6 retain 5.67B parameters at the reported precision. Under this shared budget, sensitivity-aware allocation improves the nine-task average by 3.64 percentage points.