acceptodds
Under review as a conference paper at ICLR 2027

MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs

Abstract

Multi-view representations enable pretrained image and video vision-language models to understand 3D scenes, but their long visual token sequences make inference costly. Redundancy arises both from repeated observations of the same scene regions across views and from processing all visual tokens at every LLM layer, regardless of their relevance to the current computation. We propose GeoGR, a two-stage framework that jointly compresses scene representations and visual computation. At the projector level, GeoSemZip consolidates cross-view observations into 3D voxel tokens and combines semantic saliency with spatial coverage to construct a compact scene representation. Unselected voxel features are aggregated into contextual anchors to retain scene information that would otherwise be discarded. At the LLM level, we identify an effective visual computation window and observe that query-conditioned token relevance remains locally stable but changes over longer layer intervals. Guided by these findings, GroupRoute combines late entry and early exit with recoverable group-wise skipping. Low-relevance tokens bypass complete LLM layers while retaining their hidden states and can rejoin the active group at subsequent anchor layers selected under a computation budget. For compression-aware post-training, we construct a compact dataset of 67K samples by retaining 30% of the original training pool. With this protocol, GeoGR retains 95.2% and 96.8% of the dense post-trained models' performance on LLaVA-OneVision-7B and Video-3D LLM-7B, respectively, while accelerating LLM prefill by and . These results demonstrate the consistent effectiveness of GeoGR across 3D VLMs with and without explicit 3D spatial encoding, highlighting a favorable performance-efficiency trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLMs, Vision Token Pruning, Token Compression, Operation Compression, Efficiency and Compression, Interpretability and Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZMxdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MWOP reduces multimodal decoder computation by independently pruning V2V, T2V, and T2T attention paths within each head and selecting FFN channels separately for visual and text tokens. First-order Taylor scores guide pruning, and FFN importance is recomputed under the selected attention mask before LoRA recovery.

On LLaVA-OneVision-7B, the paper reports 43.2% fewer decoder FLOPs, 99.7% average relative performance kpt, and a 1.6× decoder prefill speedup.

Critical strengths
  1. Independent path selection is a useful refinement of attention pruning. The paper shows that different modality paths within the same head need not share redundancy. At 30% masking, path-wise selection retains 95.3% of grounding performan. This supports the proposed granularity and its potential to preserve fine-grained visual capabilities.

  2. The method has practical efficiency evidence and broad evaluation coverage. The authors measure decoder prefill latency rather than relying only on FLOPs. The evaluation covers understanding, reasoning, OCR, and grounding on two models, with additional multi-image and video results. Post-trained dense references and recovery-based baseline comparisons help separate compression effects from the benefit of additional training.

  3. The evaluation covers capabilities that compression can easily damage. The 12 benchmarks include OCR and grounding alongside general understanding and reasoning. Evaluation on a second model and transfer to multi-image and video tasks after single-image recovery provide evidence that the method is useful across different tasks and input formats.

  4. The recovery experiments include useful controls. The dense reference is also post-trained, and the authors provide baseline results using the same recovery data. Comparisons against random selection support the importance criterion, while the FFN rescoring ablation tests whether attention pruning should affect subsequent channel selection. These controls help separate the pruning decisions from the benefit of additional training.

Critical weaknesses
  • The methodological novelty over closely related work is limited. RedundancyLens already combines sparse visual attention with visual FFN channel selection, while YOPO includes visual attention head and FFN channel pruning. ViCA selectively retains cross-modal attention while removing visual attention and FFN updates. These works already establish the value of reducing visual computation differently from text computation. MWOP's main advance is finer selection of V2V, T2V, and T2T paths within each head and its coordination with FFN pruning. Besides, some relevant works, like ATV-pruning and focusllava, are also missing from the discussion.

  • The added value of independent path pruning is not isolated after recovery. Both adopted configurations keep text FFNs dense, and Table 5 shows that most latency savings come from visual FFN compression. The training-free path comparisons establish the benefit of finer granularity before recovery, while the FFN rescoring ablation supports recomputing channel importance. Neither comparison shows how much path pruning adds over visual FFN pruning alone under the same recovery protocol. A comparison reporting both performance and latency is needed to establish the contribution of the main new component.

  • The sensitivity analysis and deployed attention use different masking operators. Section 3.1 removes path contributions after softmax without renormalization, whereas Appendices A.1 and D.5 describe masking before softmax and renormalizing retained keys at inference. These operations generally produce different outputs for T2V and T2T pruning. The Taylor score therefore measures sensitivity to a different intervention from the deployed pruning operation. This weakens the connection between the path-redundancy analysis and the implemented method, and the operator used during recovery and benchmark evaluation needs clarification.

Questions for authors
  1. Clarification and independent validation. Is the full benchmark suite used in Algorithm 1 identical to the samples reported in the result tables? Please clarify the separation between calibration, candidate selection, recovery training, and final evaluation. If evaluation samples were used for selection, can you report results after fixing all configurations using a separate development set?

  2. Evidence for the additional contribution. Under the same recovery protocol and comparable FLOP budgets, how do visual FFN pruning alone and full MWOP compare in benchmark performance and measured prefill latency? A comparison with a simpler visual-attention strategy combined with the same FFN pruning would further clarify the benefit of independent path selection.

  3. Clarification of the masking operator. Section 3.1 removes path contributions after softmax without renormalization, whereas Appendices A.1 and D.5 describe masking before softmax and renormalizing retained keys at inference. Which operator is used during recovery and benchmark evaluation? Do the quality evaluation and timed kernels implement the same operator, and do the key path-sensitivity findings hold for that operator?

8 comments
AuthorsOpenReview ↗

Response to ZMxd (Part 1/6)

We thank you for your thorough review and appreciate both the positive feedback and the constructive comments. Please find our responses to your points below. In the revised manuscript, newly added or substantially modified text is highlighted in blue, while previously existing passages that already address your concerns are highlighted in yellow for ease of reference. Please let us know if you have any further questions or suggestions.

W1: The methodological novelty over closely related work is limited. RedundancyLens already combines sparse visual attention with visual FFN channel selection, while YOPO includes visual attention head and FFN channel pruning. ViCA selectively retains cross-modal attention while removing visual attention and FFN updates. These works already establish the value of reducing visual computation differently from text computation. MWOP's main advance is finer selection of V2V, T2V, and T2T paths within each head and its coordination with FFN pruning. Besides, some relevant works, like ATV-pruning and focusllava, are also missing from the discussion.

A1: We respectfully disagree with the assessment of limited novelty. MWOP provides a unified and systematic view of modality-aware width compression across attention and FFNs. Our analyses examine V2V, T2V, and T2T interactions within attention heads, together with visual- and text-side executions of shared FFN channels, revealing differences in redundancy and task sensitivity. This provides the community with a more systematic understanding of which computations can be reduced and how pruning affects performance. Building on these findings, MWOP treats each head–path pair and each modality-specific FFN channel execution as a separately selectable pruning unit, enabling more flexible allocation of compression budgets across attention paths, modalities, and layers.

  • Relation to the closest methods: Table R1 summarizes the compression targets, granularity, action spaces, and selection rules of the four methods.
    • Compression targets and granularity: For attention, RedundancyLens targets V2V at selected layers, YOPO couples V2V and T2V in head-level pruning, and ViCA removes all V2V while selecting T2V by layer. For FFNs, RedundancyLens and YOPO perform channel-level selection for visual-token execution, whereas ViCA removes visual FFN updates entirely. Their compression strategies keep T2T and text-side FFNs dense. MWOP systematically considers all three attention paths and both FFN modalities, making each head–path pair and each modality-specific execution of a shared FFN channel separately selectable. These choices are motivated by our findings that importance differs across paths within the same head and across modalities within the same channel.

Table R1. Summary of MWOP and related methods across compression targets, granularity, action spaces and selection rules.

DimensionMWOPRedundancyLensYOPOViCA
Targets: AttnV2V, T2V, T2TV2VV2V, T2VV2V, T2V
Granularity: AttnLayer × Path × HeadLayerLayer × HeadLayer × Path
Action Space: AttnSeparately prune V2V, T2V, T2T within each headReplace V2V with local-window attention in selected layers.① Replace V2V with local-window attention ② Coupled V2V&T2V head pruning ③ Visual-related Early Exit (V2V&T2V)① Prune all V2V ② retain or disable T2V by layer
Selection rule: Attn① Taylor-based head–path units ranking ② Staged search for path-specific pruning ratiosValidation-based greedy layer ranking, with late-layer priority① Visual/text head-output ratio thresholding ② preset window size and exit depth① Fixed removal for V2V ② Identify key T2V layers using KL output-impact and representation-change diagnostics
Targets: FFNVision, TextVisionVisionVision
Granularity: FFNLayer × Modality × ChannelLayer × ChannelLayer × ChannelLayer
Action Space: FFNSeparately prune channels for visual and text execution.Prune channels for visual execution in selected layers① Prune channels for visual execution ② Visual-related Early Exit (Visual FFN)Prune all visual execution
Selection rule: FFN① Taylor-based modality-channel units ranking ② Staged search for modality-specific pruning ratios① Layer ranking ② Online top-20% selection by mean absolute probe activation① Random channel selection at the configured ratio ② preset exit depth.Fixed removal
AuthorsOpenReview ↗

Response to ZMxd (Part 2/6)

W1: The methodological novelty over closely related work is limited. RedundancyLens already combines sparse visual attention with visual FFN channel selection, while YOPO includes visual attention head and FFN channel pruning. ViCA selectively retains cross-modal attention while removing visual attention and FFN updates. These works already establish the value of reducing visual computation differently from text computation. MWOP's main advance is finer selection of V2V, T2V, and T2T paths within each head and its coordination with FFN pruning. Besides, some relevant works, like ATV-pruning and focusllava, are also missing from the discussion.

  • Relation to the closest methods (continued): See Table R1 in Part 1 for the comparison overview.

    • Compression actions and selection rules: RedundancyLens and YOPO replace full causal V2V attention with local windows, reducing token connectivity within heads. MWOP selects which heads execute each path while preserving the original causal connectivity of retained paths. These are different compression dimensions, as illustrated by YOPO’s additional coupled V2V/T2V head pruning on top of local-window attention. ViCA removes all V2V and retains T2V only at selected layers. For FFNs, RedundancyLens dynamically selects visual channels using probe activations in selected layers. YOPO randomly prunes visual channels and stops visual FFN computation after early exit, while ViCA removes visual FFN updates entirely. MWOP uses modality-specific Taylor importance to select separate visual and text channel masks, and staged search to determine their pruning ratios. Both attention and FFN widths can be allocated nonuniformly across layers.
    • Summary and Empirical Evidence: Taken together, these three methods establish the value of treating visual and text computation differently, while keeping T2T and text-side FFNs dense. MWOP provides a systematic understanding of width redundancy across both modalities and uses these findings to guide separate execution choices within shared heads and FFN channels. In Table R2, MWOP achieves 99.7% ATR, compared with 98.0% for RedundancyLens and 66.0% for YOPO, while retaining 99.9% General/Reasoning and 99.6% OCR/Grounding performance. For ViCA, we evaluate a variant retaining T2V at every layer, which achieves 64.1% ATR and 35.2% OCR/Grounding retention. These results support the effectiveness of MWOP’s fine-grained, modality-aware width allocation in preserving near-dense performance across both task groups.

    Table R2. LLaVA-OneVision-7B: retention relative to the dense model (%). MWOP, recovered RedundancyLens and YOPO correspond to Table 3; the ViCA row is our additional reproduction.

    MethodFLOPs (%)ATR (%)GRR (%)OGR (%)OCR (%)Grounding (%)
    RedundancyLens58.498.098.897.297.796.7
    YOPO70.066.091.140.841.640.0
    ViCA (Keep all T2V)12.564.193.035.246.224.2
    MWOP56.899.799.999.698.9100.2
    • Additional Related Work: We have added discussions of ATV-Pruning and FocusLLaVA to the revised Related Work section. The key distinction lies in the object of compression. ATV-Pruning uses modality-aware calibration to sparsify weights shared by visual and text tokens. FocusLLaVA compresses visual representations through region-adaptive downsampling and instruction-guided token selection. MWOP directly selects head–path and modality-specific FFN channel executions, controlling the computation applied to retained tokens. This complements visual token compression, as demonstrated by our combinations with PyramidDrop and ZOO-Prune: adding MWOP to ZOO-Prune reduces OV-7B decoder prefill latency by 31.8% relative to ZOO-Prune alone.
AuthorsOpenReview ↗

Response to ZMxd (Part 3/6)

W2: The added value of independent path pruning is not isolated after recovery. Both adopted configurations keep text FFNs dense, and Table 5 shows that most latency savings come from visual FFN compression. The training-free path comparisons establish the benefit of finer granularity before recovery, while the FFN rescoring ablation supports recomputing channel importance. Neither comparison shows how much path pruning adds over visual FFN pruning alone under the same recovery protocol. A comparison reporting both performance and latency is needed to establish the contribution of the main new component.

Q2: Evidence for the additional contribution. Under the same recovery protocol and comparable FLOP budgets, how do visual FFN pruning alone and full MWOP compare in benchmark performance and measured prefill latency? A comparison with a simpler visual-attention strategy combined with the same FFN pruning would further clarify the benefit of independent path selection.

A2: To quantify the contribution of attention-path pruning after recovery, we compare FFN-only and joint pruning with fixed FFN configurations (Table G1), and five compressed configurations at comparable FLOP budgets (Table G2), all under the same recovery protocol.

Contribution of path pruning with fixed FFN configurations. Our manuscript compares attention-first (AF) and FFN-first (FA) search in Table 7 and Appendix E. Here, we use FA as an example, keeping the FFN pruning configuration fixed and applying the same recovery protocol to F and FA. Across the three models, adding attention-path pruning maintains comparable ATR while further reducing prefill latency by 0.86–2.70%. This comparison shows that additional attention computation can be removed after FFN pruning has achieved near-lossless performance retention; the resulting speedup depends on input length.

Table G1. Recovered FFN-only versus joint pruning with fixed FFN configurations. Mask order (% pruned): V2V / T2V / T2T / V-FFN / T-FFN.

ModelConfigurationMask (%)ATR (%)GRR (%)OGR (%)Prefill (ms)
LLaVA-OneVision-7BF0 / 0 / 0 / 50 / 1099.699.799.6151.99
FA50 / 70 / 0 / 50 / 10100.099.3100.6147.88
Qwen2.5-VL-7BF0 / 0 / 0 / 60 / 099.699.699.734.43
FA50 / 50 / 0 / 60 / 099.599.699.333.91
Qwen2.5-VL-3BF0 / 0 / 0 / 50 / 099.799.799.620.99
FA50 / 50 / 0 / 50 / 099.599.599.420.81
  • ①→②→③→⑥: Visual FFN pruning provides substantial short-input acceleration, while joint attention–FFN budget allocation further improves aggregate retention at comparable latency. In Table G2, relative to the dense model’s 49.8 ms (1.00×), RedundancyLens FFN-only achieves 99.2% ATR at 36.5 ms (1.36×), whereas MWOP FFN-only achieves 99.3% ATR at 33.3 ms (1.50×) under the same recovery protocol and approximately matched FLOPs, demonstrating the effectiveness of MWOP’s FFN strategy itself. Full MWOP further raises ATR to 99.4% at the same reported latency, with GRR increasing from 99.4% to 99.6%. Thus, MWOP combines an effective FFN strategy with joint attention–FFN allocation that provides an additional aggregate quality gain.

  • ①→④→⑤→⑥: MWOP path selection improves the quality–latency trade-off over local-window attention under the same FFN selection rule. In Table G2, relative to the dense reference, RedundancyLens achieves 99.2% ATR at 37.5 ms (1.33×). Replacing its FFN strategy with MWOP FFN pruning reduces latency to 34.6 ms (1.44×), with 99.1% ATR. Keeping the MWOP FFN selection rule, replacing local-window attention with MWOP path selection further raises ATR to 99.4% and reduces prefill latency to 33.3 ms (1.50×). Attention latency decreases from 9.4 ms (0.87×) to 7.9 ms (1.04×). This directly addresses the requested simpler-attention comparison: full MWOP achieves higher ATR, GRR, and OGR with approximately 3.8% lower prefill latency than the hybrid configuration.

Table G2. Recovered Qwen2.5-VL-7B configurations at approximately 51.4% retained decoder FLOPs, with the dense reference.

ConfigurationATR (%)GRR (%)OGR (%)Attention (ms)FFN (ms)Prefill (ms)
① Dense100.0100.0100.08.2 (1.00×)40.7 (1.00×)49.8 (1.00×)
② Red. FFN-only99.299.199.38.1 (1.01×)27.2 (1.50×)36.5 (1.36×)
③ MWOP FFN-only99.399.499.38.1 (1.01×)24.1 (1.69×)33.3 (1.50×)
④ RedundancyLens99.299.199.29.4 (0.87×)27.1 (1.50×)37.5 (1.33×)
⑤ Red. Attn + MWOP FFN99.199.399.09.4 (0.87×)24.2 (1.68×)34.6 (1.44×)
⑥ Full MWOP99.599.699.37.9 (1.04×)24.4 (1.67×)33.3 (1.50×)
Reviewer_8T2nOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MWOP, which prunes modality-specific attention paths within individual heads and modality-conditioned FFN channel executions using Taylor importance scores, followed by LoRA recovery training. Evaluations on two 7B multimodal models demonstrate reduced decoder-prefill latency with near-dense performance retention and additional acceleration when combined with token compression.

Critical strengths

The paper demonstrates differing redundancy across attention paths and modality-specific FFN executions, providing empirical support for more selective pruning.

MWOP achieves practical decoder-prefill speedups with near-dense performance retention and further acceleration when combined with token compression.

Evaluation covers general understanding, reasoning, OCR, and grounding on two 7B models, with additional multi-image and video experiments.

Critical weaknesses

The most distinctive contribution is pruning modality-specific paths within individual attention heads, but this component contributes little to the measured speedup; most savings come from reducing visual-side FFN channel computation. Although the paper evaluates FFN rescoring after attention pruning, it lacks a direct comparison between FFN-only pruning and joint attention–FFN pruning under the same recovery protocol. The additional accuracy–latency benefit of path-level pruning therefore remains unclear.

Sparsity selection relies on staged ratio sweeps and separate recovery training for multiple candidates. This procedure could require substantial time and compute, yet the paper does not report the total number of recovered candidates, search time, or GPU hours. The rules for selecting and expanding the candidate set are also underspecified, making the practical cost and reproducibility difficult to assess.

Newer, stronger models increasingly incorporate linear or hybrid attention. For example, Qwen3.8-27B combines linear and full attention, so the proposed softmax path-pruning scheme may not apply directly across its layers. Larger models would also make repeated candidate evaluation and recovery training more expensive, raising concerns about the approach’s applicability and optimization cost beyond the evaluated 7B models.

Questions for authors
  1. How does FFN-only pruning compare with joint attention–FFN pruning under the same recovery protocol and comparable compute or latency budgets? This would help isolate the practical benefit of attention-path pruning.

  2. How are candidates selected and expanded during sparsity search? Please specify the rules behind DropCandidates and report the number of recovery runs and total GPU hours required to obtain the final configuration.

  3. Can the method be adapted to hybrid linear/full-attention models such as Qwen3.8-27B? Which components would remain applicable, and which would require redesign?

8 comments
AuthorsOpenReview ↗

# Response to 8T2n (Part 1/6)

We thank you for your thorough review and appreciate both the positive feedback and the constructive comments. Please find our responses to your points below. In the revised manuscript, newly added or substantially modified text is highlighted in blue, while previously existing passages that already address your concerns are highlighted in yellow for ease of reference. Please let us know if you have any further questions or suggestions.

W1:The most distinctive contribution is pruning modality-specific paths within individual attention heads, but this component contributes little to the measured speedup; most savings come from reducing visual-side FFN channel computation. Although the paper evaluates FFN rescoring after attention pruning, it lacks a direct comparison between FFN-only pruning and joint attention–FFN pruning under the same recovery protocol. The additional accuracy–latency benefit of path-level pruning therefore remains unclear.

Q1:How does FFN-only pruning compare with joint attention–FFN pruning under the same recovery protocol and comparable compute or latency budgets? This would help isolate the practical benefit of attention-path pruning.

A1: MWOP combines a systematic analysis of modality-dependent redundancy with a pruning framework spanning both attention and FFNs. Our findings motivate separately selecting head–path pairs and modality-specific executions of shared FFN channels, enabling joint allocation of compression budgets across layers for both attention paths and visual- and text-side FFN channel executions. Below, we quantify the additional contribution of attention-path pruning through comparisons between recovered FFN-only and joint attention–FFN configurations, together with latency measurements across input lengths.

  • Direct comparison of FFN-only and joint attention–FFN pruning. Our manuscript examines both attention-first (AF) and FFN-first (FA) search, with comparisons in Table 7 and detailed configurations and results in Appendix E. AF selects attention-path budgets before FFN budgets, whereas FA reverses this order. Here, Table G1 uses FA as an example to compare FFN-only pruning (F) with joint pruning (FA) under the same recovery protocol, while keeping the FFN pruning configuration fixed. In this example, FFN pruning provides most of the short-input acceleration. Adding attention-path pruning further reduces prefill latency by 0.86–2.70% across the three models while maintaining comparable ATR. Our main focus here is how much additional attention computation can be removed while preserving comparable performance after FFN pruning has achieved near-lossless retention. The resulting speedup depends strongly on input length: attention accounts for a relatively small share of runtime on short inputs, but its contribution grows as inputs become longer.

    Table G1. Recovered FFN-only versus joint pruning with fixed FFN configurations. This expanded table includes dense baselines, GRR, OGR, and component latencies. F denotes FFN-only pruning; FA denotes joint pruning obtained through FFN-first search. Mask order: V2V / T2V / T2T / V-FFN / T-FFN. Latency pairs show Dense → pruned, with speedups relative to dense in parentheses.

    ModelConfigurationMask (%)ATR (%)GRR (%)OGR (%)Attention (ms)FFN (ms)Prefill (ms)
    LLaVA-OneVision-7BDense0 / 0 / 0 / 0 / 0100.0100.0100.046.97181.03230.57
    F0 / 0 / 0 / 50 / 1099.699.799.646.97 → 47.24 (0.99×)181.03 → 102.84 (1.76×)230.57 → 151.99 (1.52×)
    FA50 / 70 / 0 / 50 / 10100.099.3100.646.97 → 42.98 (1.09×)181.03 → 102.69 (1.76×)230.57 → 147.88 (1.56×)
    Qwen2.5-VL-7BDense0 / 0 / 0 / 0 / 0100.0100.0100.08.4439.2948.45
    F0 / 0 / 0 / 60 / 099.699.699.78.44 → 8.46 (1.00×)39.29 → 24.56 (1.60×)48.45 → 34.43 (1.41×)
    FA50 / 50 / 0 / 60 / 099.599.699.38.44 → 8.33 (1.01×)39.29 → 24.54 (1.60×)48.45 → 33.91 (1.43×)
    Qwen2.5-VL-3BDense0 / 0 / 0 / 0 / 0100.0100.0100.07.8117.9625.78
    F0 / 0 / 0 / 50 / 099.799.799.67.81 → 7.88 (0.99×)17.96 → 12.76 (1.41×)25.78 → 20.99 (1.23×)
    FA50 / 50 / 0 / 50 / 099.599.599.47.81 → 7.54 (1.04×)17.96 → 12.85 (1.40×)25.78 → 20.81 (1.24×)
AuthorsOpenReview ↗

Response to 8T2n (Part 2/6)

W1:The most distinctive contribution is pruning modality-specific paths within individual attention heads, but this component contributes little to the measured speedup; most savings come from reducing visual-side FFN channel computation. Although the paper evaluates FFN rescoring after attention pruning, it lacks a direct comparison between FFN-only pruning and joint attention–FFN pruning under the same recovery protocol. The additional accuracy–latency benefit of path-level pruning therefore remains unclear.

Q1:How does FFN-only pruning compare with joint attention–FFN pruning under the same recovery protocol and comparable compute or latency budgets? This would help isolate the practical benefit of attention-path pruning.

A1 (continued)

  • Comparison at approximately matched FLOP budgets. Table G2 compares compressed Qwen2.5-VL-7B configurations under the same recovery protocol at approximately 51.4% retained decoder FLOPs. MWOP FFN-only achieves higher ATR and lower latency than RedundancyLens FFN-only, demonstrating the contribution of our FFN strategy. Full MWOP further raises ATR from 99.3% to 99.5% at the same reported prefill latency of 33.3 ms. Compared with Red. Attn + MWOP FFN, Full MWOP also achieves higher ATR, GRR, and OGR with 3.8% lower prefill latency, supporting the additional contribution of path selection under the same FFN selection rule.

    Table G2. Recovered Qwen2.5-VL-7B configurations at approximately 51.4% retained decoder FLOPs, with the dense reference. Parenthesized values indicate speedups relative to dense. "Red." denotes RedundancyLens.

    ConfigurationATR (%)GRR (%)OGR (%)Attention (ms)FFN (ms)Prefill (ms)
    ① Dense100.0100.0100.08.2 (1.00×)40.7 (1.00×)49.8 (1.00×)
    ② Red. FFN-only99.299.199.38.1 (1.01×)27.2 (1.50×)36.5 (1.36×)
    ③ MWOP FFN-only99.399.499.38.1 (1.01×)24.1 (1.69×)33.3 (1.50×)
    ④ RedundancyLens99.299.199.29.4 (0.87×)27.1 (1.50×)37.5 (1.33×)
    ⑤ Red. Attn + MWOP FFN99.199.399.09.4 (0.87×)24.2 (1.68×)34.6 (1.44×)
    ⑥ Full MWOP99.599.699.37.9 (1.04×)24.4 (1.67×)33.3 (1.50×)
  • Input-length dependence of acceleration. At fixed model width, attention interactions scale quadratically with sequence length, whereas FFN computation scales linearly. Consistent with this trend, Table G4 shows that dense attention latency catches up with FFN latency at approximately 43,000 visual tokens and exceeds it at longer inputs. At 50,000 tokens, joint pruning achieves 1.84× attention acceleration and 1.68× FFN acceleration, reducing total prefill latency from 5,925.07 to 3,406.91 ms (1.74×) relative to dense. These results show that FFN compression remains effective, while attention pruning contributes substantial latency savings at longer input lengths, supporting joint compression of both components.

    Table G4. Dense and jointly pruned Qwen2.5-VL-7B decoder latencies across visual input lengths. Each pair reads Dense → joint pruning; parenthesized values are speedups relative to dense at the same length. All latencies are in milliseconds. Attention and FFN are timed separately from total prefill.

    Visual tokensAttention latency (ms)FFN latency (ms)Prefill latency (ms)
    50010.59 → 10.73 (0.99×)43.94 → 34.80 (1.26×)55.17 → 45.29 (1.22×)
    5,00086.72 → 75.09 (1.15×)291.60 → 185.41 (1.57×)357.84 → 255.55 (1.40×)
    10,000195.60 → 157.25 (1.24×)513.10 → 323.32 (1.59×)751.98 → 502.22 (1.50×)
    20,000529.42 → 356.23 (1.49×)1,064.75 → 637.21 (1.67×)1,694.48 → 1,074.33 (1.58×)
    40,0001,992.40 → 1,110.01 (1.79×)2,107.30 → 1,251.05 (1.68×)4,188.79 → 2,480.00 (1.69×)
    42,5002,272.28 → 1,259.45 (1.80×)2,299.89 → 1,366.06 (1.68×)4,657.64 → 2,724.58 (1.71×)
    43,0002,299.42 → 1,269.69 (1.81×)2,269.81 → 1,351.17 (1.68×)4,646.45 → 2,720.26 (1.71×)
    43,5002,340.47 → 1,308.23 (1.79×)2,309.53 → 1,378.08 (1.68×)4,758.29 → 2,775.46 (1.71×)
    50,0003,037.13 → 1,648.06 (1.84×)2,622.47 → 1,559.91 (1.68×)5,925.07 → 3,406.91 (1.74×)
AuthorsOpenReview ↗

Response to 8T2n (Part 3/6)

W2:Sparsity selection relies on staged ratio sweeps and separate recovery training for multiple candidates. This procedure could require substantial time and compute, yet the paper does not report the total number of recovered candidates, search time, or GPU hours. The rules for selecting and expanding the candidate set are also underspecified, making the practical cost and reproducibility difficult to assess.

Q2:How are candidates selected and expanded during sparsity search? Please specify the rules behind DropCandidates and report the number of recovery runs and total GPU hours required to obtain the final configuration.

A2: We use the attention-first (Attn-first) search as an example to explain the complete workflow and report its measured and estimated costs.

  • Search procedure. In Attn-first search, we sequentially select pruning ratios for V2V, T2V, T2T, visual FFNs, and text FFNs. Previously selected masks remain fixed when searching the next component. The procedure comprises four steps:

    • Importance estimation. We use 256 samples per single-task calibration to estimate first-order Taylor importance. Attention-path scores are collected in a single scoring pass over the calibration samples, using activations and their corresponding loss gradients. After selecting the attention masks, we recompute FFN importance under those masks. The resulting rankings are reused across all pruning ratios within each stage, avoiding repeated importance estimation for individual candidates.

    • Training-free candidate screening. We sweep pruning ratios in steps of 10 percentage points using 100 examples from each of the 14 evaluation splits. For each task, we record the first ratio at which retention drops by more than five percentage points from the preceding ratio or reaches 80% of the dense reference or below. At the first ratio where at least three distinct tasks have triggered, DropCandidates selects that ratio and its valid immediate neighbors. For example, a trigger at 50% yields 40%, 50%, and 60%. This produces at most three candidates per stage. The candidate set is fixed before recovery and is not expanded afterward.

    • Independent recovery and stage selection. Each candidate starts from the same initial model weights and undergoes one epoch of LoRA recovery with its cumulative pruning mask fixed. We compare recovered candidates on the full benchmark splits used for final reporting and select the candidate whose overall performance retention, ATR, is closest to 100%. Only the selected mask is carried forward to the next stage; the recovered weights are not. Once these candidates have been recovered and evaluated, the search proceeds to the next component.

    • Final configuration selection. After all stages, we start from the final-stage selection and scan the recovered candidates in search order. A candidate replaces the current selection only if it achieves both higher ATR and lower FLOPs.

  • Search cost and cost reduction.

    • Measured cost and current limitation. Table R2a reports the measured mean GPU costs of one recovery run and one full benchmark evaluation in the Attn-first search, alongside estimated costs with execution optimizations enabled. The measured costs were obtained without enabling the execution optimizations used in our accelerated inference implementation. GPU·h is calculated as elapsed time in hours multiplied by the number of GPUs used. Recovery is the more expensive of these two steps for all three backbones. Independently recovering and evaluating multiple candidates incurs substantial offline computation, which is a limitation of our current search procedure.

      Table R2a. Measured baseline costs and estimated costs with training and evaluation optimizations (GPU·h per run). Optimized costs are extrapolated from training microbenchmarks and sampled benchmark timings. Parenthesized percentages indicate estimated reductions relative to the corresponding measured baseline costs.

      MLLMRecovery (measured)Optimized recovery (estimated)Evaluation (measured)Optimized evaluation (estimated)
      OneVision-7B271.4214.2 (21.1%)51.818.7 (63.9%)
      Qwen2.5-VL-7B41.531.2 (24.8%)20.912.3 (41.1%)
      Qwen2.5-VL-3B28.513.7 (51.9%)20.49.0 (55.9%)
Reviewer_ERnhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new method, MWOP, for pruning MLLMs. Most existing methods prune a combination of tokens and depth to reduce FLOPs. MWOP extends this and proposes pruning of model components, such as attention and FFN, that should be modality-dependent; i.e., attention heads and FFN channels can have different importance for text and visual tokens. The idea is intuitive, and the method is easy to understand and implement. The authors evaluate LLaVA-7B and Qwen-7B models and show prefill speedups with minimal performance degradation after LoRA fine-tuning.

Critical strengths
  1. The paper is easy to read, and the method is intuitive. The authors show that different modality tokens should be handled separately during pruning, and their analysis of LLaVA supports this claim.

  2. Extensive empirical evaluation supports the claim. MWOP outperform the existing method both in terms of performance retention and prefill speedup (~2x).

  3. Detailed ablation and analysis are provided to verify that different modality tokens have different impacts on pruning attention and FFN.

Critical weaknesses
  1. Limited comparison for pruning criteria: The method uses Taylor pruning and compares it with a random mask. Comparison with different pruning heuristics will be helpful for readers.

  2. Results are evaluated with both 7B-sized models. One thing that will make the analysis stronger is to establish whether the same redundancy pattern holds for smaller-sized models.

Questions for authors

MWOP performs head-wise path-level attention pruning in the order V2V → T2V→ T2T, guided by first-order Taylor importance. (L184)

How was this order decided? Did you try pruning with different orders?

5 comments
AuthorsOpenReview ↗

Response to ERnh (Part 1/3)

We thank you for your thorough review and appreciate both the positive feedback and the constructive comments. Please find our responses to your points below. In the revised manuscript, newly added or substantially modified text is highlighted in blue, while previously existing passages that already address your concerns are highlighted in yellow for ease of reference. Please let us know if you have any further questions or suggestions.

W1: Limited comparison for pruning criteria: The method uses Taylor pruning and compares it with a random mask. Comparison with different pruning heuristics will be helpful for readers.

A1: We have added Fisher [1] and GateProbe [2] to the attention-path and FFN-channel comparisons on LLaVA-OneVision-7B. Besides random masking, the original submission included Attention Mass and Attention Contribution for attention paths, and Activation and Wanda-style for FFN channels. All comparisons are performed without recovery training, using the same pruning units, calibration samples, and matched masking ratios, with matched per-layer channel budgets for FFN pruning.

  • Figure 7 (Appendix A.1) and Figure 13 (Appendix B.1) show how performance changes with the masking ratio for separately masked attention paths and FFN modalities, respectively. For convenient reference within this response, Tables R1 and R2 summarize performance retention relative to the unmasked baseline at 20%, 40%, 60%, and 80% masking. Each benchmark entry equally averages 12 settings in Table R1 (three path types × four ratios) or eight settings in Table R2 (two FFN modalities × four ratios). Mean averages the four benchmark entries equally; bold marks the highest value in each column.

  • Attention-path pruning. Taylor achieves the highest average retention on all four benchmarks, with a mean of 85.5%, compared with 81.2% for GateProbe. Its largest advantage is on RefCOCOg: 78.3%, versus 67.9% for the best alternative, Attention Mass.

    Table R1. Attention-path pruning without recovery: average performance retention (%).

    CriterionGQAMMStarTextVQARefCOCOgMean
    Attention Mass89.379.784.867.980.4
    Attention Contribution75.270.371.748.766.5
    Fisher88.878.587.962.979.5
    GateProbe90.481.588.164.681.2
    Taylor90.982.889.978.385.5
  • FFN-channel pruning. Taylor achieves the highest average retention on GQA, MMStar, and TextVQA, and the highest four-benchmark mean (89.0%, versus 88.4% for Fisher). Wanda-style leads on RefCOCOg (75.8%, versus 75.3% for Taylor), while the detailed curves show that Fisher and GateProbe match or exceed Taylor in some individual settings.

    Table R2. FFN-channel pruning without recovery: average performance retention (%).

    CriterionGQAMMStarTextVQARefCOCOgMean
    Activation91.489.092.074.786.8
    Wanda-style91.089.190.975.886.7
    Fisher93.291.093.775.688.4
    GateProbe92.589.793.473.987.4
    Taylor94.491.994.575.389.0
  • Together, these results support Taylor as an effective shared importance criterion for attention paths and modality-specific FFN channel executions under the evaluated settings.

[1] Theis L, Korshunova I, Tejani A, et al. Faster gaze prediction with dense networks and fisher pruning[J]. arXiv preprint arXiv:1801.05787, 2018.

[2] Sun G, Feng K, He S, et al. Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?[J]. arXiv preprint arXiv:2606.27755, 2026.

AuthorsOpenReview ↗

Response to ERnh (Part 2/3)

W2: Results are evaluated with both 7B-sized models. One thing that will make the analysis stronger is to establish whether the same redundancy pattern holds for smaller-sized models.

A2: We added experiments on Qwen2.5-VL-3B, extending our evaluation from the two original 7B models (LLaVA-OneVision-7B and Qwen2.5-VL-7B) to a 3B model. Given the rebuttal time constraints, we focused the 3B search on V2V, T2V, and visual FFNs. T2T and text-side FFN masking were not included in this search, and both components were kept dense.

  • After recovery, attention-only pruning with 50% V2V and 60% T2V masking retains 99.6% ATR (Table 21, Appendix D.3), while visual-FFN-only pruning at 50% masking retains 99.7% ATR (Table 25, Appendix E.3). As shown in Table G5, joint pruning retains 99.5% ATR under both search orders, with prefill speedups of 1.18× for attention-first (AF) and 1.24× for FFN-first (FA). These results show exploitable redundancy in both visual attention and visual FFN computation at the 3B scale, consistent with the visual compressibility observed in the 7B models.

    Table G5. Joint pruning on Qwen2.5-VL-3B after recovery. This expanded table includes attention-only and FFN-only configurations and component latencies. Mask entries report removal percentages in the order [V2V / T2V / T2T / V-FFN / T-FFN]. Performance retention is relative to the corresponding post-trained dense model. Latencies are reported as Dense → Pruned (speedup).

    ConfigurationMask (%)ATR (%)GRR (%)OGR (%)Attention latency (ms)FFN latency (ms)Prefill latency (ms)
    Attn-only50 / 60 / 0 / 0 / 099.699.999.37.81 → 7.45 (1.05×)17.96 → 17.78 (1.01×)25.78 → 25.72 (1.00×)
    FFN-only0 / 0 / 0 / 50 / 099.799.799.67.81 → 7.88 (0.99×)17.96 → 12.76 (1.41×)25.78 → 20.99 (1.23×)
    AF50 / 60 / 0 / 40 / 099.599.899.27.81 → 7.48 (1.04×)17.96 → 13.96 (1.29×)25.78 → 21.82 (1.18×)
    FA50 / 50 / 0 / 50 / 099.599.599.47.81 → 7.54 (1.04×)17.96 → 12.85 (1.40×)25.78 → 20.81 (1.24×)
AuthorsOpenReview ↗

Response to ERnh (Part 3/3)

Q1: MWOP performs head-wise path-level attention pruning in the order V2V → T2V→ T2T, guided by first-order Taylor importance. How was this order decided? Did you try pruning with different orders?

A3: The path order is a heuristic motivated by information flow and sensitivity observations. It is not uniquely required by causality, and our additional experiments show that changing the order of attention and FFN search can change the selected operating point.

  • Path Order Based on Information Flow and Pruning Sensitivity: In the visual-before-text setting considered in our analysis, the causal mask permits V2V, T2V, and T2T interactions while blocking V2T interactions, since visual queries cannot attend to subsequent text tokens. This causal structure motivates our heuristic search order from the perspective of information flow: visual aggregation (V2V), text queries reading visual information (T2V), and text-side interaction (T2T). Our masking experiments further show that V2V and T2V generally tolerate more pruning than T2T, motivating its placement last in the search. The causal mask constrains token visibility, but does not prescribe the pruning order or establish that V2V-first outperforms T2V-first.

  • FFN Order Based on Computational Cost and Pruning Sensitivity: Visual tokens account for most FFN computation in our single-image setting, while Figure 4(b) shows that text-side FFNs generally require a more conservative pruning budget. Searching visual FFNs before text FFNs therefore prioritizes the component with greater acceleration potential and lower observed sensitivity.

  • Effect of Module Search Order: In our additional order experiments, we kept V2V → T2V → T2T fixed and varied only the order of attention and FFN search, comparing attention-first (AF) with FFN-first (FA). Each stage selects a component’s pruning ratio with previously selected masks fixed and unsearched components kept dense, so changing the order can change the selected pruning ratios across components. On Qwen2.5-VL-7B, FA selects more visual FFN pruning and less attention-path pruning than AF. As shown in Table G3, both configurations retain 99.5% ATR, with comparable GRR and OGR. Relative to AF, FA reduces FFN latency from 30.09 to 24.54 ms, while attention latency increases slightly from 8.22 to 8.33 ms. Total prefill latency decreases from 39.25 to 33.91 ms, with speedup over the dense model increasing from 1.23× to 1.43×.

    Table G3. Effect of module search order on recovered Qwen2.5-VL-7B. Mask entries report removal percentages in the order [V2V / T2V / T2T / V-FFN / T-FFN]. T2T paths and text FFNs remain dense in both settings. Performance retention is relative to the corresponding dense model. Latencies are reported as Dense → Pruned (speedup).

    Search orderMask (%)ATR (%)GRR (%)OGR (%)Attention latency (ms)FFN latency (ms)Prefill latency (ms)
    AF60 / 70 / 0 / 40 / 099.599.799.28.44 → 8.22 (1.03×)39.29 → 30.09 (1.31×)48.45 → 39.25 (1.23×)
    FA50 / 50 / 0 / 60 / 099.599.699.38.44 → 8.33 (1.01×)39.29 → 24.54 (1.60×)48.45 → 33.91 (1.43×)

Comments on the paper

5 comments
AuthorsOpenReview ↗

Global Response (Part 1/5)

We thank all reviewers for their thorough reviews and constructive feedback. We appreciate their recognition of MWOP’s fine-grained, modality-aware width pruning across attention paths and FFN channel executions (all reviewers). They also highlight its strong performance retention across diverse tasks and model backbones (all reviewers). Results on compression-sensitive OCR and grounding tasks, together with transfer to multi-image and video inputs after recovery on single-image data, support its practical applicability (ZMxd). In terms of efficiency, reviewers recognize its measured prefill speedups (all reviewers) and further acceleration when combined with token compression (8T2n).

Across the reviews, we identified a few shared themes, which we address below along with additional points we would like to highlight:

  • Novelty (ZMxd):

    • MWOP provides a systematic understanding of modality-dependent width redundancy across attention and FFNs. Our analysis covers V2V, T2V, and T2T interactions within attention heads, together with visual- and text-side executions of shared FFN channels. It reveals that importance differs across paths within the same head and across modalities within the same channel, while pruning sensitivity also varies across tasks. These findings motivate independently selecting head–path pairs and modality-specific FFN channel executions, with nonuniform width budgets across layers for both attention and FFNs. MWOP thus connects a systematic characterization of width redundancy with a concrete strategy for exploiting it, providing the community with evidence about which computations can be reduced and how this affects different capabilities. Compared with the three closest methods discussed in our response to Reviewer ZMxd (W1), MWOP brings both visual and text computation into width-pruning decisions; detailed comparisons are provided there.

    • Attention and FFN compression are equally important parts of our framework. Their relative contributions to acceleration depend on input length: the FFN-dominated speedups in our main efficiency table reflect the short-input setting, where FFNs account for most runtime. As visual inputs grow, attention occupies an increasing share of runtime. We discuss this input-length dependence and the contributions of both components in the next point.

  • Technical Contributions and Supporting Evidence (8T2n, ZMxd): We summarize the evidence for fine-grained width selection, the practical contributions of attention and FFN compression, and how these contributions depend on search order and input length.

    • Value of fine-grained selection: Our analyses reveal differences in redundancy across paths within the same head and across modalities within shared FFN channels, together with task-dependent pruning sensitivity. These findings directly inform compression decisions: removing an entire head can discard useful interactions alongside redundant ones. On OV-7B, at the same 30% masking ratio, independent path selection retains 95.3% of Grounding performance before recovery, compared with 49.3% for whole-head pruning (Figure 3(b)). The modality-dependent importance of shared FFN channels likewise motivates separate visual and text execution masks. These analyses provide evidence about where width can be reduced while preserving different capabilities.
AuthorsOpenReview ↗

Global Response (Part 2/5)

  • Technical Contributions and Supporting Evidence (8T2n, ZMxd) (Continued)
    • Component contributions after recovery:
      • We first compare FFN-only pruning (F) with joint pruning obtained through FFN-first search (FA), keeping the FFN configuration fixed and using the same recovery protocol. Table G1 reports the pruning configurations alongside performance and latency. Across three models, adding attention-path pruning maintains comparable ATR while further reducing short-input prefill latency by 0.86–2.70%. Our main focus here is how much additional attention computation can be removed while preserving comparable performance after FFN pruning has achieved near-lossless retention. The resulting speedup depends on input length, as shown in Table G4.

        Table G1. Recovered FFN-only versus joint pruning with fixed FFN configurations. Mask entries are removal percentages in the order V2V / T2V / T2T / V-FFN / T-FFN. ATR is relative to the corresponding dense model.

        ModelF mask (%)FA mask (%)ATR (%), F → FAPrefill (ms), F → FA
        LLaVA-OneVision-7B0 / 0 / 0 / 50 / 1050 / 70 / 0 / 50 / 1099.6 → 100.0151.99 → 147.88
        Qwen2.5-VL-7B0 / 0 / 0 / 60 / 050 / 50 / 0 / 60 / 099.6 → 99.534.43 → 33.91
        Qwen2.5-VL-3B0 / 0 / 0 / 50 / 050 / 50 / 0 / 50 / 099.7 → 99.520.99 → 20.81
      • We also compare configurations under the same recovery protocol at approximately matched FLOP budgets (Table G2). MWOP FFN-only achieves higher ATR and lower latency than RedundancyLens FFN-only, supporting the effectiveness of our FFN strategy itself. Full MWOP further raises ATR from 99.3% to 99.5% at the same reported prefill latency of 33.3 ms. Compared with RedundancyLens attention combined with the same MWOP FFN selection rule, full MWOP achieves higher ATR, GRR, and OGR with 3.8% lower prefill latency. These comparisons distinguish the value of FFN selection from the additional contribution of attention-path selection and joint budget allocation.

        Table G2. Recovered Qwen2.5-VL-7B configurations at approximately 51.4% retained decoder FLOPs, with the dense reference. “Red.” denotes RedundancyLens. Parenthesized values indicate speedups relative to dense. Module latencies and total prefill are timed separately.

        ConfigurationATR (%)GRR (%)OGR (%)Attention (ms)FFN (ms)Prefill (ms)
        Dense100.0100.0100.08.2 (1.00×)40.7 (1.00×)49.8 (1.00×)
        Red. FFN-only99.299.199.38.1 (1.01×)27.2 (1.50×)36.5 (1.36×)
        MWOP FFN-only99.399.499.38.1 (1.01×)24.1 (1.69×)33.3 (1.50×)
        RedundancyLens99.299.199.29.4 (0.87×)27.1 (1.50×)37.5 (1.33×)
        Red. Attn + MWOP FFN99.199.399.09.4 (0.87×)24.2 (1.68×)34.6 (1.44×)
        Full MWOP99.599.699.37.9 (1.04×)24.4 (1.67×)33.3 (1.50×)
AuthorsOpenReview ↗

Global Response (Part 3/5)

  • Technical Contributions and Supporting Evidence (8T2n, ZMxd) (Continued)
    • Effect of module search order: We compare attention-first (AF) and FFN-first (FA) search while keeping the V2V→T2V→T2T order fixed. On Qwen2.5-VL-7B, FA selects more visual FFN pruning and less attention-path pruning than AF. Both configurations retain 99.5% ATR, but FA reduces prefill latency from 39.25 to 33.91 ms, increasing speedup over dense from 1.23× to 1.43× (Table G3). This shows that module search order affects the final allocation of compression budgets and the resulting quality–latency trade-off.

      Table G3. Effect of module search order on recovered Qwen2.5-VL-7B. Mask entries are removal percentages in the order V2V / T2V / T2T / V-FFN / T-FFN. Latency entries show Dense → Pruned, with speedups relative to dense in parentheses.

      Search orderMask (%)ATR (%)GRR (%)OGR (%)Attention (ms)FFN (ms)Prefill (ms)
      AF60 / 70 / 0 / 40 / 099.599.799.28.44 → 8.22 (1.03×)39.29 → 30.09 (1.31×)48.45 → 39.25 (1.23×)
      FA50 / 50 / 0 / 60 / 099.599.699.38.44 → 8.33 (1.01×)39.29 → 24.54 (1.60×)48.45 → 33.91 (1.43×)
    • Input-length-dependent acceleration: At fixed model width, attention interactions scale quadratically with sequence length, whereas FFN computation scales linearly. In our Qwen2.5-VL-7B measurements, dense attention latency becomes comparable to FFN latency at approximately 43,000 visual tokens and exceeds it at longer inputs. At 50,000 tokens, joint pruning achieves 1.84× attention acceleration and 1.68× FFN acceleration, reducing prefill latency from 5,925.07 to 3,406.91 ms (1.74× relative to dense). These measurements explain why FFNs dominate the short-input speedups in the main efficiency table, while attention pruning also saves substantial time at longer inputs, supporting compression of both components.

      Table G4. Dense and jointly pruned Qwen2.5-VL-7B decoder latencies across visual input lengths. Each pair reads Dense → Joint; parenthesized values are speedups relative to dense at the same length. All latencies are in milliseconds. Attention and FFN are timed separately from total prefill.

      Visual tokensAttention latency (ms)FFN latency (ms)Prefill latency (ms)
      50010.59 → 10.73 (0.99×)43.94 → 34.80 (1.26×)55.17 → 45.29 (1.22×)
      5,00086.72 → 75.09 (1.15×)291.60 → 185.41 (1.57×)357.84 → 255.55 (1.40×)
      10,000195.60 → 157.25 (1.24×)513.10 → 323.32 (1.59×)751.98 → 502.22 (1.50×)
      20,000529.42 → 356.23 (1.49×)1,064.75 → 637.21 (1.67×)1,694.48 → 1,074.33 (1.58×)
      40,0001,992.40 → 1,110.01 (1.79×)2,107.30 → 1,251.05 (1.68×)4,188.79 → 2,480.00 (1.69×)
      42,5002,272.28 → 1,259.45 (1.80×)2,299.89 → 1,366.06 (1.68×)4,657.64 → 2,724.58 (1.71×)
      43,0002,299.42 → 1,269.69 (1.81×)2,269.81 → 1,351.17 (1.68×)4,646.45 → 2,720.26 (1.71×)
      43,5002,340.47 → 1,308.23 (1.79×)2,309.53 → 1,378.08 (1.68×)4,758.29 → 2,775.46 (1.71×)
      50,0003,037.13 → 1,648.06 (1.84×)2,622.47 → 1,559.91 (1.68×)5,925.07 → 3,406.91 (1.74×)
    • Complementarity with token compression: Token compression reduces sequence length, while MWOP reduces the attention-path and FFN-channel computation applied to retained tokens. Their complementarity is demonstrated by our combinations with PyramidDrop and ZOO-Prune, with task performance reported in Table 4 and practical efficiency in Table 5 (left). In particular, adding MWOP to ZOO-Prune reduces OV-7B decoder prefill latency from 119.0 to 81.2 ms, a further 31.8% reduction relative to ZOO-Prune alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.