- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MWOP reduces multimodal decoder computation by independently pruning V2V, T2V, and T2T attention paths within each head and selecting FFN channels separately for visual and text tokens. First-order Taylor scores guide pruning, and FFN importance is recomputed under the selected attention mask before LoRA recovery.
On LLaVA-OneVision-7B, the paper reports 43.2% fewer decoder FLOPs, 99.7% average relative performance kpt, and a 1.6× decoder prefill speedup.
Critical strengths
-
Independent path selection is a useful refinement of attention pruning. The paper shows that different modality paths within the same head need not share redundancy. At 30% masking, path-wise selection retains 95.3% of grounding performan. This supports the proposed granularity and its potential to preserve fine-grained visual capabilities.
-
The method has practical efficiency evidence and broad evaluation coverage. The authors measure decoder prefill latency rather than relying only on FLOPs. The evaluation covers understanding, reasoning, OCR, and grounding on two models, with additional multi-image and video results. Post-trained dense references and recovery-based baseline comparisons help separate compression effects from the benefit of additional training.
-
The evaluation covers capabilities that compression can easily damage. The 12 benchmarks include OCR and grounding alongside general understanding and reasoning. Evaluation on a second model and transfer to multi-image and video tasks after single-image recovery provide evidence that the method is useful across different tasks and input formats.
-
The recovery experiments include useful controls. The dense reference is also post-trained, and the authors provide baseline results using the same recovery data. Comparisons against random selection support the importance criterion, while the FFN rescoring ablation tests whether attention pruning should affect subsequent channel selection. These controls help separate the pruning decisions from the benefit of additional training.
Critical weaknesses
-
The methodological novelty over closely related work is limited. RedundancyLens already combines sparse visual attention with visual FFN channel selection, while YOPO includes visual attention head and FFN channel pruning. ViCA selectively retains cross-modal attention while removing visual attention and FFN updates. These works already establish the value of reducing visual computation differently from text computation. MWOP's main advance is finer selection of V2V, T2V, and T2T paths within each head and its coordination with FFN pruning. Besides, some relevant works, like ATV-pruning and focusllava, are also missing from the discussion.
-
The added value of independent path pruning is not isolated after recovery. Both adopted configurations keep text FFNs dense, and Table 5 shows that most latency savings come from visual FFN compression. The training-free path comparisons establish the benefit of finer granularity before recovery, while the FFN rescoring ablation supports recomputing channel importance. Neither comparison shows how much path pruning adds over visual FFN pruning alone under the same recovery protocol. A comparison reporting both performance and latency is needed to establish the contribution of the main new component.
-
The sensitivity analysis and deployed attention use different masking operators. Section 3.1 removes path contributions after softmax without renormalization, whereas Appendices A.1 and D.5 describe masking before softmax and renormalizing retained keys at inference. These operations generally produce different outputs for T2V and T2T pruning. The Taylor score therefore measures sensitivity to a different intervention from the deployed pruning operation. This weakens the connection between the path-redundancy analysis and the implemented method, and the operator used during recovery and benchmark evaluation needs clarification.
Questions for authors
-
Clarification and independent validation. Is the full benchmark suite used in Algorithm 1 identical to the samples reported in the result tables? Please clarify the separation between calibration, candidate selection, recovery training, and final evaluation. If evaluation samples were used for selection, can you report results after fixing all configurations using a separate development set?
-
Evidence for the additional contribution. Under the same recovery protocol and comparable FLOP budgets, how do visual FFN pruning alone and full MWOP compare in benchmark performance and measured prefill latency? A comparison with a simpler visual-attention strategy combined with the same FFN pruning would further clarify the benefit of independent path selection.
-
Clarification of the masking operator. Section 3.1 removes path contributions after softmax without renormalization, whereas Appendices A.1 and D.5 describe masking before softmax and renormalizing retained keys at inference. Which operator is used during recovery and benchmark evaluation? Do the quality evaluation and timed kernels implement the same operator, and do the key path-sensitivity findings hold for that operator?
8 comments
Response to ZMxd (Part 2/6)
W1: The methodological novelty over closely related work is limited. RedundancyLens already combines sparse visual attention with visual FFN channel selection, while YOPO includes visual attention head and FFN channel pruning. ViCA selectively retains cross-modal attention while removing visual attention and FFN updates. These works already establish the value of reducing visual computation differently from text computation. MWOP's main advance is finer selection of V2V, T2V, and T2T paths within each head and its coordination with FFN pruning. Besides, some relevant works, like ATV-pruning and focusllava, are also missing from the discussion.
-
Relation to the closest methods (continued): See Table R1 in Part 1 for the comparison overview.
- Compression actions and selection rules: RedundancyLens and YOPO replace full causal V2V attention with local windows, reducing token connectivity within heads. MWOP selects which heads execute each path while preserving the original causal connectivity of retained paths. These are different compression dimensions, as illustrated by YOPO’s additional coupled V2V/T2V head pruning on top of local-window attention. ViCA removes all V2V and retains T2V only at selected layers. For FFNs, RedundancyLens dynamically selects visual channels using probe activations in selected layers. YOPO randomly prunes visual channels and stops visual FFN computation after early exit, while ViCA removes visual FFN updates entirely. MWOP uses modality-specific Taylor importance to select separate visual and text channel masks, and staged search to determine their pruning ratios. Both attention and FFN widths can be allocated nonuniformly across layers.
- Summary and Empirical Evidence: Taken together, these three methods establish the value of treating visual and text computation differently, while keeping T2T and text-side FFNs dense. MWOP provides a systematic understanding of width redundancy across both modalities and uses these findings to guide separate execution choices within shared heads and FFN channels. In Table R2, MWOP achieves 99.7% ATR, compared with 98.0% for RedundancyLens and 66.0% for YOPO, while retaining 99.9% General/Reasoning and 99.6% OCR/Grounding performance. For ViCA, we evaluate a variant retaining T2V at every layer, which achieves 64.1% ATR and 35.2% OCR/Grounding retention. These results support the effectiveness of MWOP’s fine-grained, modality-aware width allocation in preserving near-dense performance across both task groups.
Table R2. LLaVA-OneVision-7B: retention relative to the dense model (%). MWOP, recovered RedundancyLens and YOPO correspond to Table 3; the ViCA row is our additional reproduction.
Method FLOPs (%) ATR (%) GRR (%) OGR (%) OCR (%) Grounding (%) RedundancyLens 58.4 98.0 98.8 97.2 97.7 96.7 YOPO 70.0 66.0 91.1 40.8 41.6 40.0 ViCA (Keep all T2V) 12.5 64.1 93.0 35.2 46.2 24.2 MWOP 56.8 99.7 99.9 99.6 98.9 100.2 - Additional Related Work: We have added discussions of ATV-Pruning and FocusLLaVA to the revised Related Work section. The key distinction lies in the object of compression. ATV-Pruning uses modality-aware calibration to sparsify weights shared by visual and text tokens. FocusLLaVA compresses visual representations through region-adaptive downsampling and instruction-guided token selection. MWOP directly selects head–path and modality-specific FFN channel executions, controlling the computation applied to retained tokens. This complements visual token compression, as demonstrated by our combinations with PyramidDrop and ZOO-Prune: adding MWOP to ZOO-Prune reduces OV-7B decoder prefill latency by 31.8% relative to ZOO-Prune alone.
Response to ZMxd (Part 3/6)
W2: The added value of independent path pruning is not isolated after recovery. Both adopted configurations keep text FFNs dense, and Table 5 shows that most latency savings come from visual FFN compression. The training-free path comparisons establish the benefit of finer granularity before recovery, while the FFN rescoring ablation supports recomputing channel importance. Neither comparison shows how much path pruning adds over visual FFN pruning alone under the same recovery protocol. A comparison reporting both performance and latency is needed to establish the contribution of the main new component.
Q2: Evidence for the additional contribution. Under the same recovery protocol and comparable FLOP budgets, how do visual FFN pruning alone and full MWOP compare in benchmark performance and measured prefill latency? A comparison with a simpler visual-attention strategy combined with the same FFN pruning would further clarify the benefit of independent path selection.
A2: To quantify the contribution of attention-path pruning after recovery, we compare FFN-only and joint pruning with fixed FFN configurations (Table G1), and five compressed configurations at comparable FLOP budgets (Table G2), all under the same recovery protocol.
Contribution of path pruning with fixed FFN configurations. Our manuscript compares attention-first (AF) and FFN-first (FA) search in Table 7 and Appendix E. Here, we use FA as an example, keeping the FFN pruning configuration fixed and applying the same recovery protocol to F and FA. Across the three models, adding attention-path pruning maintains comparable ATR while further reducing prefill latency by 0.86–2.70%. This comparison shows that additional attention computation can be removed after FFN pruning has achieved near-lossless performance retention; the resulting speedup depends on input length.
Table G1. Recovered FFN-only versus joint pruning with fixed FFN configurations. Mask order (% pruned): V2V / T2V / T2T / V-FFN / T-FFN.
| Model | Configuration | Mask (%) | ATR (%) | GRR (%) | OGR (%) | Prefill (ms) |
|---|---|---|---|---|---|---|
| LLaVA-OneVision-7B | F | 0 / 0 / 0 / 50 / 10 | 99.6 | 99.7 | 99.6 | 151.99 |
| FA | 50 / 70 / 0 / 50 / 10 | 100.0 | 99.3 | 100.6 | 147.88 | |
| Qwen2.5-VL-7B | F | 0 / 0 / 0 / 60 / 0 | 99.6 | 99.6 | 99.7 | 34.43 |
| FA | 50 / 50 / 0 / 60 / 0 | 99.5 | 99.6 | 99.3 | 33.91 | |
| Qwen2.5-VL-3B | F | 0 / 0 / 0 / 50 / 0 | 99.7 | 99.7 | 99.6 | 20.99 |
| FA | 50 / 50 / 0 / 50 / 0 | 99.5 | 99.5 | 99.4 | 20.81 |
-
①→②→③→⑥: Visual FFN pruning provides substantial short-input acceleration, while joint attention–FFN budget allocation further improves aggregate retention at comparable latency. In Table G2, relative to the dense model’s 49.8 ms (1.00×), RedundancyLens FFN-only achieves 99.2% ATR at 36.5 ms (1.36×), whereas MWOP FFN-only achieves 99.3% ATR at 33.3 ms (1.50×) under the same recovery protocol and approximately matched FLOPs, demonstrating the effectiveness of MWOP’s FFN strategy itself. Full MWOP further raises ATR to 99.4% at the same reported latency, with GRR increasing from 99.4% to 99.6%. Thus, MWOP combines an effective FFN strategy with joint attention–FFN allocation that provides an additional aggregate quality gain.
-
①→④→⑤→⑥: MWOP path selection improves the quality–latency trade-off over local-window attention under the same FFN selection rule. In Table G2, relative to the dense reference, RedundancyLens achieves 99.2% ATR at 37.5 ms (1.33×). Replacing its FFN strategy with MWOP FFN pruning reduces latency to 34.6 ms (1.44×), with 99.1% ATR. Keeping the MWOP FFN selection rule, replacing local-window attention with MWOP path selection further raises ATR to 99.4% and reduces prefill latency to 33.3 ms (1.50×). Attention latency decreases from 9.4 ms (0.87×) to 7.9 ms (1.04×). This directly addresses the requested simpler-attention comparison: full MWOP achieves higher ATR, GRR, and OGR with approximately 3.8% lower prefill latency than the hybrid configuration.
Table G2. Recovered Qwen2.5-VL-7B configurations at approximately 51.4% retained decoder FLOPs, with the dense reference.
| Configuration | ATR (%) | GRR (%) | OGR (%) | Attention (ms) | FFN (ms) | Prefill (ms) |
|---|---|---|---|---|---|---|
| ① Dense | 100.0 | 100.0 | 100.0 | 8.2 (1.00×) | 40.7 (1.00×) | 49.8 (1.00×) |
| ② Red. FFN-only | 99.2 | 99.1 | 99.3 | 8.1 (1.01×) | 27.2 (1.50×) | 36.5 (1.36×) |
| ③ MWOP FFN-only | 99.3 | 99.4 | 99.3 | 8.1 (1.01×) | 24.1 (1.69×) | 33.3 (1.50×) |
| ④ RedundancyLens | 99.2 | 99.1 | 99.2 | 9.4 (0.87×) | 27.1 (1.50×) | 37.5 (1.33×) |
| ⑤ Red. Attn + MWOP FFN | 99.1 | 99.3 | 99.0 | 9.4 (0.87×) | 24.2 (1.68×) | 34.6 (1.44×) |
| ⑥ Full MWOP | 99.5 | 99.6 | 99.3 | 7.9 (1.04×) | 24.4 (1.67×) | 33.3 (1.50×) |
Response to ZMxd (Part 1/6)
We thank you for your thorough review and appreciate both the positive feedback and the constructive comments. Please find our responses to your points below. In the revised manuscript, newly added or substantially modified text is highlighted in blue, while previously existing passages that already address your concerns are highlighted in yellow for ease of reference. Please let us know if you have any further questions or suggestions.
A1: We respectfully disagree with the assessment of limited novelty. MWOP provides a unified and systematic view of modality-aware width compression across attention and FFNs. Our analyses examine V2V, T2V, and T2T interactions within attention heads, together with visual- and text-side executions of shared FFN channels, revealing differences in redundancy and task sensitivity. This provides the community with a more systematic understanding of which computations can be reduced and how pruning affects performance. Building on these findings, MWOP treats each head–path pair and each modality-specific FFN channel execution as a separately selectable pruning unit, enabling more flexible allocation of compression budgets across attention paths, modalities, and layers.
Table R1. Summary of MWOP and related methods across compression targets, granularity, action spaces and selection rules.