- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on efficient pruning for multimodal MoE. It proposes a more efficient expert-sensitivity estimation method, used to allocate each expert's channel budget. Channels are then selected based on token activation, specifically arguing that selection should use the activation of the modality tokens that an expert specializes in. Finally, the paper performs engineering optimizations that reduce inference-time memory usage and speed up inference.
Critical strengths
- Uses a Gauss-Newton matrix to measure expert importance, which is more efficient.
- Proposes MACS, which makes channel pruning for multimodal experts more precise, and proposes cross-layer rearrangement (CLER) to improve deployment efficiency (lower memory, higher throughput) under the channel-pruning scheme.
Critical weaknesses
The paper claims an exact Gauss-Newton matrix for measuring expert importance, but this "exactness" holds only under one particular definition of removal (zeroing out, without renormalization). The theoretical framework of REAP, a baseline the paper itself cites, shows that another common removal definition (renormalization) introduces an independent, non-vanishing error term ("renormalization error"). The paper does not discuss whether this error exists under its own setup, or how large it would be, which undermines the "exact expert-importance criterion" claimed as one of its core contributions.
Questions for authors
Figure 1(a)'s Pareto curve does not include ACP, even though ACP is the best-performing baseline on average and the most closely related method, sharing MAES's channel-level granularity. Could the authors add ACP's curve to this figure?
3 comments
Response to Reviewer RZoT (2/2)
[Q1]Figure 1(a)'s Pareto curve does not include ACP, even though ACP is the best-performing baseline on average and the most closely related method, sharing MAES's channel-level granularity. Could the authors add ACP's curve to this figure?
We thank the reviewer for pointing out our omission. As per the reviewer's advice, we run ACP on Kimi-VL-A3B-Instruct on GQA at seven pruning ratios, from target to . We have added the ACP results to the Pareto frontier in Figure 1(a). The corresponding results are shown below.
| Model | Method | p0.1 | p0.2 | p0.3 | p0.4 | p0.5 | p0.6 | p0.7 |
|---|---|---|---|---|---|---|---|---|
| Kimi-VL | ACP | 62.32 | 62.25 | 61.95 | 60.86 | 59.09 | 51.52 | 45.99 |
| Kimi-VL | MAES | 62.65 | 62.89 | 62.04 | 61.23 | 60.68 | 58.48 | 54.35 |
Table R2: Accuracy of ACP and MAES on GQA at different pruning ratios (Kimi-VL-A3B-Instruct).
We can observe that ACP and MAES are close, but ACP degrades sharply at higher ratios. Thus, MAES lies on the Pareto frontier of compression ratio and accuracy, especially in the high-compression regime.
Revision
- Restate the exactness claim of our SOES in Sec. 4.2, and narrow the claim to single-expert removal (zeroed, without renormalization) under fixed routing, where the second-order expansion of the squared reconstruction loss is exact.
- Clarify that our method does not use renormalization of expert weights, and add Table R1 (the
zeroedvs.renormalizedcomparison from REAP), in Appendix F.4 (Table 22 of the revised paper), which compares the block-output MSE and the number of samples whose first answer token differs from the unpruned baseline. - Add ACP to the Pareto frontier in Figure 1(a), and add the table of ACP vs. MAES accuracy on GQA across pruning ratios (Table R2) in Appendix F.1.
Thank you for the additional experiments and the detailed response. All of my concerns have been addressed. I will continue to maintain a positive evaluation.
Response to Reviewer RZoT (1/2)
We'd like to thank Reviewer RZoT for pointing out an important scoping issue in our exactness claim and for the constructive comments on ACP.
Response to [W1]: Yes, we agree that the exactness claim is valid only when the expert output on a given token is zeroed out without renormalization, and this is, in fact, exactly the setting in our paper. The probe sets βe=0 for the removed expert and keeps all other gates at 1, without renormalization. We have made this point clearer in our revised paper (Sec. 4.2).
Response to [W2]: We agree that renormalization introduces the "renormalization error" pointed out by REAP. Under the renormalization definition, when an expert is removed, the gates of the remaining selected experts are rescaled by 1/(1−ge), which adds an error term that does not vanish.
Response to [W3]: We would like to clarify that our implementation does not have this renormalization error. Pruned experts are skipped after the top-K selection: the router still selects the top-K experts over the full (unpruned) expert pool. The pruned experts are handled as follows:
zeroed). This is mathematically the same operator as the probe's βe=0. The probe therefore measures exactly what is executed at inference time, and the renormalization error term does not arise in our setup.We also add an experiment in Table R1 that compares
zeroedandrenormalized. Forrenormalized, we renormalize the remaining expert from the original top-8 experts. We compute the MSE on the block output, andrenormalizedhas 23.3% higher MSE thanzeroed. At the output answer level, the difference is small: on the 1024 tested samples, the first answer token differs from the unpruned baseline in 73/1024 samples (7.13%) forzeroedvs. 81/1024 samples (7.91%) forrenormalized, a difference of 8 samples. Thus, in our setting, zeroing is consistent with the probe and also slightly closer to the unpruned model than renormalization.Table R1:
zeroedvs.renormalized