acceptodds
Under review as a conference paper at ICLR 2027

MAES: Modality-Aware Expert Slimming for Multimodal Mixture-of-Experts

Abstract

The rapid adoption of Mixture-of-Experts (MoE) in multimodal large language models has enabled remarkable capabilities, yet at the cost of critical bottlenecks in storage, memory, and inference speed. Structural pruning offers fine-grained compression that permanently reduces memory and computation, but existing methods treat all tokens uniformly and ignore the fundamental heterogeneity between modalities. Through systematic analysis, we uncover two key phenomena in multimodal MoEs: (i) experts exhibit pronounced modality affinity, inducing modality-specific redundancy; and (ii) different modalities exhibit different activation magnitude distributions, rendering modality-agnostic channel scoring unreliable. These findings motivate our Modality-Aware Expert Slimming (MAES), a training-free structural pruning and inference framework that proposes Expert Modality Affinity (EMA)-guided budget allocation according to each modality's contribution. To rank expert redundancy, we further introduce an exact second-order expert-importance criterion, replacing noisy proxies used in prior work. For deployment, we rearrange experts jointly across layers to balance weight-memory load across GPUs, which seamlessly integrates with modern inference frameworks and preserves efficient expert parallelism. Across 12 image and video benchmarks and 3 model families, MAES consistently outperforms existing MoE pruning methods, retaining over 98% of original performance at 30% channel pruning while attaining lower memory usage and higher throughput, achieving the state-of-the-art accuracy–efficiency trade-off. Our code is released at https://anonymous.4open.science/r/MAES.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MoE Compression, Mixture of experts, Expert parallelism, Model pruning, Sparsification

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_RZoTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on efficient pruning for multimodal MoE. It proposes a more efficient expert-sensitivity estimation method, used to allocate each expert's channel budget. Channels are then selected based on token activation, specifically arguing that selection should use the activation of the modality tokens that an expert specializes in. Finally, the paper performs engineering optimizations that reduce inference-time memory usage and speed up inference.

Critical strengths
  1. Uses a Gauss-Newton matrix to measure expert importance, which is more efficient.
  2. Proposes MACS, which makes channel pruning for multimodal experts more precise, and proposes cross-layer rearrangement (CLER) to improve deployment efficiency (lower memory, higher throughput) under the channel-pruning scheme.
Critical weaknesses

The paper claims an exact Gauss-Newton matrix for measuring expert importance, but this "exactness" holds only under one particular definition of removal (zeroing out, without renormalization). The theoretical framework of REAP, a baseline the paper itself cites, shows that another common removal definition (renormalization) introduces an independent, non-vanishing error term ("renormalization error"). The paper does not discuss whether this error exists under its own setup, or how large it would be, which undermines the "exact expert-importance criterion" claimed as one of its core contributions.

Questions for authors

Figure 1(a)'s Pareto curve does not include ACP, even though ACP is the best-performing baseline on average and the most closely related method, sharing MAES's channel-level granularity. Could the authors add ACP's curve to this figure?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RZoT (1/2)

We'd like to thank Reviewer RZoT for pointing out an important scoping issue in our exactness claim and for the constructive comments on ACP.

[W1] The paper claims an exact Gauss-Newton matrix for measuring expert importance, but this "exactness" holds only under one particular definition of removal (zeroing out, without renormalization).

[W2] The theoretical framework of REAP, a baseline the paper itself cites, shows that another common removal definition (renormalization) introduces an independent, non-vanishing error term ("renormalization error").

[W3] The paper does not discuss whether this error exists under its own setup, or how large it would be, which undermines the "exact expert-importance criterion" claimed as one of its core contributions.

Response to [W1]: Yes, we agree that the exactness claim is valid only when the expert output on a given token is zeroed out without renormalization, and this is, in fact, exactly the setting in our paper. The probe sets βe=0\beta_e=0 for the removed expert and keeps all other gates at 1, without renormalization. We have made this point clearer in our revised paper (Sec. 4.2).

Response to [W2]: We agree that renormalization introduces the "renormalization error" pointed out by REAP. Under the renormalization definition, when an expert is removed, the gates of the remaining selected experts are rescaled by 1/(1−ge)1/(1-g_e), which adds an error term that does not vanish.

Response to [W3]: We would like to clarify that our implementation does not have this renormalization error. Pruned experts are skipped after the top-K selection: the router still selects the top-K experts over the full (unpruned) expert pool. The pruned experts are handled as follows:

  1. After tokens are assigned to experts, the token IDs routed to a pruned expert are mapped to -1.
  2. These tokens are skipped in token dispatch and do not enter the Grouped GEMM computation.
  3. When the expert outputs are aggregated, the logits of the pruned experts stay in the softmax denominator, so the weights of the remaining experts are not rescaled.
  4. A pruned expert never computes the token, so its output for that token–expert pair can be treated as zero (zeroed). This is mathematically the same operator as the probe's βe=0\beta_e=0. The probe therefore measures exactly what is executed at inference time, and the renormalization error term does not arise in our setup.

We also add an experiment in Table R1 that compares zeroed and renormalized. For renormalized, we renormalize the remaining expert from the original top-8 experts. We compute the MSE on the block output, and renormalized has 23.3% higher MSE than zeroed. At the output answer level, the difference is small: on the 1024 tested samples, the first answer token differs from the unpruned baseline in 73/1024 samples (7.13%) for zeroed vs. 81/1024 samples (7.91%) for renormalized, a difference of 8 samples. Thus, in our setting, zeroing is consistent with the probe and also slightly closer to the unpruned model than renormalization.

MetricZeroedRenormalized
Block MSE compared to unpruned baseline2.57×10−42.57\times10^{-4}3.17×10−43.17\times10^{-4}
Samples with a different first token compared to unpruned baseline73/1024 (7.13%)81/1024 (7.91%)

Table R1: zeroed vs. renormalized

AuthorsOpenReview ↗

Response to Reviewer RZoT (2/2)

[Q1] Figure 1(a)'s Pareto curve does not include ACP, even though ACP is the best-performing baseline on average and the most closely related method, sharing MAES's channel-level granularity. Could the authors add ACP's curve to this figure?

We thank the reviewer for pointing out our omission. As per the reviewer's advice, we run ACP on Kimi-VL-A3B-Instruct on GQA at seven pruning ratios, from target p=0.1p=0.1 to p=0.7p=0.7. We have added the ACP results to the Pareto frontier in Figure 1(a). The corresponding results are shown below.

ModelMethodp0.1p0.2p0.3p0.4p0.5p0.6p0.7
Kimi-VLACP62.3262.2561.9560.8659.0951.5245.99
Kimi-VLMAES62.6562.8962.0461.2360.6858.4854.35

Table R2: Accuracy of ACP and MAES on GQA at different pruning ratios (Kimi-VL-A3B-Instruct).

We can observe that ACP and MAES are close, but ACP degrades sharply at higher ratios. Thus, MAES lies on the Pareto frontier of compression ratio and accuracy, especially in the high-compression regime.


Revision

  1. Restate the exactness claim of our SOES in Sec. 4.2, and narrow the claim to single-expert removal (zeroed, without renormalization) under fixed routing, where the second-order expansion of the squared reconstruction loss is exact.
  2. Clarify that our method does not use renormalization of expert weights, and add Table R1 (the zeroed vs. renormalized comparison from REAP), in Appendix F.4 (Table 22 of the revised paper), which compares the block-output MSE and the number of samples whose first answer token differs from the unpruned baseline.
  3. Add ACP to the Pareto frontier in Figure 1(a), and add the table of ACP vs. MAES accuracy on GQA across pruning ratios (Table R2) in Appendix F.1.

Thank you for the additional experiments and the detailed response. All of my concerns have been addressed. I will continue to maintain a positive evaluation.

Reviewer_VaSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MAES, a training-free framework for structural pruning of multimodal MoE models. It combines modality-aware channel selection, a second-order expert sensitivity criterion, and cross-layer expert placement for efficient deployment. Experiments across three multimodal models and twelve benchmarks report improved average accuracy over pruning baselines, alongside reduced deployment memory usage.

Critical strengths

The paper addresses a practically important compression problem and motivates its approach with observations of modality-dependent routing and activation patterns. The reported average accuracy improvements are consistent across all six model–pruning-ratio settings. The deployment evaluation also goes beyond parameter counts by measuring memory usage and prefill/decode throughput, providing useful evidence of practical applicability.

Critical weaknesses

The novelty of SOES is insufficiently established given ACP’s existing exact second-order variant; a clearer distinction and direct comparison are needed. The necessity of EMA-based allocation also remains unclear, as the ablations do not isolate its contribution from modality-specific scoring or compare it against simple normalization and equal-weight alternatives.å

Questions for authors
  1. How does SOES differ from—and compare against—ACP’s exact second-order variant?
  2. Does EMA outperform simple modality normalization or equal-weight ranking under fixed expert budgets?
  3. Can you isolate width rounding from expert placement in the CLER ablation?
6 comments
AuthorsOpenReview ↗

Response to Reviewer VaSr (1/4)

We thank Reviewer VaSr for the insightful comments.

[W1] The novelty of SOES is insufficiently established given ACP's existing exact second-order variant; a clearer distinction and direct comparison are needed.

[Q1] How does SOES differ from—and compare against—ACP's exact second-order variant?

We thank the reviewer for the careful reading. As the reviewer notes, SOES and ACP do share one thing in common: both use second-order information to evaluate experts. In particular, ACP (Appendix C.3.2 and Table 14 of its paper) already perturbs each expert's output with a scalar αe\alpha_e and computes a second-order score. Its implementation explicitly requires αe<1\alpha_e<1 and uses αe=0.95\alpha_e=0.95 as the default setting (the same as its first-order setting).

Theoretically, ACP's score is not exact because it scales all experts simultaneously by αe=0.95\alpha_e=0.95. When it computes the second-order score of one expert, the other experts are scaled at the same time, which introduces a background error. We measure this error per expert as ∣ΔLe/De−1∣|\Delta L_e/D_e-1|, where ΔLe\Delta L_e is the removal increment of expert ee under the αe=0.95\alpha_e=0.95 setting and De=12HeeD_e=\tfrac12 H_{ee} is the identity-point removal loss (fixed routing, blockwise quadratic MSE loss).

In contrast, SOES derives a closed form of the expert-removal loss from the blockwise MSE loss with an identity probe β=1\beta=\mathbf1, ΔLℓ,eremove=12Hee=1Nℓ∑h∈Dwℓ,h∥aℓ,e(h)∥22\Delta\mathcal L^{\mathrm{remove}}_{\ell,e}=\tfrac12H_{ee}=\frac{1}{N_\ell}\sum_{h\in\mathcal D}w_{\ell,h}\|a_{\ell,e}(h)\|_2^2 (where aℓ,e=gℓ,ezℓ,ea_{\ell,e}=g_{\ell,e}z_{\ell,e} is the router-weighted expert output, and wℓ,h=1/(∥yℓteacher(h)∥22+10−6)w_{\ell,h}=1/(\|y^{\rm teacher}_\ell(h)\|_2^2+10^{-6}) normalizes each of the NℓN_\ell scoring tokens by the fixed teacher output energy of the decoder block), and proves that the diagonal entries are exact when only a single expert is removed and routing is unchanged. The core derivation is in Section 4.2 and Appendix C.

Experimentally, we find that this background error is nonnegligible. We measure it on Qwen3-VL-30B-A3B-Instruct with a 512-sample mixed calibration set, collect the per-layer expert Hessian under α=0.95\alpha=0.95 scaling, and report the results in Table R1. The results show that ACP's second-order score is not exact, especially in the first and last layers: the median error is 10.41% at the first layer and 28.62% at the last layer, while it stays around 1% in the middle layers. SOES avoids this error by expanding at the identity point β=1\beta=\mathbf 1, so it is exact for single-expert removal. We clarify that this is the core difference between SOES and ACP, which is also the main theoretical contribution of SOES.

Layer# expertsMedianMean ± STDP90
L012810.41%11.93% ± 6.40%19.96%
L11282.69%3.22% ± 1.93%5.78%
L21282.84%3.49% ± 2.28%6.61%
L31281.00%1.14% ± 0.82%2.19%
L101280.85%0.96% ± 0.69%1.58%
L201280.98%1.07% ± 0.59%1.76%
L301280.61%0.69% ± 0.43%1.26%
L401280.80%0.91% ± 0.66%1.42%
L451260.52%0.60% ± 0.51%1.17%
L461242.13%2.68% ± 1.97%4.66%
L4712328.62%29.30% ± 13.03%45.87%

Table R1: Per-expert error ∣ΔLe/De−1∣|\Delta L_e/D_e-1| of the ACP second-order variant, which scales all experts within one layer simultaneously using αe=0.95\alpha_e=0.95.

To compare the two estimators directly, we also replace SOES with ACP's second-order score inside MAES on Qwen3-VL-30B-A3B-Instruct. Both variants use MACS with EMA for channel selection. We disable CLER, however, because it rounds expert widths to a few discrete levels, which could hide the differences between the channel sets selected by the two estimators.

Table R2 shows that SOES outperforms ACP on all 8 tasks at both pruning ratios. The gap is small at 30% pruning (Δ0.32% in average accuracy) but widens at 50% pruning (Δ1.16%). We attribute this larger gap at 50% to the tighter budget: more experts lie near the pruning boundary, so the ranking errors caused by ACP's background error (Table R1) are more likely to change the channel allocation.

Pruning ratioMethod / ΔChartQAMMBenchTextVQAGQAMMStarMMVetRealWorldQAMMEAvg. accuracy (%)
30%SOES86.7686.0082.8163.0668.5168.9972.682351.176.60
ACP86.7285.2282.7963.0068.1368.9072.292329.576.28
Δ−0.04−0.78−0.02−0.06−0.38−0.09−0.39−21.6−0.32
50%SOES83.5683.4282.3260.9166.4163.3571.632183.073.70
ACP83.0481.8781.9260.8165.3261.4271.112094.872.54
Δ−0.52−1.55−0.40−0.10−1.09−1.93−0.52−88.2−1.16

Table R2: Downstream performance of SOES and ACP on Qwen3-VL-30B-A3B-Instruct. Δ = ACP − SOES.

AuthorsOpenReview ↗

Response to Reviewer VaSr (2/4)

[W2] The necessity of EMA-based allocation also remains unclear, as the ablations do not isolate its contribution from modality-specific scoring or compare it against simple normalization and equal-weight alternatives.

[Q2] Does EMA outperform simple modality normalization or equal-weight ranking under fixed expert budgets?

We thank the reviewer for pointing out this gap. EMA allocates each expert's channel budget across modalities according to which modality the expert processes more often. As per the reviewer's constructive advice, we have added the ablation results for EMA in our revision, which are summarized as follows.

We compare our EMA with three alternatives, including the two suggested by the reviewer, under the same fixed expert budget: (1) Equal quota, which fixes the visual/text weights to 1/2 each; (2) Modality normalization, which divides each expert's visual and text scores by the mean score of its own modality before fusing them with equal weights; (3) Equal-weight ranking (original top-K), which directly uses the raw visual and text scores without normalization. All four variants use the same modality-specific channel scores and the same per-expert channel budget, but differ only in the allocation rule. Thus, the comparison isolates the contribution of EMA from that of modality-specific scoring. The experiments are conducted on Qwen3-VL-30B-A3B-Instruct with 7 image benchmarks and 1 video benchmark, using the full MAES pipeline with both SOES and CLER components. For VideoMME, we only have the time to test on 1 random seed, and we will update the standard deviation in our final version.

The results in Table R3 show that no allocation rule is always best on every task. At 30% pruning, the four rules have very similar performance (averages of 76.93%–77.13%). At the more aggressive 50% pruning ratio, EMA gives the best average, which is 0.30%–0.49% higher than the other three rules, and it is also the best-performing rule on 6 of the 8 tasks. These results suggest that the benefits of EMA are particularly obvious when the channel budget is tight.

Pruning ratioMethodGQATextVQAChartQARealWorldQAMMBenchMMEMMStarVideoMME (1 seed)Avg
30%Equal quota63.02±0.0282.84±0.1286.65±0.2671.81±0.8085.88±0.472322.04±31.0869.23±0.7073.1176.93
Modality normalization63.02±0.2082.87±0.1886.76±0.1471.94±0.4985.80±0.352340.52±19.8869.02±0.5272.6376.95
Equal-weight ranking63.11±0.0782.82±0.0987.03±0.1671.85±0.5985.74±0.152346.12±15.4069.67±0.2572.7877.10
EMA63.13±0.0682.89±0.0486.77±0.2371.98±0.3885.99±0.482347.24±20.1669.24±0.6773.1977.13
50%Equal quota61.44±0.0581.89±0.3682.61±1.3671.58±0.5483.16±0.482158.80±19.3266.90±0.7569.8174.31
Modality normalization61.24±0.1081.64±0.3383.32±1.3471.11±0.3983.39±0.642155.44±38.6466.17±0.8469.2674.14
Equal-weight ranking61.32±0.2481.62±0.3683.28±1.3271.42±0.4983.22±0.352156.00±16.8066.98±0.5769.8174.33
EMA61.25±0.1582.33±0.5183.68±0.7671.63±0.4082.99±0.412189.88±7.2867.11±0.3669.8574.63

Table R3: Four allocation rules under a fixed budget: EMA, Equal quota, Modality normalization, and Equal-weight ranking.

AuthorsOpenReview ↗

Response to Reviewer VaSr (3/4)

[Q3] Can you isolate width rounding from expert placement in the CLER ablation?

We apologize for the lack of clarity, which may have confused the reviewer. The two are separated by construction.

  1. Rounding: rounding only changes each expert's width to align it with a set of predefined width levels, which affects only the accuracy of the pruned model. In our ablation study (Table 5 of the revised paper), the accuracy change brought by CLER is thus due only to width rounding, and the placement process does not change the accuracy, as shown in Table 5.
  2. Placement: After rounding, we place the experts of the same width level onto EP ranks, which affects only inference efficiency. Regarding its effectiveness, Figure 1(b) shows the results of the throughput and peak memory usage as compared with other naive placement implementations. Among them, padded means directly calling the vLLM engine without modifications, multi_kernel means launching separate kernels for different widths on each EP rank, and single_width means permanently assigning each EP rank the experts of one width level.

We have revised the manuscript accordingly to make this clearer.

Furthermore, to better evaluate the real efficiency of our placement strategy, we provide an end-to-end comparison of TTFT and decoding time at both the 30% and 50% pruning ratios in Table R4, with a batch size of 64. The results show that cross_layer (ours) matches unpruned in TTFT and per-token latency, while reducing the non-KV peak memory by about 21% at 30% pruning (18.6 → 14.7 GiB/GPU) and about 35% at 50% pruning (18.6 → 12.0 GiB/GPU). multi_kernel executes the same GFLOPs as cross_layer but is much slower, with about 2× the per-token latency and about half the decode throughput, and it has a higher non-KV peak memory. single_width permanently assigns each EP rank the experts of one width level; its latency stays close to padded, but at 30% pruning the widest level concentrates most routed channels on one rank, giving the highest non-KV peak of all strategies (28.4 GiB/GPU) and a 19.7 GiB spread across ranks. This shows that the cost of naive width-based placement is mainly memory imbalance rather than speed, and cross_layer is the only strategy that achieves both a low peak and balanced memory.

Pruning ratioConfigPrefill TTFT mean ± std (s)Time per output token, mean ± std (s)Throughput, mean (req/s)Non-KV peak, bs1 (GiB/GPU)GFLOPs/token (prefill)
0unpruned0.484±0.1480.04503±0.0011859.4618.63.62
0.3padded0.492±0.1460.04541±0.0010679.3718.63.62
multi_kernel0.718±0.1720.09343±0.0011654.8624.72.85
single_width0.521±0.1460.04655±0.0011679.1028.42.85
cross_layer (ours)0.528±0.1720.04564±0.0010639.2614.72.85
0.5padded0.498±0.1410.04471±0.0010399.5018.63.62
multi_kernel0.700±0.1510.09270±0.00081964.9019.22.14
single_width0.478±0.1340.04537±0.0010479.3715.52.14
cross_layer (ours)0.492±0.1440.04394±0.0010379.6312.02.14

Table R4: End-to-end latency, memory and FLOPs of the unpruned model and of different expert-placement strategies.

Reviewer_pX75OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MAES is a training-free structural pruning framework for multimodal Mixture-of-Experts models. It uses expert routing preferences to allocate modality-specific channel quotas, a local second-order reconstruction score to distribute pruning budgets, and cross-layer expert rearrangement to deploy experts with heterogeneous widths. The study evaluates three multimodal MoE families on twelve image and video benchmarks and reports favorable accuracy, memory, and throughput.

Critical strengths

The method addresses a real multimodal compression problem: visual and text tokens can use the same experts and channels differently. Its pruning and deployment components form a coherent pipeline. The Hessian derivation is exact for the stated affine expert-output probe with squared block reconstruction loss. Experiments span multiple models, benchmarks, pruning ratios, and deployment settings; the component ablations are useful. The presentation is clear, and the deployment result makes the contribution relevant beyond an accuracy-only pruning study.

Critical weaknesses

The exactness claim is too broad. A Hessian diagonal gives the exact local loss increase for removing one expert, but simultaneous removal includes cross-expert terms. It does not establish an exact score for the final multi-channel pruning decision (Sec. 4.2). Routing frequency sets a hard modality quota while activation magnitude ranks channels. The paper does not show that this coupling remains effective under a changed calibration mixture or test distribution. Baseline fairness needs clearer accounting. Equal nominal pruning ratios may yield different retained parameters, FLOPs, and memory across channel- and expert-level methods. Accuracy tables lack uncertainty estimates, and the MME scale requires an explicit account of how the reported average is computed. The deployment measurements are tied to one hardware and expert-parallel setup. End-to-end latency and communication costs would better establish the practical gain.

Questions for authors
  1. How large are the off-diagonal Hessian terms when multiple experts are pruned together?
  2. Are baselines matched by actual retained parameters, FLOPs, or peak memory, and how is MME handled in the average?
8 comments
AuthorsOpenReview ↗

Response to Reviewer pX75 (1/8)

We thank Reviewer pX75 for the careful review and very constructive comments.

[W1] The exactness claim is too broad. A Hessian diagonal gives the exact local loss increase for removing one expert, but simultaneous removal includes cross-expert terms. It does not establish an exact score for the final multi-channel pruning decision (Sec. 4.2).

We truly appreciate the reviewer's insightful comments. We agree that the exactness claim is too broad. We have narrowed our claim to single-expert removal under fixed routing, where the second-order expression is exact. We clarify that it is not exact for joint removal of multiple experts, nor for the final pruning, whose configuration mixes fully kept experts, partially pruned experts, and fully removed experts. In particular, Tables R1–R2 below show that, on average over layers, the interaction terms account for 0.1%–0.9% of the joint loss when 2–8 experts are removed jointly, and for 1.68% / 3.52% in the final configuration at p=0.3p=0.3 / p=0.5p=0.5. As per the reviewer's advice, we have revised the exactness claim in Section 4.2 and clarified its scope explicitly:

  • Exact: removal of a single expert, with the removed expert zeroed and the router unchanged. On the 128 experts of layer 0 of Qwen3-VL, compared one by one against independent ablations, the maximum relative error is 6.6×10−76.6\times10^{-7}.
  • Not exact: joint removal of multiple experts within one layer. The true joint loss is 121S⊤H1S\frac{1}{2} \mathbf{1}_S^\top H \mathbf{1}_S, which contains the off-diagonal terms HefH_{ef}.

We have also clarified how SOES enters the final multi-channel decision. SOES only ranks experts by importance, and each expert's channel budget is allocated according to this ranking; which channels are kept inside an expert is then decided by MACS.

AuthorsOpenReview ↗

Response to Reviewer pX75 (2/8)

[Q1] How large are the off-diagonal Hessian terms when multiple experts are pruned together?

Many thanks for the comment. We clarify that in our experiments, the off-diagonal Hessian terms are small for most layers. When 8 experts are removed jointly, they account for 0.944% of the joint loss on average over the 48 layers. In the final channel configuration of MAES, they account for 1.68% at p=0.3p=0.3 and 3.52% at p=0.5p=0.5. They are larger only in the first and last layers. In the tables below, we quantify the interaction terms both for joint expert removal (Table R1) and for the final channel configuration produced by MAES (Table R2).

To further measure how large the off-diagonal terms are when multiple experts are removed together, we also take the full Hessian (128×128128\times128) on all MoE layers of Qwen3-VL-30B-A3B-Instruct, and compute the fraction of the joint loss contributed by the interaction terms when kk experts are removed simultaneously, ∣Ljoint−Ldiag∣/∣Ljoint∣|L_{\rm joint}-L_{\rm diag}|/|L_{\rm joint}| using 1000 random sets per layer (48,000 in total).

Table R1 shows that the share of the off-diagonal HefH_{ef} is larger in the first and last layers (4.87% and 11.05% when 8 experts are removed jointly), but stays below about 1% in the middle layers.

Overall, the off-diagonal share is small.

Layer / kk248
L00.65%2.07%4.87%
L50.093%0.250%0.711%
L100.069%0.241%0.629%
L150.054%0.164%0.394%
L200.056%0.231%0.630%
L250.120%0.399%1.04%
L300.064%0.180%0.396%
L350.062%0.212%0.522%
L400.070%0.190%0.534%
L471.41%4.58%11.05%
48-layer mean0.116%0.381%0.944%

Table R1: Share of the interaction term HefH_{ef} when k∈{2,4,8}k\in\{2,4,8\} experts are randomly selected.

Meanwhile, the same decomposition also applies to the final multi-channel decision. The final configuration contains fully kept experts, partially pruned experts, and fully removed experts. Let δℓ,e(h)=aℓ,e(h)−a~ℓ,e(h)\delta_{\ell,e}(h)=a_{\ell,e}(h)-\tilde a_{\ell,e}(h) be the change in the router-weighted output of expert ee on token hh, where a~ℓ,e\tilde a_{\ell,e} is the output under the final channel mask. Here, δℓ,e=0\delta_{\ell,e}=0 for a fully kept expert, and δℓ,e=aℓ,e\delta_{\ell,e}=a_{\ell,e} for a fully removed one. Our reconstruction loss normalizes each token by the energy of the fixed teacher output of the decoder block, wℓ,h=1/(∥yℓteacher(h)∥22+10−6)w_{\ell,h}=1/(\|y^{\rm teacher}_\ell(h)\|_2^2+10^{-6}), and averages over the NℓN_\ell scoring tokens. With the input and routing of the layer fixed, the loss decomposes as

Ljoint=1Nℓ∑hwℓ,h∥∑eδℓ,e(h)∥22=1Nℓ∑hwℓ,h∑e∥δℓ,e(h)∥22⏟Ldiag+1Nℓ∑hwℓ,h∑e≠f⟨δℓ,e(h),δℓ,f(h)⟩⏟interaction.L_{\rm joint}=\frac{1}{N_\ell}\sum_{h}w_{\ell,h}\Big\|\sum_e\delta_{\ell,e}(h)\Big\|_2^2=\underbrace{\frac{1}{N_\ell}\sum_{h}w_{\ell,h}\sum_e\|\delta_{\ell,e}(h)\|_2^2}_{L_{\rm diag}}+\underbrace{\frac{1}{N_\ell}\sum_{h}w_{\ell,h}\sum_{e\neq f}\langle\delta_{\ell,e}(h),\delta_{\ell,f}(h)\rangle}_{\text{interaction}}.

Because the teacher normalization is fixed, the loss remains an exact quadratic form. Therefore, the single-expert exactness above is unaffected. Whole-expert removal is the special case δℓ,e=aℓ,e\delta_{\ell,e}=a_{\ell,e}, for which the two terms reduce to the diagonal and off-diagonal Hessian entries in Table R1. Only the experts selected for the same token can interact. Unlike the random sets in Table R1, however, in the final configuration several of the 8 experts active on a token may carry nonzero residuals, so we measure the interaction term directly in Table R2.

AuthorsOpenReview ↗

Response to Reviewer pX75 (3/8)

We take the final channel masks of MAES at p=0.3p=0.3 and p=0.5p=0.5, and compute the interaction share ∣Ljoint−Ldiag∣/∣Ljoint∣|L_{\rm joint}-L_{\rm diag}|/|L_{\rm joint}|. To check whether the small net share hides large pairwise terms that cancel each other, we also report the absolute pairwise share, i.e., the sum of the absolute values of all pairwise terms divided by LjointL_{\rm joint}. The results in Table R2 show that the interaction share is small in most layers in both pruning ratio settings: averaged over the 48 layers, it is 1.68% at p=0.3p=0.3 and 3.52% at p=0.5p=0.5, and it stays below 5% in 46/48 layers at p=0.3p=0.3 and 43/48 layers at p=0.5p=0.5. Consistent with Table R1, the first and last layers have a higher interaction share, reaching 9.60% / 21.53% at L0 and 14.42% / 25.49% at L47 (p=0.3p=0.3 / p=0.5p=0.5). The absolute pairwise share is larger than the net share (5.39% vs. 1.68% at p=0.3p=0.3 and 8.77% vs. 3.52% at p=0.5p=0.5, averaged over layers), which indicates that part of the pairwise terms cancel each other.

Layerp=0.3p=0.3p=0.5p=0.5
L09.60%21.53%
L51.65%3.74%
L101.12%2.61%
L150.74%1.69%
L201.11%2.59%
L250.97%2.73%
L301.80%3.17%
L350.82%1.94%
L400.86%1.89%
L4714.42%25.49%
48-layer mean1.68%3.52%
Layers below 5%46/4843/48
Absolute pairwise share (upper bound; 48-layer mean)5.39%8.77%

Table R2: Share of the interaction term in the block reconstruction loss in the final deployed pruned model.

In summary, SOES is not an exact score for the final multi-channel decision, and we have revised the paper accordingly to clarify this point. On the deployed configuration, the diagonal sum approximates the layer-wise joint reconstruction loss with an average interaction share of 1.68% at p=0.3p=0.3 and 3.52% at p=0.5p=0.5. These results suggest that accounting for expert interactions in the first and last layers could further improve the allocation, which we have discussed as a limitation and leave to future work.

Reviewer_5BBVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAES, a training-free channel-level pruning and inference framework for multimodal MoE LLMs. Based on the observation that experts exhibit modality affinity and that different modalities induce different activation distributions, the method (i) scores channels separately per modality and allocates each expert's budget according to its modality traffic, (ii) estimates expert importance with an exact second-order criterion, and (iii) rounds and rearranges experts across GPUs to fit modern expert-parallel inference. Experiments on 3 model families and 12 image/video benchmarks show high performance retention at 30–50% pruning, with lower memory and higher throughput under vLLM.

Critical strengths

The sensitivity estimation (SOES) can significantly reduce complexity: it computes expert importance in closed form from forward passes, replacing costly per-expert ablation sweeps.

Critical weaknesses
  1. The pruning idea itself is simple.
  2. The estimations rest on some implicit assumptions; some are supported by proofs or experiments, but not all.
  3. Whether the method still holds when these assumptions are not satisfied has not been studied.
Questions for authors
  1. Which of the implicit assumptions are proven or experimentally validated, and which are not?
  2. How does the method behave when the assumptions are not satisfied?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5BBV (1/5)

We'd like to thank the reviewer for the constructive comments.

[W1] The pruning idea itself is simple.

Thank you for the comments. Indeed, each component is simple, and this is, in fact, by design. Together, the three components in MAES are tailored to jointly solve a practical constrained optimization problem, i.e., which channels to keep when pruning a multimodal MoE model and how to deploy the resulting irregular expert widths efficiently.

To the best of our knowledge, the proposed MAES is the first dedicated method tailored for channel-level pruning of multimodal MoE models that also addresses the practical deployment problem under expert parallelism. In contrast, prior works on MoE compression have predominantly focused on different granularities or different settings: expert-wise skipping and pruning for text-only MoEs (e.g., NAEE, MC-MoE, DiEP); multimodal MoE quantization (e.g., Q-VLM, MC#); and token compression (e.g., MoDES, FastMMoE).

Channel-level pruning of multimodal MoEs remains challenging and largely unexplored in prior works for three reasons identified in our pilot experiments, and MAES is dedicated to addressing each of them:

  1. Routing preference and activation intensity can disagree in multimodal MoE token routing. 79% of experts receive tokens predominantly from one modality, yet some show a smaller activation magnitude on that very modality, so scoring channels with a single mixed signal misleads the selection.
  2. There is no exact and efficient expert-importance estimation method. Previous proxies are either imprecise or inefficient. Loss-based and gradient-based metrics can be fragile because they are sensitive to the loss construction and to outlier calibration prompts, while true ablation is too slow.
  3. Pruned experts have arbitrary widths that inference engines do not natively support. Padding to the maximum width can remove the deployment gains of weight pruning.

The three components of the proposed MAES are dedicated to simultaneously addressing three identified challenges as follows:

  • MACS solves challenge 1: It measures the modality preference of each expert and keeps the channels with the highest scores on that modality under a modality quota. The progressive-expansion rule in Eq. (1) guarantees that, during channel selection among multiple modalities, the number of retained channels never exceeds the expert's channel quota.
  • SOES solves challenge 2: A closed-form score replaces the expert-by-expert ablation and needs only a single batched forward pass. The score exactly matches the true ablation loss of removing a single expert (e.g., on Kimi-VL-A3B-Instruct at p=0.5p=0.5, Table 6: RMSE 0.0, Pearson 1.0), and the cross-expert interaction is small (0.944% of the joint loss on average when 8 experts are removed together; see [Q2-2]). By comparison, token counts, router-logit magnitude and the first-order gradient ∣gℓ,e∣\lvert g_{\ell,e}\rvert reach Pearson correlations of only 0.8109, 0.1111 and 0.6939, respectively.
  • CLER solves challenge 3: It aligns the widths and rearranges experts across GPUs, so expert parallelism runs without padding and our pruned model achieves real end-to-end efficiency gains. The non-KV peak memory per GPU is about 21.2% lower than the unpruned model at p=0.3p=0.3 (18.63 → 14.68 GiB) and 35.5% lower at p=0.5p=0.5 (18.63 → 12.02 GiB).

In our revised version, we have added a discussion of these three challenges to the Introduction and Related Work, and a comparison table of prior MoE compression methods (by compression granularity, multimodal support and training requirement, including ACP) to Appendix D.2 (Table 12 of the revised paper), to make the distinction clearer.

AuthorsOpenReview ↗

Response to Reviewer 5BBV (2/5)

[W2] The estimations rest on some implicit assumptions...Whether the method still holds when these assumptions are not satisfied...

[Q1] Which of the implicit assumptions are proven or experimentally validated, and which are not?

[Q2] How does the method behave when the assumptions are not satisfied?

We sincerely thank the reviewer for pointing out this oversight in our writing. In the revision, we have clarified these implicit assumptions explicitly and quantified the gap between each assumption and practice. In particular, Table R1 summarizes the assumptions made in this paper:

AssumptionStatus and evidence
The layer output is affine in the probe scalar β\betaProved in Appendix C. β\beta is inserted after the expert transform and before aggregation, so the layer output is an affine transformation of the probe scalar β\beta.
Squared reconstruction loss ⇒ the remainder of third and higher order is zeroProved in Appendix C. L(β)=(β−1)⊤G(β−1)L(\beta)=(\beta-\mathbf{1})^\top G(\beta-\mathbf{1}) is a strict quadratic form, H=2GH=2G, and derivatives of order k≥3k\ge 3 are identically zero.
Routing is fixed during the probeHolds for the local layer: β\beta only scales expert outputs and does not change that layer's router input. It does not hold under whole-network pruning: averaged over the 48 layers at 30% pruning, the top-8 expert overlap is 87.76% and the top-1 agreement is 84.97% (see [Q2-1]).
The Hessian diagonal sufficiently supports the pruning decision (off-diagonal HefH_{ef} is negligible)Largely satisfied. [Q2-2] provides the full Hessian distribution over all layers of the model and the interaction share under joint removal; experiments show that the expert-interaction share is relatively small: for k=8, 0.39%–1.04% in the middle layers and 0.944% on average over the 48 layers. On the final deployed channel configuration, the interaction share is 1.68% at p=0.3p=0.3 and 3.52% at p=0.5p=0.5 on average.
Activation magnitude reflects channel importance within an expertNot proven in our paper; adopted from common practice in LLM and MoE pruning. Wanda [1] scores each weight by its magnitude times the norm of its input activation; SEAP [2] prunes neurons by their task-specific activation patterns; He et al. [3] apply Wanda-based pruning inside MoE experts; PuzzleMoE [4] measures saliency as activation times weight.
The calibration data represent the test distributionNot guaranteed; studied empirically. We vary the calibration mixture over ten settings (see [Q2-3]). Per-task accuracy depends on the calibration source, and mixed calibration reduces this effect. How to choose calibration data for an unknown test distribution is left to future work.

Table R1. Assumptions made in our paper, as well as whether they are verified or consistent with real practice.

The first two assumptions hold exactly by derivation, and the activation-magnitude assumption follows common practice in prior studies.

The subsections below ([Q2-1] to [Q2-3]) aim to quantify how far the remaining assumptions deviate in practice and how MAES behaves when they are violated.

AuthorsOpenReview ↗

Response to Reviewer 5BBV (3/5)

[Q2-1] Is routing fixed during layer-wise calibration?

Routing is not fixed across layers, i.e., pruning earlier layers shifts the routing of later layers. However, as demonstrated below, such a drift, in fact, has little impact on the channels MAES selects or on downstream accuracy.

To investigate this, we conduct an ablation study of sequential calibration, which prunes the preceding layers before collecting the channel scores of the next layer, so that the scores account for the routing drift caused by earlier pruning. In Tables R2, R3, and R4 below, we compare the two calibration strategies. We report two metrics: (1) top-8 overlap, defined as ∣Sℓbase∩Sℓpruned∣/8|S_\ell^{\rm base}\cap S_\ell^{\rm pruned}|/8; (2) top-1 agreement, i.e., whether the highest-weight expert is the same. The experiments are conducted on Qwen3-VL-30B-A3B-Instruct using 128 GQA samples.

Results

In Table R2, we compute the fraction of channels that are selected differently under the two strategies. The results in Table R2 show that sequential calibration changes only a small share of the selected channels (3.92% at 30% and 5.85% at 50% on average over the 48 layers).

LayerChannel Diff at 30% (%)Channel Diff at 50% (%)
L00.000.00
L52.574.13
L102.774.46
L153.395.34
L205.587.31
L253.955.17
L306.219.13
L355.687.03
L404.316.21
L471.943.27
48-layer mean3.925.85

Table R2: Channel selection difference (%) between sequential calibration (pruned prefix) and the default calibration (unpruned prefix).

In Table R3, we report the top-8 overlap and top-1 agreement with the unpruned model under the two strategies. The results show that the two strategies reach almost the same routing agreement: the 48-layer mean top-8 overlap is 87.76% vs. 87.77% at 30% and 83.85% vs. 83.82% at 50%, and the top-1 agreement is also almost the same.

LayerTop-8 overlap at 30% (%)Top-1 agreement at 30% (%)Top-8 overlap at 50% (%)Top-1 agreement at 50% (%)
L0100.00 / 100.00100.00 / 100.00100.00 / 100.00100.00 / 100.00
L594.84 / 94.7994.00 / 94.1393.07 / 93.0792.18 / 91.90
L1092.23 / 92.1091.22 / 91.3189.02 / 88.9887.29 / 86.63
L1589.89 / 89.8888.18 / 88.3186.22 / 86.2883.71 / 83.82
L2089.37 / 89.4287.54 / 87.6185.44 / 85.3082.55 / 82.41
L2588.03 / 88.0385.71 / 85.8883.69 / 83.7579.78 / 80.02
L3087.11 / 87.0984.79 / 85.0282.86 / 82.7579.71 / 80.22
L3585.41 / 85.4483.78 / 83.8580.73 / 80.7078.17 / 77.95
L4081.35 / 81.4074.18 / 74.3676.11 / 76.0666.91 / 66.68
L4775.53 / 75.6164.22 / 64.6270.24 / 70.2356.68 / 56.65
48-layer mean87.76 / 87.7784.97 / 84.9683.85 / 83.8280.02 / 79.93

Table R3: Routing agreement with the unpruned model under the two calibration strategies. Each cell reports unpruned prefix / pruned prefix.

In Table R4, we compare them on several downstream tasks. At 30% pruning, the two strategies are almost identical (the accuracy differs by at most 0.26% on GQA, TextVQA, ChartQA, MMStar and RealWorldQA). At 50% pruning, the default calibration is generally better: it is higher on 5 of the 6 tasks, by 0.05%–0.84%, and only MME is slightly higher with sequential calibration.

Pruning ratioCalibration strategyGQATextVQAChartQAMMStarRealWorldQAMMEAvg.
30%default63.2582.9486.8470.1171.502306.6476.17
sequential63.3283.0486.7670.3771.502308.6076.24
Δ+0.06+0.10−0.08+0.260.00+1.96+0.07
50%default62.9181.9883.0467.4772.162176.4474.22
sequential62.4181.9382.2066.9471.902191.0073.94
Δ−0.50−0.05−0.84−0.53−0.26+14.56−0.28

Table R4: Downstream accuracy of the default (unpruned-prefix) and sequential (pruned-prefix) calibration. Avg. is the mean across the six tasks, with MME divided by 28.

In summary, the above results demonstrate that replacing the unpruned prefix with the pruned prefix during calibration changes the channel selection only slightly (Table R2), leaves the routing agreement essentially unchanged (Table R3), and at 30% pruning makes no clear difference on the downstream benchmarks, while at 50% pruning the default calibration is slightly better (Table R4). Calibrating each layer on an unpruned copy of the previous layers is therefore a close approximation in practice.

Comments on the paper

1 comment

Dear Reviewers and Area Chairs,

We thank all reviewers (5BBV, pX75, VaSr, RZoT) for their constructive comments and for the positive assessment of MAES. Your feedback has largely helped us improve the paper.

We appreciate that:

  • Reviewers agree that MAES targets a real compression problem for multimodal MoE models, where routing and activation patterns depend on modality (pX75, VaSr), and that pruning and deployment form one coherent pipeline (pX75).
  • Reviewer 5BBV values SOES as a closed-form estimate of expert importance that replaces per-expert ablation. Reviewer RZoT notes the efficiency of our importance estimation and the deployment benefits of CLER.
  • Reviewers pX75 and VaSr value the breadth of our experiments, the component ablations, and the consistent gains across model families and benchmarks.
  • Reviewers pX75 and VaSr find the memory and throughput measurements useful, since they go beyond parameter counts.

We respond to each reviewer individually and summarise the main additions here.

  1. Exactness and assumptions. (1) We limit the exactness claim to single-expert removal (zeroed, no renormalization) with routing fixed. (2) We show that the off-diagonal share on the full 128×128 Hessians of all 48 layers is small: 1.68% / 3.52% at 30% / 50% pruning, and it comes mostly from the first and last layers (5BBV, pX75, RZoT). (3) A new assumption table separates what we prove from what we check empirically (5BBV).
  2. Ablations for components. (1) A new ablation compares EMA allocation with equal-quota, modality-normalization and equal-weight rules, and EMA is best on average. (2) We clarify that width rounding and expert placement in CLER are separate by construction (VaSr). (3) ACP is now included in the Pareto curve in Figure 1(a), where MAES is ahead at high compression (VaSr, RZoT). We also show that ACP's second-order variant carries a per-expert background error, and that swapping SOES for ACP inside MAES lowers accuracy on all 8 tasks at both pruning ratios. (4) Sequential calibration, which solves routing drift caused by pruning earlier layers, changes 3.9% / 5.9% of the selected channels and leaves routing agreement and accuracy nearly unchanged (5BBV).
  3. Robustness under calibration data mixtures and uncertainty. (1) The average accuracy on various benchmarks varies by only 0.80% over ten calibration data mixtures with different data sources and combination ratios (5BBV, pX75). (2) We also report standard deviations over calibration seeds on our MAES (pX75).
  4. Controlled deployment for fairness. (1) We report retained parameters, FLOPs and non-KV peak memory for every baseline. (2) We also compare efficiency under the same accuracy target, MAES lowers TTFT by 3.2%–14.6%, TPOT by 4.7%–7.6% and peak memory by 22.6%–36.1% (pX75). (3) For deployment, we add three expert-parallel setups, one of them cross-node. MAES cuts non-KV peak memory by 34%–38%, while latency and throughput stay on par with the unpruned model (pX75, VaSr).
  5. Revised paper. The new analyses appear in Sec. 4.2, Sec. 5.2–5.4, Appendix C.1, C.3, E.1–E.2 and F.1–F.7 (Tables 8, 10, 12, 18–26), and Figure 1(a) is updated. All changes are marked in red.

We hope these responses address your concerns, and we are glad to discuss anything that remains open.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.