acceptodds
Under review as a conference paper at ICLR 2027

MoRA: MoE Pruning via Router Bias Learning and Expert Approximation

Abstract

Mixture-of-Experts (MoE) models enable parameter scaling with limited per-token computation by activating only a small subset of experts for each token, but deploying them still requires loading the complete expert pool into memory. Structured expert pruning can effectively reduce the memory usage by removing experts. However, existing pruning methods either use expert ranking criteria that are not well aligned with model performance or rely on effective expert subset searching that is computationally expensive. Moreover, these methods typically overlook the routing-behavior redundancy among the retained experts. In this paper, we propose MoE Pruning via Router Bias Learning and Expert Approximation (MoRA), a framework for structured MoE expert pruning. We introduce a learnable router bias for each expert and optimize these biases by minimizing the language-modeling loss and a routing-diversity regularizer. The learned router biases sharpen the routing probability distributions to identify experts critical to model performance while encouraging the selection of experts with diverse routing preferences. In addition, we introduce an expert approximation mechanism as a post-pruning enhancement. It leverages the remaining experts to approximate the outputs of pruned experts by affine transformation, further improving the performance of the pruned model. We evaluate MoRA on Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B, removing 25% and 50% of the routed experts in each MoE layer. Extensive experiments on nine zero-shot benchmarks show that MoRA outperforms state-of-the-art pruning algorithms. Our code will be released.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Mixture-of-experts, MoE pruning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RXC4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoRA, a post-training expert-pruning framework for sparse MoE language models. It freezes the pretrained model, learns one scalar bias per router logit on calibration data using language-modeling loss and a diversity regularizer, and ranks experts by the learned biases. Removed experts can then be approximated by retained proxy experts with channel-wise affine transformations. Experiments on Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B report improved zero-shot accuracy over several expert pruning and merging baselines at 25% and 50% expert removal.

Critical strengths
  1. The core idea is simple and looks useful in practice. Learning only router-logit biases keeps calibration lightweight, and separating expert selection from proxy approximation makes the method easy to implement. The gains at 25% and 50% removal across three fairly different fine-grained MoE families also suggest that this is not a one-checkpoint result.

  2. The paper does more than report downstream averages. Table 5 separates selection, diversity regularization, and approximation, and the deletion-sensitivity analysis gives some insight into why the method works. I found these analyses more informative than a standard pruning benchmark.

Critical weaknesses
  1. I am not yet convinced that the inference procedure or the diversity mechanism is fully pinned down. It is unclear how the original routing weights, removed expert identities, target-specific affine maps, and multiple mappings to the same proxy interact at inference time. More importantly, router-row cosine is only a proxy for actual co-routing, and Table 5 does not show that the gain from the regularizer really comes from preserving routing diversity. Equation (6) also seems to allow all biases to shift downward together, reducing the regularizer without changing softmax routing. This makes both the interpretation and reproduction of the method harder than they should be.

  2. The comparison with recent MoE compression work is still incomplete. DiEP is relevant to optimization-based expert selection, and ConMoE is close to the idea of keeping router identities while mapping them to a smaller expert pool. Tables 1–3 do include MoNE (2026), but many of the main baselines are still older expert-level deletion, replacement, or merging methods. The experiments also miss the recent expert-channel-level structured pruning methods HEAPr (ICLR 2026) and CAMERA-P (AAAI 2026). I would like to see whether MoRA still has an advantage in both effectiveness and efficiency under comparable MoE settings.

Questions for authors
  1. What exactly happens in the forward pass after pruning? Does the router still score every original expert identity, and how are the routing weights and target-specific affine maps applied when a removed identity is mapped to a proxy? If several selected identities map to the same proxy, is that proxy computed once and reused?

  2. On the existing calibration routes, how well does router-row cosine actually correlate with expert co-selection or top-k assignment overlap? Does the diversity term measurably reduce co-routing among the retained experts?

  3. For a softmax router, adding the same constant to all learned biases leaves the routing probabilities and ranking unchanged, while shifting every bias downward reduces the diversity penalty. Are the biases centered, normalized, or otherwise constrained? If not, what do their layer-wise mean trajectories look like, and why does this common-shift direction not make the regularizer ineffective during calibration?

  4. What are the main method-level differences from DiEP and ConMoE? Also, how does MoRA compare in effectiveness and efficiency with HEAPr and CAMERA-P, both expert-channel-level structured pruning methods, under comparable MoE settings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer RXC4

Thank you for your careful review and constructive feedback. Your questions helped us identify where the inference procedure and the role of the routing diversity regularizer needed clearer explanations. Below, we describe how MoRA routes tokens after pruning, report measured routing overlap among retained experts, and compare MoRA with recent compression methods under matched parameter budgets.

[Weakness 1/ Q1] The clarification of the inference procedure and the diversity mechanism

MoRA uses learned router biases to select which experts to prune. At inference, the router scores all original expert identities and masks any removed identity whose proxy fails to meet the fidelity threshold. We describe each step below.

We rank the biases in each layer for expert pruning. After pruning, we remove the weights of the pruned experts and discard the learned router biases.

For each removed expert, we evaluate how well a retained expert can approximate its output using a channel-wise affine transformation. We enable expert approximation for a removed identity only if its assigned proxy passes the proxy fidelity threshold. Otherwise, we mask that identity at inference. Retained experts remain eligible and do not require an affine transformation.

For each token with hidden state hh, the pretrained router scores the original expert identities. After masking ineligible identities, it selects the top-kk eligible identities according to the native routing rule. A selected retained identity ii contributes pi(h)fi(h)p_i(h)f_i(h). A selected removed identity ii instead contributes pi(h)[ai⊙fj(i)(h)+bi]p_i(h)[a_i\odot f_{j(i)}(h)+b_i], where j(i)j(i) is its retained proxy. The routing weight pi(h)p_i(h) belongs to the selected identity, not the proxy, and is applied after the affine transformation.

When multiple selected identities for the same token use the same retained expert, we compute that expert’s output once. We then reuse it for each route, applying the corresponding affine transformation only for removed identities. Each route is weighted separately before the contributions are summed.

[Weakness 1/ Q2] The correlation between diversity loss and routing overlap

The correlation between router weight similarity and top-k assignment overlap

We quantify the top-k assignment overlap using the overlap coefficient for each pair of experts. The overlap coefficient measures the fraction of the selected token set shared by both experts. Let AiA_i be the set of tokens for which the native top-kk router selects expert ii in a given layer. The overlap coefficient is defined as Oij=∣Ai∩Aj∣/min⁡(∣Ai∣,∣Aj∣)O_{ij}=\lvert A_i\cap A_j\rvert/\min(\lvert A_i\rvert,\lvert A_j\rvert).

We use Spearman’s rank correlation coefficient ρ\rho to measure the relationship between the cosine similarity of pretrained router weight rows and the observed routing overlap. The coefficient ranges from −1-1 to 11; a positive value indicates that expert pairs with more similar router weight rows tend to have greater routing overlap. For each model, we use the step-1,000 checkpoints for LM-only and LM+Div and evaluate both at 25% and 50% pruning ratios. Within each model, we use the same calibration tokens across all settings. Table 1 reports the mean across MoE layers, with ρ\rho computed across expert pairs within each layer.

Table 1. Spearman correlation between router-row cosine similarity and overlap coefficient.

ModelPruneMean Spearman ρ\rho (LM-only)Mean Spearman ρ\rho (LM+Div)
Qwen325%0.3680.469
Qwen350%0.4160.520
DeepSeek-V2-Lite25%0.6270.670
DeepSeek-V2-Lite50%0.5340.569
Moonlight25%0.2440.337
Moonlight50%0.1720.312

The mean correlations are positive across all three models and both pruning ratios, suggesting that router-row cosine similarity is an informative proxy for routing overlap.

The reduction of co-routing among retained experts by the diversity term

We compare co-routing using the overlap coefficient with and without the diversity loss. For each model, we use the step-1,000 checkpoints to compute the mean overlap coefficient within each layer and then across layers at 25% and 50% pruning ratios. Table 2 reports the mean overlap coefficients and their relative changes.

Table 2. Mean overlap coefficient with and without diversity loss.

ModelPruneMean OijO_{ij} (LM-only)Mean OijO_{ij} (LM+Div)Relative change
Qwen325%0.1150.106−7.83%
Qwen350%0.1700.152−10.59%
DeepSeek-V2-Lite25%0.1170.111-5.13%
DeepSeek-V2-Lite50%0.1860.177-4.84%
Moonlight25%0.1320.125−5.30%
Moonlight50%0.2010.193−3.98%

The mean overlap coefficient decreases when adding diversity loss, which indicates that adding the diversity loss can reduce co-routing.

AuthorsOpenReview ↗

Response to Reviewer RXC4

[Weakness 1/ Q3] The clarification of initialization settings and common shift in router biases

We initialize the router biases to zero and optimize them with AdamW. Although theoretically, a common downward shift of all biases within a layer can reduce the regularization term, we did not observe all biases within a layer decreasing by the same amount, even without centering, normalization, or other constraints.

We compute the mean bias across routed experts in each layer during training process and then average these layer means. Table 3 shows how the cross-layer mean changes across checkpoints.

Table 3. Cross-layer mean router bias during training.

ModelInitializationStep 100Step 500Step 1,000
Qwen30−0.021−0.102−0.201
DeepSeek-V2-Lite0−0.015−0.073−0.141
Moonlight0−0.032−0.155−0.303

We observe that the mean bias gradually decreases during training process, but the individual biases do not all decrease together. Table 4 reports the number of biases that increased or decreased from their zero initialization after 1,000 steps.

Table 4. Number of router biases that increased or decreased after training.

ModelNumber of total biasesNumber of increased biasesNumber of decreased biases
Qwen36,1441,1235,021
DeepSeek-V2-Lite1,6645501,114
Moonlight1,6641771,487

The regularization term in the loss function produces expert-specific gradients rather than the same gradient for every bias. At initialization, all biases are set to 0, so the regularizer skips all pairs until the biases are updated by LM gradients. Once the biases differ, the highest-bias expert receives no direct diversity gradient, while lower-bias experts receive similarity-weighted gradients from the diversity loss. The LM gradient also differs across experts and can favor either increasing or decreasing a bias. Thus, the learned biases do not move uniformly downward during practical optimization.

We also add a bias centering ablation. We subtract the mean bias within each layer and use the centered biases only in the diversity loss. Table 5 reports the average performance (%) over nine benchmarks.

Table 5. Performance of pruned models with and without centering router biases.

ModelPruneUncentered biasesCentered biases
Qwen325%72.1769.57
Qwen350%65.3052.58
DeepSeek-V2-Lite25%63.2561.62
DeepSeek-V2-Lite50%54.0251.59
Moonlight25%67.1963.45
Moonlight50%54.8251.35

The centered variant scored lower than the uncentered reference on all three models at both pruning ratios. We speculate that centering modifies the gradients of diversity loss, which can shift the balance between LM loss and diversity loss for experts near the pruning borderline. Thus, even when the LM forward pass is unchanged, the relative bias ranking and retained expert set is still affected. This comparison shows that not adding centering is a more feasible setting in our implementation.

AuthorsOpenReview ↗

Response to Reviewer RXC4

[Weakness 2/ Q4] The comparison with recent MoE compression work

We compare MoRA with three open-source methods: DiEP, HEAPr, and CAMERA-P. We did not include ConMoE in the empirical comparison because we could not find a publicly available official implementation. For channel-wise pruning (HEAPr and CAMERA-P), we match the fraction of backbone parameters actually retained by MoRA rather than the fraction of experts removed. We use the same nine zero-shot benchmarks. Table 6 reports the average performance (%) at 25% and 50% pruning ratios.

Table 6. Comparison with recent MoE compression methods at 25% and 50% pruning ratios.

ModelPruneMoRADiEPHEAPrCAMERA-P
Qwen325%72.5064.9470.4472.05
Qwen350%65.8356.3558.7965.44
DeepSeek-V2-Lite25%63.6858.1362.7460.51
DeepSeek-V2-Lite50%54.4049.1353.4851.50
Moonlight25%67.4864.4666.8165.60
Moonlight50%55.3650.4854.7244.32

MoRA achieves the highest nine-task average among the compared methods on all three models at both pruning ratios.

The difference among MoRA, DiEP and ConMoE

Distinction to DiEP

DiEP alternates between updating expert importance scores and layer scales under an objective that combines cross-entropy with a model-output reconstruction penalty. It applies softmax to token-independent expert scores to mix expert outputs, rather than adding them to router logits. It ranks experts globally by the score–scale product. MoRA fixes the per-layer budgets and learns expert-specific router biases by jointly optimizing the LM loss and the diversity loss.

Distinction to ConMoE

ConMoE uses Anchor Coverage to construct a retained expert pool, then assigns removed identities to retained experts and uses the proxies’ unmodified outputs. In contrast, MoRA learns router biases for expert selection using the LM loss and the diversity loss, and fits a target-specific channel-wise affine transformation for expert approximation. Two removed identities can therefore receive different approximations even when they share the same proxy output.

comparison with HEAPr and CAMERA-P in effectiveness and efficiency

HEAPr and CAMERA-P prune intermediate FFN channels within experts, whereas MoRA removes entire experts. We therefore match the fraction of backbone parameters retained after pruning, rather than the fraction of experts removed. We benchmarked compact models on one RTX 6000D with a batch size of 1, 512 input tokens, 32 generated tokens, and KV caching enabled. Table 7 reports time to first token (TTFT, ms), decode throughput (DT, tokens/s), and peak PyTorch allocated memory (PM, GiB).

Table 7. Hardware inference comparison with HEAPr and CAMERA-P at matched retained backbone parameter budgets.

TTFT (ms)↓│DT (tokens/s)↑│PM (GiB)
ModelPruneMoRAHEAPrCAMERA-P│MoRAHEAPrCAMERA-P│MoRAHEAPrCAMERA-P
Qwen325%597818634│10.953.934.57│43.5843.6043.62
Qwen350%5099821,011│10.673.623.52│30.0830.0730.08
DeepSeek-V2-Lite25%233270292│21.1512.5612.67│23.0423.3923.67
DeepSeek-V2-Lite50%171289305│21.0712.5912.23│16.3416.8616.96
Moonlight25%218242253│20.1812.5612.09│23.8424.1624.21
Moonlight50%168253254│19.9112.6212.38│17.1317.6517.93

In these implementations, MoRA has lower TTFT than both channel-pruning baselines in all six settings and achieves 1.58–2.95× the decode throughput of the faster baseline in each setting. Peak allocated memory is similar across the three pruning methods.

Reviewer_c4zJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MoRA aims to preserve the predictive performance of MoE models by pruning their constituent experts. The method learns a router bias for each expert using both a LM loss and a routing-diversity regularizer, which determines which experts should be retained. During the expert approximation phase, the model leverages the outputs of the retained experts to compensate for the functions lost due to the removal of other experts. Experiments cover three different models, two pruning ratios, and nine zero-shot tasks. The results demonstrate that this approach outperforms the evaluated baselines in terms of average scores. Ablation studies confirm the effectiveness of the combination of loss functions and indicate that the performance gains are primarily attributable to the expert selection mechanism.

Critical strengths
  1. MoRA method focuses on how expert pruning can preserve superior performance. Its approach to addressing this issue is intuitive and the integration of a bias term and affine transformations represents a sound perspective. MoRA method achieves significant improvements in model performance by jointly optimizing for both LM loss and diversity loss.

  2. MoRA generally improves performance on three models and two pruning ratios. On Qwen3-30B-A3B at 50% pruning, it scores 65.83 versus 59.40 for the baseline. These results show that the MoRA procedure can preserve better predictive performance after removing experts. The experiments also show that most of the gain remains without the affine module and that removing either objective term reduces performance.

Critical weaknesses
  1. The Table 5 suggests that penalizing diversity loss is beneficial. But it remains unclear whether these gains stem from the preservation of experts that exhibit distinct routing behaviors. Equation (5) compares router weights—a departure from the standard practice of using embeddings—which may not accurately reflect routing decisions for actual tokens, nor does it necessarily capture the diversity in the experts' outputs. Furthermore, regarding the softmax routing in Equation (6), there is an issue that subtracting a constant from the biases of all experts within the same layer leaves both the routing decisions and the expert rankings unchanged, the penalty term yet can be driven toward zero. The paper does not specify whether any constraints are imposed on these biases nor does it demonstrate whether the actual routing overlap between experts is effectively reduced.

  2. Expert selection and Expert Approximation are two separate stages and most of the gain comes from selection. On Qwen3 at 50% pruning the joint objective improves the score from 58.08 with LM loss alone to 65.30, while Expert Approximation raises it to 65.83. A comparison would be between the proposed joint objective and other optimization-driven expert selection methods. Now the baselines involve merging and replacement-based compensation. So they don’t establish the advantage of the proposed selection objective over alternatives, and whether another selector would benefit as much, or produce a better final combination. Other selection methods may benefit from the same affine module. The cost is also unclear while learning only a few biases does not make 1,000 optimization steps cheap.

Questions for authors
  1. Do the router weight similarities effectively correspond to routing overlaps on the calibration tokens? Does the diversity loss reduce the overlap among the retained experts? For softmax routing, how does the actual implementation handle the joint shift of the bias terms in Equation (6)? Please specify the initialization and optimization settings and indicate whether any centering method or other constraints are applied.

  2. With the set of retained experts fixed and other routing configurations held constant, how do the results differ when the learned biases for the retained experts are kept versus when they are discarded? Could you also include a comparison with another pruning method to evaluate performance both with and without the "Expert Approximation" technique using comparable experimental settings?

  3. How many calibration tokens were used for the 1,000 optimization iterations, and what were the total runtime and peak GPU memory consumption? Furthermore, what is the computational cost of the affine fitting step alone?

5 comments
AuthorsOpenReview ↗

Response to Reviewer c4zJ

Thank you for asking us to distinguish the effects of expert selection and expert approximation and to provide direct evidence on routing overlap. We address each of your concerns and questions below.

[Weakness 1/ Q1] Clarification of routing overlap, settings, and the joint shift

The correlation between router weight similarity and routing overlap

We quantify routing overlap using the overlap coefficient for each pair of experts. The overlap coefficient measures the fraction of the selected token set shared by both experts. Let AiA_i be the set of tokens for which the native top-kk router selects expert ii in a given layer. The overlap coefficient is defined as Oij=∣Ai∩Aj∣/min⁡(∣Ai∣,∣Aj∣)O_{ij}=\lvert A_i\cap A_j\rvert/\min(\lvert A_i\rvert,\lvert A_j\rvert).

We use Spearman’s rank correlation coefficient ρ\rho to measure the relationship between the cosine similarity of pretrained router weight rows and the observed routing overlap. The coefficient ranges from −1-1 to 11; a positive value indicates that expert pairs with more similar router weight rows tend to have greater routing overlap. For each model, we use the step-1,000 checkpoints for LM-only and LM+Div and evaluate both at 25% and 50% pruning ratios. Within each model, we use the same calibration tokens across all settings. Table 1 reports the mean Spearman correlation across MoE layers, with ρ\rho computed across expert pairs within each layer.

Table 1. Spearman correlation between router-row cosine similarity and overlap coefficient.

ModelPruneMean Spearman ρ\rho (LM-only)Mean Spearman ρ\rho (LM+Div)
Qwen325%0.3680.469
Qwen350%0.4160.520
DeepSeek-V2-Lite25%0.6270.670
DeepSeek-V2-Lite50%0.5340.569
Moonlight25%0.2440.337
Moonlight50%0.1720.312

The mean correlations are positive across all three models and both pruning ratios, suggesting that router-row cosine similarity is an informative proxy for routing overlap.

The reduction of routing overlap by diversity loss

We compare routing overlap using the overlap coefficient with and without the diversity loss. For each model, we use the step-1,000 checkpoints to compute mean overlap coefficient within each layer and then across layers at 25% and 50% pruning ratios. Table 2 reports the mean overlap coefficients and their relative changes.

Table 2. Mean overlap coefficient with and without diversity loss.

ModelPruneMean OijO_{ij} (LM-only)Mean OijO_{ij} (LM+Div)Relative change
Qwen325%0.1150.106−7.83%
Qwen350%0.1700.152−10.59%
DeepSeek-V2-Lite25%0.1170.111-5.13%
DeepSeek-V2-Lite50%0.1860.177-4.84%
Moonlight25%0.1320.125−5.30%
Moonlight50%0.2010.193−3.98%

The mean overlap coefficient decreases when adding diversity loss, which indicates that adding diversity loss can reduce routing overlap.

AuthorsOpenReview ↗

Response to Reviewer c4zJ

The initialization, optimization settings, and joint shift in router biases

We initialize the router biases to zero and optimize them with AdamW. Although theoretically, a joint downward shift of all biases within a layer can reduce the regularization term, we did not observe all biases within a layer decreasing by the same amount, even without centering, normalization, or other constraints.

We compute the mean bias across routed experts in each layer during the training process and then average these layer means. Table 3 shows how the cross-layer mean bias changes during training.

Table 3. Cross-layer mean router bias during training.

ModelInitializationStep 100Step 500Step 1,000
Qwen30−0.021−0.102−0.201
DeepSeek-V2-Lite0−0.015−0.073−0.141
Moonlight0−0.032−0.155−0.303

We observe that the mean bias gradually decreases during the training process, but the individual biases do not all decrease together. Table 4 reports the number of biases that increased or decreased from their zero initialization after 1,000 steps.

Table 4. Number of router biases that increased or decreased after training.

ModelNumber of total biasesNumber of increased biasesNumber of decreased biases
Qwen36,1441,1235,021
DeepSeek-V2-Lite1,6645501,114
Moonlight1,6641771,487

The regularization term in the loss function produces expert-specific gradients rather than the same gradient for every bias. At initialization, all biases are set to 0, so the regularizer skips all pairs until the biases are updated by LM gradients. Once the biases differ, the highest-bias expert receives no direct diversity gradient, while lower-bias experts receive similarity-weighted gradients from the diversity loss. The LM gradient also differs across experts and can favor either increasing or decreasing a bias. Thus, the learned biases do not move uniformly downward during practical optimization.

We also add a bias centering ablation. We subtract the mean bias within each layer and use the centered biases only in the diversity loss. Table 5 reports the average performance (%) over nine benchmarks.

Table 5. Performance of pruned models with and without centering router biases.

ModelPruneUncentered biasesCentered biases
Qwen325%72.1769.57
Qwen350%65.3052.58
DeepSeek-V2-Lite25%63.2561.62
DeepSeek-V2-Lite50%54.0251.59
Moonlight25%67.1963.45
Moonlight50%54.8251.35

The centered variant scored lower than the uncentered reference on all three models at both pruning ratios. We speculate that centering modifies the gradients of diversity loss, which can shift the balance between LM loss and diversity loss for experts near the pruning borderline. Thus, even when the LM forward pass is unchanged, the relative bias ranking and retained expert set is still affected. This comparison shows that not adding centering is a more feasible setting in our implementation.

[Q2] Analysis on whether to keep the learned biases at inference

The learned router biases are used to rank experts and discarded after pruning. Once the retained expert set is fixed, adding the biases back at inference changes only how tokens are routed among the retained experts. The learned biases adjust the pretrained router logits before routing scores are computed.

We fix the retained expert set to analyze the impact of these routing changes on downstream performance. We compare inference with and without the learned biases. Expert approximation is disabled in both settings. Table 6 reports the average performance (%) over nine benchmarks.

Table 6. Performance of pruned models with and without learned router biases at inference.

ModelPruneDiscard learned router biasesKeep learned router biases
Qwen325%72.1772.11 (-0.06)
Qwen350%65.3065.49 (+0.19)
DeepSeek-V2-Lite25%63.2563.76 (+0.51)
DeepSeek-V2-Lite50%54.0254.44 (+0.44)
Moonlight25%67.1967.08 (-0.11)
Moonlight50%54.8254.51 (-0.31)

Keeping the learned router biases does not consistently improve performance. To limit changes to the pretrained router at inference, we use the pretrained router without applying the additional learned bias term.

AuthorsOpenReview ↗

Response to Reviewer c4zJ

[Weakness 2/ Q2] The comparison with other pruning methods with and without the "Expert Approximation"

We applied affine transformations to the outputs of retained experts in MC-SMoE and HC-SMoE without changing their routing. Table 7 reports nine-task averages (%). EA denotes expert approximation with affine transformation. Values in parentheses are gains over the corresponding baseline.

Table 7. Comparison with MC-SMoE and HC-SMoE with and without expert approximation (nine-task average, %).

ModelPruneMC-SMoEMC-SMoE + EAHC-SMoEHC-SMoE + EAMoRA w/o EAMoRA (ours)
Qwen325%71.3672.09(+0.73)\textcolor{green}{72.09 (+0.73)}64.9865.27(+0.29)\textcolor{green}{65.27 (+0.29)}72.17‾\underline{72.17}72.50\textcolor{green}{72.50}
Qwen350%47.6148.05(+0.44)\textcolor{green}{48.05 (+0.44)}46.9047.26(+0.36)\textcolor{green}{47.26 (+0.36)}65.30‾\underline{65.30}65.83\textcolor{green}{65.83}
DeepSeek-V2-Lite25%57.6657.95(+0.29)\textcolor{green}{57.95 (+0.29)}62.6662.93(+0.27)\textcolor{green}{62.93 (+0.27)}63.25‾\underline{63.25}63.68\textcolor{green}{63.68}
DeepSeek-V2-Lite50%45.8346.66(+0.83)\textcolor{green}{46.66 (+0.83)}53.8453.96(+0.12)\textcolor{green}{53.96 (+0.12)}54.02‾\underline{54.02}54.40\textcolor{green}{54.40}
Moonlight25%63.4663.83(+0.37)\textcolor{green}{63.83 (+0.37)}56.8757.12(+0.25)\textcolor{green}{57.12 (+0.25)}67.19‾\underline{67.19}67.48\textcolor{green}{67.48}
Moonlight50%44.5545.01(+0.46)\textcolor{green}{45.01 (+0.46)}44.5444.86(+0.32)\textcolor{green}{44.86 (+0.32)}54.82‾\underline{54.82}55.36\textcolor{green}{55.36}

Even without expert approximation, MoRA outperforms both MC-SMoE and HC-SMoE on all three models at both pruning ratios, supporting the effectiveness of expert selection based on learned router biases. The consistent improvement of the EA variants further shows its benefits extend beyond MoRA.

[Q3] The calibration cost, peak GPU memory, and cost of fitting the affine transformations

Using C4 as the calibration dataset, we measured wall-clock time, peak GPU memory usage, and the time required to fit the channel-wise affine transformations on a single RTX 6000D. For router bias optimization, we report the estimated wall-clock time for 1,000 iterations and the observed peak GPU memory usage. We use 512 C4 blocks of 2,048 tokens each; with one block per iteration, 1,000 iterations process 2.048 million token positions, counting repeated blocks.

For expert approximation, the fit and scoring splits each contain 64 C4 blocks of 1,024 tokens. We report the combined time to collect weighted moments on the fit split and fit the affine transformations, as well as the time for weighted ridge regression. Table 8 summarizes the time and memory measurements.

Table 8. Router bias optimization time, peak GPU memory, and affine fitting time on a single RTX 6000D.

ModelRouter bias optimization (minutes)Peak GPU memory during optimization (GiB)Moment collection + affine fitting (minutes)Ridge regression (minutes)
Qwen3-30B-A3B12.867.42.450.10
DeepSeek-V2-Lite28.144.20.740.03
Moonlight-16B-A3B19.660.00.700.03
Reviewer_Ge55OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MoRA addresses expert pruning in MoE models to reduce parameter storage while preserving predictive performance. It learns expert-specific router biases using a language-modeling objective and a routing-diversity regularizer, then retains experts according to their learned biases. After pruning, channel-wise affine transformations of retained expert outputs approximate removed experts, with proxies selected using a separate scoring split. Experiments on three MoE models, two pruning ratios, and nine zero-shot benchmarks report the highest average scores among the evaluated pruning and merging baselines in all six model–pruning configurations.

Critical strengths
  1. The approach is well motivated by the need to preserve predictive utility under a limited expert budget.
  2. The method achieves competitive results across the evaluated models and pruning settings, supported by component ablations.
Critical weaknesses
  1. The contribution of MoRA relative to existing expert pruning and compensation methods could be further clarified. Although STUN and MoNE are discussed, a more explicit comparison of their selection and compensation mechanisms with MoRA would help identify the methodological advances introduced by learned router biases and input-dependent expert approximation.

  2. The analysis could provide more insight into the method’s limitations. The component ablations and routing visualizations support its effectiveness, but offer less explanation of the conditions associated with larger performance degradation or limited recovery from expert approximation. A focused error analysis would help characterize the method’s practical boundaries.

Questions for authors
  1. Could you elaborate on the specific advances over STUN and MoNE? In particular, how does prediction-driven bias learning differ from similarity-based expert selection, and what additional capability does the input-dependent affine proxy provide over lightweight output replacement? A focused discussion of these differences and their significance would help position the contribution more precisely.

  2. Could you discuss a few failure cases or task conditions in which pruning produces larger degradation and expert approximation offers limited recovery? A representative setting from the existing evaluation, with a few examples or task-level failure patterns, would be sufficient. Explaining what the proxy can and cannot recover would make the applicability of the method easier to assess.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ge55

Thank you for your thoughtful comments. Below, we clarify how MoRA differs from STUN and MoNE and discuss its limitations and failure cases. We will incorporate these clarifications into the revision.

[Weakness 1/ Q1] Clarification of contribution and distinctions

Distinction to STUN

STUN first clusters experts by behavioral similarity using distances between pretrained router rows. For expert pruning, it keeps one expert per cluster by selecting the expert whose own parameters are closest to the cluster mean. In MoRA, we learn one router bias per expert by jointly minimizing the LM loss and a pairwise regularizer, where router behavioral similarity is just used to calculate the regularization weight instead of defining clusters or directly ranking experts. The experts that contribute to next-token prediction tend to have larger biases during training, and we use the learned biases to rank experts. Here, the regularizer is used to avoid assigning high biases to multiple experts with similar routing preferences.

Distinction to MoNE

After pruning, MoNE replaces the removed expert with its mean calibration output, which is used as the fixed output for every token assigned to that expert identity. That is, the lightweight output replacement strategy in MoNE is input-agnostic\textit{input-agnostic}. By contrast, MoRA uses an input-dependent\textit{input-dependent} approximation strategy. Specifically, MoRA uses a target-specific affine transformation (ai⊙fj(i)(h)+bia_i \odot f_{j(i)}(h)+b_i) of the retained expert's output. The approximated expert output varies with input. For each removed expert, we fit the affine transformations on target-relevant calibration tokens and select the retained proxy on a disjoint scoring split.

Difference between prediction-driven bias learning and similarity-based expert selection

Similarity indicates whether the router tends to select the two experts for similar inputs; it does not tell us which expert is more useful for predicting the next token. MoRA uses the latter signal directly. The LM gradient favors increasing a selected expert’s bias when shifting the mixture toward that expert’s output would reduce prediction loss. The routing-diversity regularizer plays a complementary role. For a pair with similar routing preferences and unequal biases, the regularizer pushes down the lower bias without penalizing the higher one through that pair. This makes it less likely that experts with similar routing behaviour are retained under the same pruning budget.

Additional capability of the input-dependent affine proxy over lightweight output replacement

A constant novice can recover a removed expert’s average output, but it cannot recover how that output changes from token to token. For two tokens h1h_1 and h2h_2 routed to the same removed expert ii, the novice returns the same vector before router weighting. The affine proxy instead predicts an output difference of ai⊙[fj(i)(h1)−fj(i)(h2)]a_i \odot [f_{j(i)}(h_1)-f_{j(i)}(h_2)]. It can therefore recover input-dependent variation when that variation is present in the retained expert’s response and can be matched by channel-wise rescaling.

[Weakness 2] More insight into the method limitations and Failure cases.

Limitation discussion

The channel-wise affine transformation in learns one scale and one bias per output channel. It can correct differences in scale and offset between a removed expert and its retained proxy. It is most effective when, across tokens, each proxy output channel already varies in a way that can predict the corresponding channel of the removed expert. However, it cannot use information from other proxy channels or recover target output variation that is absent from the corresponding proxy channel. At higher pruning ratios, fewer retained experts are available as candidates, making it harder to find a high-fidelity proxy for each removed expert.

The affine transformation can be less effective when multiple removed experts share a retained proxy. When two removed experts selected for the same token share a retained proxy, their separately fitted channel-wise affine transformations operate on the same proxy output. If the shared proxy misses information needed to approximate either removed expert, fitting separate affine transformations cannot fully recover the removed expert’s output. The approximation errors can reinforce each other when the router combines the two output contributions. This is a possible failure mode at higher pruning ratios. Meanwhile, the proxy quality also matters: on Qwen3 at 25% pruning, lowering the fidelity threshold from 0.4 to 0 reduced the nine-task mean from 72.50% to 72.08% (Appendix C).

AuthorsOpenReview ↗

Response to Reviewer Ge55

[Q2] Failure cases.

At 50% pruning, MoRA does not fully recover performance on MMLU. MMLU spans 57 subjects and tests not only whether a model recalls specialized knowledge, but also whether it applies the right rule under the conditions given in the question. At 50% pruning, MMLU accuracy drops from 77.81% to 53.14% on Qwen3 and from 67.31% to 28.66% on Moonlight. The degradation is task-dependent: at the same pruning ratio, Qwen3’s WinoGrande accuracy drops by only 1.58 percentage points.

For example, in a college physics question about a reversible process, unpruned Qwen3 correctly predicts that the total entropy of the system and its surroundings remains unchanged, whereas the 50%-pruned model with expert approximation incorrectly predicts that it increases. To investigate this error, we re-enabled previously masked expert routes and restored their original expert outputs at selected token positions. Restoring these routes only at thermodynamic terms and key answer phrases recovered the correct answer in two of four answer-choice orderings. Extending the intervention to the question stem, answer choices, and answer prompt recovered the correct answer in all four. In this example, expert approximation did not compensate for the loss of expert contributions across multiple parts of the input.

Thanks for the response from authors. I will keep my score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.