- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MoRA, a post-training expert-pruning framework for sparse MoE language models. It freezes the pretrained model, learns one scalar bias per router logit on calibration data using language-modeling loss and a diversity regularizer, and ranks experts by the learned biases. Removed experts can then be approximated by retained proxy experts with channel-wise affine transformations. Experiments on Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B report improved zero-shot accuracy over several expert pruning and merging baselines at 25% and 50% expert removal.
Critical strengths
-
The core idea is simple and looks useful in practice. Learning only router-logit biases keeps calibration lightweight, and separating expert selection from proxy approximation makes the method easy to implement. The gains at 25% and 50% removal across three fairly different fine-grained MoE families also suggest that this is not a one-checkpoint result.
-
The paper does more than report downstream averages. Table 5 separates selection, diversity regularization, and approximation, and the deletion-sensitivity analysis gives some insight into why the method works. I found these analyses more informative than a standard pruning benchmark.
Critical weaknesses
-
I am not yet convinced that the inference procedure or the diversity mechanism is fully pinned down. It is unclear how the original routing weights, removed expert identities, target-specific affine maps, and multiple mappings to the same proxy interact at inference time. More importantly, router-row cosine is only a proxy for actual co-routing, and Table 5 does not show that the gain from the regularizer really comes from preserving routing diversity. Equation (6) also seems to allow all biases to shift downward together, reducing the regularizer without changing softmax routing. This makes both the interpretation and reproduction of the method harder than they should be.
-
The comparison with recent MoE compression work is still incomplete. DiEP is relevant to optimization-based expert selection, and ConMoE is close to the idea of keeping router identities while mapping them to a smaller expert pool. Tables 1–3 do include MoNE (2026), but many of the main baselines are still older expert-level deletion, replacement, or merging methods. The experiments also miss the recent expert-channel-level structured pruning methods HEAPr (ICLR 2026) and CAMERA-P (AAAI 2026). I would like to see whether MoRA still has an advantage in both effectiveness and efficiency under comparable MoE settings.
Questions for authors
-
What exactly happens in the forward pass after pruning? Does the router still score every original expert identity, and how are the routing weights and target-specific affine maps applied when a removed identity is mapped to a proxy? If several selected identities map to the same proxy, is that proxy computed once and reused?
-
On the existing calibration routes, how well does router-row cosine actually correlate with expert co-selection or top-k assignment overlap? Does the diversity term measurably reduce co-routing among the retained experts?
-
For a softmax router, adding the same constant to all learned biases leaves the routing probabilities and ranking unchanged, while shifting every bias downward reduces the diversity penalty. Are the biases centered, normalized, or otherwise constrained? If not, what do their layer-wise mean trajectories look like, and why does this common-shift direction not make the regularizer ineffective during calibration?
-
What are the main method-level differences from DiEP and ConMoE? Also, how does MoRA compare in effectiveness and efficiency with HEAPr and CAMERA-P, both expert-channel-level structured pruning methods, under comparable MoE settings?
5 comments
Response to Reviewer RXC4
[Weakness 1/ Q3] The clarification of initialization settings and common shift in router biases
We initialize the router biases to zero and optimize them with AdamW. Although theoretically, a common downward shift of all biases within a layer can reduce the regularization term, we did not observe all biases within a layer decreasing by the same amount, even without centering, normalization, or other constraints.
We compute the mean bias across routed experts in each layer during training process and then average these layer means. Table 3 shows how the cross-layer mean changes across checkpoints.
Table 3. Cross-layer mean router bias during training.
| Model | Initialization | Step 100 | Step 500 | Step 1,000 |
|---|---|---|---|---|
| Qwen3 | 0 | −0.021 | −0.102 | −0.201 |
| DeepSeek-V2-Lite | 0 | −0.015 | −0.073 | −0.141 |
| Moonlight | 0 | −0.032 | −0.155 | −0.303 |
We observe that the mean bias gradually decreases during training process, but the individual biases do not all decrease together. Table 4 reports the number of biases that increased or decreased from their zero initialization after 1,000 steps.
Table 4. Number of router biases that increased or decreased after training.
| Model | Number of total biases | Number of increased biases | Number of decreased biases |
|---|---|---|---|
| Qwen3 | 6,144 | 1,123 | 5,021 |
| DeepSeek-V2-Lite | 1,664 | 550 | 1,114 |
| Moonlight | 1,664 | 177 | 1,487 |
The regularization term in the loss function produces expert-specific gradients rather than the same gradient for every bias. At initialization, all biases are set to 0, so the regularizer skips all pairs until the biases are updated by LM gradients. Once the biases differ, the highest-bias expert receives no direct diversity gradient, while lower-bias experts receive similarity-weighted gradients from the diversity loss. The LM gradient also differs across experts and can favor either increasing or decreasing a bias. Thus, the learned biases do not move uniformly downward during practical optimization.
We also add a bias centering ablation. We subtract the mean bias within each layer and use the centered biases only in the diversity loss. Table 5 reports the average performance (%) over nine benchmarks.
Table 5. Performance of pruned models with and without centering router biases.
| Model | Prune | Uncentered biases | Centered biases |
|---|---|---|---|
| Qwen3 | 25% | 72.17 | 69.57 |
| Qwen3 | 50% | 65.30 | 52.58 |
| DeepSeek-V2-Lite | 25% | 63.25 | 61.62 |
| DeepSeek-V2-Lite | 50% | 54.02 | 51.59 |
| Moonlight | 25% | 67.19 | 63.45 |
| Moonlight | 50% | 54.82 | 51.35 |
The centered variant scored lower than the uncentered reference on all three models at both pruning ratios. We speculate that centering modifies the gradients of diversity loss, which can shift the balance between LM loss and diversity loss for experts near the pruning borderline. Thus, even when the LM forward pass is unchanged, the relative bias ranking and retained expert set is still affected. This comparison shows that not adding centering is a more feasible setting in our implementation.
Response to Reviewer RXC4
[Weakness 2/ Q4] The comparison with recent MoE compression work
We compare MoRA with three open-source methods: DiEP, HEAPr, and CAMERA-P. We did not include ConMoE in the empirical comparison because we could not find a publicly available official implementation. For channel-wise pruning (HEAPr and CAMERA-P), we match the fraction of backbone parameters actually retained by MoRA rather than the fraction of experts removed. We use the same nine zero-shot benchmarks. Table 6 reports the average performance (%) at 25% and 50% pruning ratios.
Table 6. Comparison with recent MoE compression methods at 25% and 50% pruning ratios.
| Model | Prune | MoRA | DiEP | HEAPr | CAMERA-P |
|---|---|---|---|---|---|
| Qwen3 | 25% | 72.50 | 64.94 | 70.44 | 72.05 |
| Qwen3 | 50% | 65.83 | 56.35 | 58.79 | 65.44 |
| DeepSeek-V2-Lite | 25% | 63.68 | 58.13 | 62.74 | 60.51 |
| DeepSeek-V2-Lite | 50% | 54.40 | 49.13 | 53.48 | 51.50 |
| Moonlight | 25% | 67.48 | 64.46 | 66.81 | 65.60 |
| Moonlight | 50% | 55.36 | 50.48 | 54.72 | 44.32 |
MoRA achieves the highest nine-task average among the compared methods on all three models at both pruning ratios.
The difference among MoRA, DiEP and ConMoE
Distinction to DiEP
DiEP alternates between updating expert importance scores and layer scales under an objective that combines cross-entropy with a model-output reconstruction penalty. It applies softmax to token-independent expert scores to mix expert outputs, rather than adding them to router logits. It ranks experts globally by the score–scale product. MoRA fixes the per-layer budgets and learns expert-specific router biases by jointly optimizing the LM loss and the diversity loss.
Distinction to ConMoE
ConMoE uses Anchor Coverage to construct a retained expert pool, then assigns removed identities to retained experts and uses the proxies’ unmodified outputs. In contrast, MoRA learns router biases for expert selection using the LM loss and the diversity loss, and fits a target-specific channel-wise affine transformation for expert approximation. Two removed identities can therefore receive different approximations even when they share the same proxy output.
comparison with HEAPr and CAMERA-P in effectiveness and efficiency
HEAPr and CAMERA-P prune intermediate FFN channels within experts, whereas MoRA removes entire experts. We therefore match the fraction of backbone parameters retained after pruning, rather than the fraction of experts removed. We benchmarked compact models on one RTX 6000D with a batch size of 1, 512 input tokens, 32 generated tokens, and KV caching enabled. Table 7 reports time to first token (TTFT, ms), decode throughput (DT, tokens/s), and peak PyTorch allocated memory (PM, GiB).
Table 7. Hardware inference comparison with HEAPr and CAMERA-P at matched retained backbone parameter budgets.
| TTFT (ms)↓ | │ | DT (tokens/s)↑ | │ | PM (GiB) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Prune | MoRA | HEAPr | CAMERA-P | │ | MoRA | HEAPr | CAMERA-P | │ | MoRA | HEAPr | CAMERA-P |
| Qwen3 | 25% | 597 | 818 | 634 | │ | 10.95 | 3.93 | 4.57 | │ | 43.58 | 43.60 | 43.62 |
| Qwen3 | 50% | 509 | 982 | 1,011 | │ | 10.67 | 3.62 | 3.52 | │ | 30.08 | 30.07 | 30.08 |
| DeepSeek-V2-Lite | 25% | 233 | 270 | 292 | │ | 21.15 | 12.56 | 12.67 | │ | 23.04 | 23.39 | 23.67 |
| DeepSeek-V2-Lite | 50% | 171 | 289 | 305 | │ | 21.07 | 12.59 | 12.23 | │ | 16.34 | 16.86 | 16.96 |
| Moonlight | 25% | 218 | 242 | 253 | │ | 20.18 | 12.56 | 12.09 | │ | 23.84 | 24.16 | 24.21 |
| Moonlight | 50% | 168 | 253 | 254 | │ | 19.91 | 12.62 | 12.38 | │ | 17.13 | 17.65 | 17.93 |
In these implementations, MoRA has lower TTFT than both channel-pruning baselines in all six settings and achieves 1.58–2.95× the decode throughput of the faster baseline in each setting. Peak allocated memory is similar across the three pruning methods.
Response to Reviewer RXC4
Thank you for your careful review and constructive feedback. Your questions helped us identify where the inference procedure and the role of the routing diversity regularizer needed clearer explanations. Below, we describe how MoRA routes tokens after pruning, report measured routing overlap among retained experts, and compare MoRA with recent compression methods under matched parameter budgets.
[Weakness 1/ Q1] The clarification of the inference procedure and the diversity mechanism
MoRA uses learned router biases to select which experts to prune. At inference, the router scores all original expert identities and masks any removed identity whose proxy fails to meet the fidelity threshold. We describe each step below.
We rank the biases in each layer for expert pruning. After pruning, we remove the weights of the pruned experts and discard the learned router biases.
For each removed expert, we evaluate how well a retained expert can approximate its output using a channel-wise affine transformation. We enable expert approximation for a removed identity only if its assigned proxy passes the proxy fidelity threshold. Otherwise, we mask that identity at inference. Retained experts remain eligible and do not require an affine transformation.
For each token with hidden state h, the pretrained router scores the original expert identities. After masking ineligible identities, it selects the top-k eligible identities according to the native routing rule. A selected retained identity i contributes pi(h)fi(h). A selected removed identity i instead contributes pi(h)[ai⊙fj(i)(h)+bi], where j(i) is its retained proxy. The routing weight pi(h) belongs to the selected identity, not the proxy, and is applied after the affine transformation.
When multiple selected identities for the same token use the same retained expert, we compute that expert’s output once. We then reuse it for each route, applying the corresponding affine transformation only for removed identities. Each route is weighted separately before the contributions are summed.
[Weakness 1/ Q2] The correlation between diversity loss and routing overlap
The correlation between router weight similarity and top-k assignment overlap
We quantify the top-k assignment overlap using the overlap coefficient for each pair of experts. The overlap coefficient measures the fraction of the selected token set shared by both experts. Let Ai be the set of tokens for which the native top-k router selects expert i in a given layer. The overlap coefficient is defined as Oij=∣Ai∩Aj∣/min(∣Ai∣,∣Aj∣).
We use Spearman’s rank correlation coefficient ρ to measure the relationship between the cosine similarity of pretrained router weight rows and the observed routing overlap. The coefficient ranges from −1 to 1; a positive value indicates that expert pairs with more similar router weight rows tend to have greater routing overlap. For each model, we use the step-1,000 checkpoints for LM-only and LM+Div and evaluate both at 25% and 50% pruning ratios. Within each model, we use the same calibration tokens across all settings. Table 1 reports the mean across MoE layers, with ρ computed across expert pairs within each layer.
Table 1. Spearman correlation between router-row cosine similarity and overlap coefficient.
The mean correlations are positive across all three models and both pruning ratios, suggesting that router-row cosine similarity is an informative proxy for routing overlap.
The reduction of co-routing among retained experts by the diversity term
We compare co-routing using the overlap coefficient with and without the diversity loss. For each model, we use the step-1,000 checkpoints to compute the mean overlap coefficient within each layer and then across layers at 25% and 50% pruning ratios. Table 2 reports the mean overlap coefficients and their relative changes.
Table 2. Mean overlap coefficient with and without diversity loss.
The mean overlap coefficient decreases when adding diversity loss, which indicates that adding the diversity loss can reduce co-routing.