- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper looks at expert pruning for MoE diffusion language models. The proposed criterion, NAPE, scores each expert by the gradient of the masked-prediction loss with respect to its gate, averaged over calibration data masked at different ratios. It is evaluated on LLaDA-MoE and SDAR at 25% and 50% pruning. The clearest win is SDAR at 50%, where it beats EAN by about 7 points on average.
Critical strengths
-
The SDAR 50% result is strong for a method that needs no retraining, and the memory savings are real since the pruned experts are actually removed.
-
The experimental setup is careful. All methods use the same calibration data and evaluation masks, and the ablations cover the obvious questions (activation-based scores, a single fixed mask ratio). It is also convenient that one setting works for both models.
Critical weaknesses
-
I am convinced that gradient scores help, less so that the multi-noise calibration does. Against the best single mask ratio, NAPE gains under a point on LLaDA and about a quarter point on SDAR. Picking the best ratio afterwards favors the baseline, I admit, but the appendix shows bigger swings than that just from changing the calibration size or the number of evaluation masks, and several main-table margins are also only a few tenths. Without repeated seeds I can't tell how much of this is noise. The retained-set recovery experiment doesn't really settle it either, since its reference is built from the same objective NAPE estimates.
-
The averages hide some uneven results. On LLaDA at 50%, NAPE is below random pruning on MMLU, and on SDAR at 50% it is behind EAN on MMLU, HellaSwag and PIQA; most of the average gain comes from BoolQ. MMLU and HellaSwag are also missing from the ablations. Separately, the gate gradient is a local estimate under the original routing, while pruning changes the routing. The removal experiment checks this, but only one expert at a time in the last layer, which says little about removing half the experts across all layers.
Questions for authors
-
Can you repeat the fixed-ratio comparison over a few calibration seeds and report the variance? Ideally the fixed ratio would be chosen on held-out data.
-
How are masks applied during SDAR calibration, over the whole sequence or within a block? Does the removal correlation still hold in earlier layers, or when several experts are removed together?
-
Why is MMLU on LLaDA at 50% below random? It would help to include MMLU and HellaSwag in the ablations.
-
A few missing details: the XSum sample count and decoding settings, and calibration time and memory compared with forward-only scoring.
5 comments
Response to Reviewer GsBz (2/3)
W2 & Q3. MMLU, per-task performance, and missing ablations
We would like to clarify that our work focuses on task-agnostic one-shot expert pruning for MoE models. We calibrate on general-domain C4 data and evaluate a single pruned model across all ten tasks to assess how well it preserves broad capabilities. No method leads on every benchmark in these comparisons, so the overall average and per-task ranks are important for assessing this goal.
Calibration data can affect which capabilities are retained after pruning [1]. For MMLU, we calibrated NAPE using 32 sequences from the validation split. On LLaDA-MoE at 50% sparsity, this yields 43.97 on the test set, compared with 28.41 under C4 calibration. This result shows that task-specific calibration can help NAPE preserve performance on a particular task.
We repeated Random pruning on LLaDA-MoE at 50% sparsity with seeds 1, 2, and 3. Each Random entry reports mean ± standard deviation over these three additional runs, alongside NAPE from Table 1.
| Method | ARC-c | ARC-e | BoolQ | COPA | HellaSwag | MMLU | OBQA | PIQA | RTE | WinoGrande | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Random (3 seeds) | 29.64 ± 0.81 | 38.86 ± 1.61 | 51.72 ± 7.17 | 60.33 ± 3.21 | 37.84 ± 1.53 | 31.54 ± 1.32 | 26.87 ± 0.64 | 57.40 ± 0.52 | 58.36 ± 1.27 | 51.25 ± 1.04 | 44.38 ± 0.88 |
| NAPE (Table 1) | 31.74 | 43.94 | 53.79 | 70.00 | 54.49 | 28.41 | 29.60 | 63.33 | 58.84 | 55.88 | 49.00 |
NAPE exceeds the Random mean on nine of ten tasks and improves the overall average by 4.62 points. Its ten-task average also exceeds that of all three Random runs, supporting the value of NAPE's expert scores for preserving performance across tasks.
The table below reports NAPE's mean per-task rank and number of first-place finishes in comparison with Random, Frequency, EAN, REAP, and MoNE.
| Model | Sparsity | NAPE mean rank ↓ | NAPE first-place tasks |
|---|---|---|---|
| LLaDA-MoE | 25% | 2.00 | 4 / 10 |
| LLaDA-MoE | 50% | 2.40 | 5 / 10 |
| SDAR | 25% | 2.00 | 5 / 10 |
| SDAR | 50% | 1.50 | 6 / 10 |
Ranks are averaged over the ten tasks, with rank 1 denoting the best result and ties assigned their average rank. NAPE has the best mean rank in all four settings, tied with Frequency on SDAR at 25%. Across all 40 comparisons, it ranks first 20 times and has the best overall mean rank of 1.98.
The gains also extend beyond BoolQ. After excluding it, NAPE's average gains over the strongest original baseline remain +0.52 / +1.62 / +0.34 / +6.36 points for LLaDA-MoE 25% / 50% and SDAR 25% / 50%, respectively. NAPE's average advantage therefore persists when BoolQ is excluded.
For the ablations, the original eight tasks already cover diverse reasoning and language-understanding abilities. Following your suggestion, we extended the fixed-ratio and masked-criteria ablations to MMLU and HellaSwag on LLaDA-MoE, covering all ten tasks from the main experiments. The table reports the ten-task fixed-ratio results at 50% sparsity, alongside NAPE from Table 1.
| Calibration | ARC-c | ARC-e | BoolQ | COPA | HellaSwag | MMLU | OBQA | PIQA | RTE | WinoG | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Fixed r=0.1 | 32.17 | 44.61 | 44.56 | 70.00 | 54.29 | 27.92 | 28.20 | 63.00 | 59.93 | 57.06 | 48.17 |
| Fixed r=0.3 | 30.72 | 44.11 | 45.11 | 72.00 | 54.67 | 28.28 | 29.80 | 63.49 | 49.46 | 55.80 | 47.34 |
| Fixed r=0.5 | 29.69 | 41.46 | 54.25 | 67.00 | 53.32 | 31.32 | 29.60 | 61.21 | 52.71 | 54.85 | 47.54 |
| Fixed r=0.7 | 31.23 | 41.37 | 44.13 | 68.00 | 52.17 | 32.21 | 29.20 | 61.97 | 51.99 | 55.80 | 46.81 |
| Fixed r=0.9 | 31.14 | 46.30 | 38.96 | 70.00 | 50.93 | 38.90 | 27.80 | 62.68 | 62.82 | 53.35 | 48.29 |
| NAPE (Table 1) | 31.74 | 43.94 | 53.79 | 70.00 | 54.49 | 28.41 | 29.60 | 63.33 | 58.84 | 55.88 | 49.00 |
HellaSwag and MMLU favor different fixed ratios, while NAPE achieves the highest ten-task average in this comparison. NAPE also outperforms both masked criteria on all ten tasks. Together, these ablations support using diffusion-loss gradients and averaging scores across noise levels to preserve performance across tasks.
Response to Reviewer GsBz (3/3)
Q4. XSum settings and calibration cost
Thank you for reading our paper so closely.
For XSum, we evaluate on 512 validation documents, using up to 512 input tokens and 64 output tokens. Decoding uses temperature 0, 32-token blocks, 32 steps per block, and low-confidence remasking.
Regarding calibration cost, we compared NAPE with forward-only scoring on LLaDA-MoE using one NVIDIA RTX 6000 Ada Generation GPU. Both use the same 128 calibration sequences of 1,024 tokens. Excluding model loading, forward-only scoring takes 47.4 seconds with 14.0 GiB peak memory, while NAPE takes 70.0 seconds with 19.1 GiB. NAPE takes about 48% longer to compute the expert scores, but this cost is incurred only during pruning and adds no inference overhead.
References
[1] Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning. EMNLP 2024.
Thank you again for the comments, which strengthened our empirical comparison and provided further insights into our method. We have included the new results in the revision, with the additions highlighted in blue. We hope our response resolves your concerns and would be very grateful if you could consider raising your score.
Response to Reviewer GsBz (1/3)
Dear Reviewer GsBz,
Thank you for the careful and constructive review. We especially appreciate your recognition of the SDAR result, the memory savings, and our rigorous experimental setup. We are happy to address each concern below.
Thank you for this suggestion. Tables 13–14 of our paper already compare multi-noise calibration with five fixed noise ratios on downstream tasks. The tables below report NAPE and the best fixed ratio for each backbone, selected by its eight-task average.
LLaDA-MoE, 50% sparsity; eight tasks from Table 13.
SDAR, 50% sparsity; eight tasks from Table 14.
NAPE achieves a higher average on both backbones using the same noise-sampling rule, while the best fixed ratio differs between the two models. These results support using multi-noise calibration without choosing a fixed ratio for each backbone.
To examine the effects of noise level and calibration sampling, we compared expert rankings across three disjoint C4 calibration sets on LLaDA-MoE. The noise-gap columns compare fixed-ratio scores across sets; the final column compares NAPE's multi-noise scores across the same sets.
The median ranking correlation is 0.945 at the same noise ratio and falls to 0.692 between the lowest and highest noise ratios. NAPE's multi-noise rankings remain consistent across calibration sets, with a median correlation of 0.945 and 3.35 of the 32 retained experts differing on average. Expert rankings change more across noise levels than across calibration sets, supporting the use of multiple noise levels to estimate expert importance.
For SDAR, we independently mask tokens across the whole sequence at the sampled noise ratio. Attention remains blockwise-causal with 32-token blocks, and the loss is computed only at masked target positions.
Regarding rerouting, our main experiments remove experts across all layers and recompute routing over the retained experts. The downstream results therefore account for both rerouting and joint removal.
To examine whether the removal analysis also holds in earlier layers and for groups of experts, we evaluated layers 0, 5, 10, and 15 of LLaDA-MoE. We removed one expert or groups of 8 or 16 experts, recomputed routing, and measured the correlation between pruning scores and held-out loss changes.
NAPE achieves the highest single-expert correlation at all four evaluated layers. For groups of 8 and 16 experts, its four-layer average correlations are 0.868 and 0.708, compared with 0.358 and 0.317 for the strongest baseline in each case. These results support NAPE's ability to identify important experts in earlier layers and when multiple experts are removed together.