acceptodds
Under review as a conference paper at ICLR 2027

Beyond Static Importance: Noise-Aware Expert Pruning for Diffusion Language Models

Abstract

Expert pruning has been widely studied for autoregressive Mixture-of-Experts (MoE) language models as an effective way to reduce memory overhead by removing less important experts. However, pruning methods developed for autoregressive models do not transfer well to diffusion language models (dLLMs), which perform multi-step denoising and may rely on different experts across noise levels, unlike autoregressive models that generate text along a single left-to-right trajectory. Our analysis shows that routing and activation statistics can mis-rank experts under the denoising objective, while expert importance varies substantially across noise levels, with different rankings at high and low noise levels. Consequently, evaluating experts under a single input distribution or fixed noise level may retain redundant experts while pruning those needed at other denoising stages. Motivated by these observations, we propose NAPE, a one-shot pruning criterion that scores each expert using the absolute gradient of the masked-prediction loss with respect to its gate and aggregates the score over mask ratios spanning high- to low-noise stages. By aggregating expert importance across noise levels, NAPE avoids biased estimates from calibration at fixed noise. Experiments on two MoE dLLM backbones across ten downstream benchmarks show that NAPE consistently outperforms existing pruning methods under different pruning ratios, improving the strongest baseline by up to 12.3% relative at 50% sparsity. We release our code at https://anonymous.4open.science/r/NAPE-35C2 to facilitate future research on efficient MoE dLLMs.

Primary area:
generative models
Keywords:
Expert Pruning, Diffusion Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GsBzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at expert pruning for MoE diffusion language models. The proposed criterion, NAPE, scores each expert by the gradient of the masked-prediction loss with respect to its gate, averaged over calibration data masked at different ratios. It is evaluated on LLaDA-MoE and SDAR at 25% and 50% pruning. The clearest win is SDAR at 50%, where it beats EAN by about 7 points on average.

Critical strengths
  1. The SDAR 50% result is strong for a method that needs no retraining, and the memory savings are real since the pruned experts are actually removed.

  2. The experimental setup is careful. All methods use the same calibration data and evaluation masks, and the ablations cover the obvious questions (activation-based scores, a single fixed mask ratio). It is also convenient that one setting works for both models.

Critical weaknesses
  1. I am convinced that gradient scores help, less so that the multi-noise calibration does. Against the best single mask ratio, NAPE gains under a point on LLaDA and about a quarter point on SDAR. Picking the best ratio afterwards favors the baseline, I admit, but the appendix shows bigger swings than that just from changing the calibration size or the number of evaluation masks, and several main-table margins are also only a few tenths. Without repeated seeds I can't tell how much of this is noise. The retained-set recovery experiment doesn't really settle it either, since its reference is built from the same objective NAPE estimates.

  2. The averages hide some uneven results. On LLaDA at 50%, NAPE is below random pruning on MMLU, and on SDAR at 50% it is behind EAN on MMLU, HellaSwag and PIQA; most of the average gain comes from BoolQ. MMLU and HellaSwag are also missing from the ablations. Separately, the gate gradient is a local estimate under the original routing, while pruning changes the routing. The removal experiment checks this, but only one expert at a time in the last layer, which says little about removing half the experts across all layers.

Questions for authors
  1. Can you repeat the fixed-ratio comparison over a few calibration seeds and report the variance? Ideally the fixed ratio would be chosen on held-out data.

  2. How are masks applied during SDAR calibration, over the whole sequence or within a block? Does the removal correlation still hold in earlier layers, or when several experts are removed together?

  3. Why is MMLU on LLaDA at 50% below random? It would help to include MMLU and HellaSwag in the ablations.

  4. A few missing details: the XSum sample count and decoding settings, and calibration time and memory compared with forward-only scoring.

5 comments
AuthorsOpenReview ↗

Response to Reviewer GsBz (1/3)

Dear Reviewer GsBz,

Thank you for the careful and constructive review. We especially appreciate your recognition of the SDAR result, the memory savings, and our rigorous experimental setup. We are happy to address each concern below.


W1 & Q1. Multi-noise calibration and repeated comparisons

Thank you for this suggestion. Tables 13–14 of our paper already compare multi-noise calibration with five fixed noise ratios on downstream tasks. The tables below report NAPE and the best fixed ratio for each backbone, selected by its eight-task average.

LLaDA-MoE, 50% sparsity; eight tasks from Table 13.

CalibrationARC-cARC-eBoolQCOPAOBQAPIQARTEWinoGAvg.
Best fixed ratio: r=0.132.1744.6144.5670.0028.2063.0059.9357.0649.94
NAPE31.7443.9453.7970.0029.6063.3358.8455.8850.89

SDAR, 50% sparsity; eight tasks from Table 14.

CalibrationARC-cARC-eBoolQCOPAOBQAPIQARTEWinoGAvg.
Best fixed ratio: r=0.542.5868.3585.9984.0037.2068.5074.7368.6766.25
NAPE43.8668.5687.0384.0038.4069.2172.5668.3566.50

NAPE achieves a higher average on both backbones using the same noise-sampling rule, while the best fixed ratio differs between the two models. These results support using multi-noise calibration without choosing a fixed ratio for each backbone.

To examine the effects of noise level and calibration sampling, we compared expert rankings across three disjoint C4 calibration sets on LLaDA-MoE. The noise-gap columns compare fixed-ratio scores across sets; the final column compares NAPE's multi-noise scores across the same sets.

Noise gap ∣ri−rj∣\lvert r_i-r_j\rvert0.00.20.40.60.8NAPE across sets
Median Spearman ρ\rho0.9450.9160.8700.7930.6920.945
Mean differing retained experts (of 32)3.024.025.236.598.093.35

The median ranking correlation is 0.945 at the same noise ratio and falls to 0.692 between the lowest and highest noise ratios. NAPE's multi-noise rankings remain consistent across calibration sets, with a median correlation of 0.945 and 3.35 of the 32 retained experts differing on average. Expert rankings change more across noise levels than across calibration sets, supporting the use of multiple noise levels to estimate expert importance.


W2 & Q2. SDAR masking, rerouting, and joint expert removal

For SDAR, we independently mask tokens across the whole sequence at the sampled noise ratio. Attention remains blockwise-causal with 32-token blocks, and the loss is computed only at masked target positions.

Regarding rerouting, our main experiments remove experts across all layers and recompute routing over the retained experts. The downstream results therefore account for both rerouting and joint removal.

To examine whether the removal analysis also holds in earlier layers and for groups of experts, we evaluated layers 0, 5, 10, and 15 of LLaDA-MoE. We removed one expert or groups of 8 or 16 experts, recomputed routing, and measured the correlation between pruning scores and held-out loss changes.

RemovalCriterionLayer 0Layer 5Layer 10Layer 15Mean
Single expertFrequency0.4180.4630.0490.4960.356
Single expertEAN0.4440.5060.488-0.0700.342
Single expertREAP0.4480.5740.539-0.1590.350
Single expertMoNE0.4360.5720.560-0.2770.323
Single expertNAPE0.5220.8250.8230.9580.782
8 expertsFrequency0.3480.670-0.1900.5380.341
8 expertsEAN0.7910.5380.394-0.2910.358
8 expertsREAP0.8260.6140.339-0.3900.347
8 expertsMoNE0.8620.5990.368-0.4710.339
8 expertsNAPE0.8430.8900.8190.9210.868
16 expertsFrequency0.1190.5050.3090.3340.317
16 expertsEAN0.6130.3750.335-0.1960.282
16 expertsREAP0.7230.4180.186-0.1310.299
16 expertsMoNE0.8000.3450.154-0.1710.282
16 expertsNAPE0.7270.6960.7290.6810.708

NAPE achieves the highest single-expert correlation at all four evaluated layers. For groups of 8 and 16 experts, its four-layer average correlations are 0.868 and 0.708, compared with 0.358 and 0.317 for the strongest baseline in each case. These results support NAPE's ability to identify important experts in earlier layers and when multiple experts are removed together.

AuthorsOpenReview ↗

Response to Reviewer GsBz (2/3)

W2 & Q3. MMLU, per-task performance, and missing ablations

We would like to clarify that our work focuses on task-agnostic one-shot expert pruning for MoE models. We calibrate on general-domain C4 data and evaluate a single pruned model across all ten tasks to assess how well it preserves broad capabilities. No method leads on every benchmark in these comparisons, so the overall average and per-task ranks are important for assessing this goal.

Calibration data can affect which capabilities are retained after pruning [1]. For MMLU, we calibrated NAPE using 32 sequences from the validation split. On LLaDA-MoE at 50% sparsity, this yields 43.97 on the test set, compared with 28.41 under C4 calibration. This result shows that task-specific calibration can help NAPE preserve performance on a particular task.

We repeated Random pruning on LLaDA-MoE at 50% sparsity with seeds 1, 2, and 3. Each Random entry reports mean ± standard deviation over these three additional runs, alongside NAPE from Table 1.

MethodARC-cARC-eBoolQCOPAHellaSwagMMLUOBQAPIQARTEWinoGrandeAvg.
Random (3 seeds)29.64 ± 0.8138.86 ± 1.6151.72 ± 7.1760.33 ± 3.2137.84 ± 1.5331.54 ± 1.3226.87 ± 0.6457.40 ± 0.5258.36 ± 1.2751.25 ± 1.0444.38 ± 0.88
NAPE (Table 1)31.7443.9453.7970.0054.4928.4129.6063.3358.8455.8849.00

NAPE exceeds the Random mean on nine of ten tasks and improves the overall average by 4.62 points. Its ten-task average also exceeds that of all three Random runs, supporting the value of NAPE's expert scores for preserving performance across tasks.

The table below reports NAPE's mean per-task rank and number of first-place finishes in comparison with Random, Frequency, EAN, REAP, and MoNE.

ModelSparsityNAPE mean rank ↓NAPE first-place tasks
LLaDA-MoE25%2.004 / 10
LLaDA-MoE50%2.405 / 10
SDAR25%2.005 / 10
SDAR50%1.506 / 10

Ranks are averaged over the ten tasks, with rank 1 denoting the best result and ties assigned their average rank. NAPE has the best mean rank in all four settings, tied with Frequency on SDAR at 25%. Across all 40 comparisons, it ranks first 20 times and has the best overall mean rank of 1.98.

The gains also extend beyond BoolQ. After excluding it, NAPE's average gains over the strongest original baseline remain +0.52 / +1.62 / +0.34 / +6.36 points for LLaDA-MoE 25% / 50% and SDAR 25% / 50%, respectively. NAPE's average advantage therefore persists when BoolQ is excluded.

For the ablations, the original eight tasks already cover diverse reasoning and language-understanding abilities. Following your suggestion, we extended the fixed-ratio and masked-criteria ablations to MMLU and HellaSwag on LLaDA-MoE, covering all ten tasks from the main experiments. The table reports the ten-task fixed-ratio results at 50% sparsity, alongside NAPE from Table 1.

CalibrationARC-cARC-eBoolQCOPAHellaSwagMMLUOBQAPIQARTEWinoGAvg.
Fixed r=0.132.1744.6144.5670.0054.2927.9228.2063.0059.9357.0648.17
Fixed r=0.330.7244.1145.1172.0054.6728.2829.8063.4949.4655.8047.34
Fixed r=0.529.6941.4654.2567.0053.3231.3229.6061.2152.7154.8547.54
Fixed r=0.731.2341.3744.1368.0052.1732.2129.2061.9751.9955.8046.81
Fixed r=0.931.1446.3038.9670.0050.9338.9027.8062.6862.8253.3548.29
NAPE (Table 1)31.7443.9453.7970.0054.4928.4129.6063.3358.8455.8849.00

HellaSwag and MMLU favor different fixed ratios, while NAPE achieves the highest ten-task average in this comparison. NAPE also outperforms both masked criteria on all ten tasks. Together, these ablations support using diffusion-loss gradients and averaging scores across noise levels to preserve performance across tasks.

AuthorsOpenReview ↗

Response to Reviewer GsBz (3/3)

Q4. XSum settings and calibration cost

Thank you for reading our paper so closely.

For XSum, we evaluate on 512 validation documents, using up to 512 input tokens and 64 output tokens. Decoding uses temperature 0, 32-token blocks, 32 steps per block, and low-confidence remasking.

Regarding calibration cost, we compared NAPE with forward-only scoring on LLaDA-MoE using one NVIDIA RTX 6000 Ada Generation GPU. Both use the same 128 calibration sequences of 1,024 tokens. Excluding model loading, forward-only scoring takes 47.4 seconds with 14.0 GiB peak memory, while NAPE takes 70.0 seconds with 19.1 GiB. NAPE takes about 48% longer to compute the expert scores, but this cost is incurred only during pruning and adds no inference overhead.

References

[1] Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning. EMNLP 2024.


Thank you again for the comments, which strengthened our empirical comparison and provided further insights into our method. We have included the new results in the revision, with the additions highlighted in blue. We hope our response resolves your concerns and would be very grateful if you could consider raising your score.

Reviewer_5DYfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is about how to do expert pruning for DLMs. The authors found DLM relies on different expert under different noise level (mask ratio), so selector base on AR models (static routing-frequency or activation norm scores) will mis-rank what expert to keep. Then they propose NAPE, which score each expert by the absolute gradient of the denoising loss with respect to its gate, accumulated over calibration states sampled across noise levels, and show it consistently beats existing criteria on two MoE dLLMs and ten benchmarks.

Critical strengths

The paper is, to my knowledge, the first to study expert pruning for MoE diffusion language models, a practically important problem since all experts must stay resident in memory regardless of how few are activated. It shows the problem is not a trivial transfer from the autoregressive setting: expert importance shifts substantially across noise levels, so routing- and activation-based criteria that work for arLLMs mis-rank experts under the diffusion objective. The proposed NAPE directly addresses this with noise-aware gate-gradient saliency and, under a controlled protocol, consistently outperforms existing criteria across two backbones and ten benchmarks.

Critical weaknesses
  1. The baseline methods are weak. NAPE is compared only against three prospective statistical metrics (Frequency, EAN, REAP) rather than MoNE (ICLR 2026), a method cited in the paper itself. Given the limited novelty of the approach, its contribution relies heavily on empirical comparison; however, the current set of baselines fails to demonstrate superiority over state-of-the-art (SOTA) methods.
  2. The scope is limited to the token-choice mechanism. Both the method and the pruning workflow are based on the token-choice Top-k routing mechanism, despite the paper's claim of applicability to various MoE dLLMs. The paper does not explore whether NAPE is applicable to other routing schemes (such as expert-choice routing, where pruning can directly reduce total computational cost).
  3. There is a significant absolute drop in performance, with no data provided on recovery effects or latency. At 50% sparsity, LLaDA-MoE's performance falls from 62.85 to 49.00, whereas random pruning alone achieves 45.09. The paper does not demonstrate a post-pruning fine-tuning process to verify model usability, nor does it provide latency or throughput data, reporting only memory savings.
Questions for authors
  1. Add MoNE baseline
  2. Add EC MoE discusion
  3. Add at least one post-pruning fine-tuning process to see improvements.

I will raise my score after the author showed discussion here.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5DYf (1/2)

Dear Reviewer 5DYf,

Thank you for the constructive review. We especially appreciate your recognition of the importance of expert pruning for MoE dLLMs and our controlled evaluation. We are happy to address each concern below.


W1 & Q1. MoNE baseline

Thank you for your suggestion. We would like to clarify that our main experiments already include current state-of-the-art one-shot MoE pruning methods, such as REAP [1], which was also accepted at ICLR 2026.

To address your concern further, we further evaluated MoNE on both backbones under the same settings as our main experiments. The tables below report MoNE and NAPE on all ten tasks, with the other baselines reported in Tables 1-2 of our paper. Bold indicates the better result within each task.

LLaDA-MoE.

SparsityMethodARC-cARC-eBoolQCOPAHellaSwagMMLUOBQAPIQARTEWinoGAvg.
25%MoNE42.9261.1173.1876.0060.1956.4032.2064.3671.8458.8859.71
25%NAPE41.4761.8374.7479.0061.0752.9831.4067.4174.0158.5660.25
50%MoNE32.3443.1049.1467.0052.0329.9227.6061.4362.4554.4647.95
50%NAPE31.7443.9453.7970.0054.4928.4129.6063.3358.8455.8849.00

SDAR.

SparsityMethodARC-cARC-eBoolQCOPAHellaSwagMMLUOBQAPIQARTEWinoGAvg.
25%MoNE49.2370.7986.4878.0064.1772.0938.6068.9378.7065.2767.23
25%NAPE50.0973.4887.7185.0067.4774.5339.6069.5380.8768.5969.69
50%MoNE35.3254.8877.2573.0054.6445.6532.8063.9874.3760.3857.23
50%NAPE43.8668.5687.0384.0064.6755.6838.4069.2172.5668.3565.23

Considering MoNE together with the original baselines, NAPE stills retains the highest ten-task average in all four settings, ranks first in 20 of 40 comparisons, and has the best overall mean rank of 1.98 among the six pruning methods. Its advantage therefore persists across both backbones and sparsity levels.

We also compared MoNE on XSum, a open generation task at 50% sparsity using the same settings as Table 5.

MethodLLaDA-MoE ROUGE-1SDAR ROUGE-1LLaDA-MoE ROUGE-LSDAR ROUGE-L
MoNE20.4119.8714.2013.86
NAPE21.3324.1515.0518.43

NAPE retains the highest ROUGE-1 and ROUGE-L among the pruning methods on both backbones when MoNE is included alongside the baselines in Table 5, showing its advantage extends to full multi-step generation.

The table below reports MoNE's denoising NLL at 50% sparsity alongside NAPE from Figure 3(b).

MethodLLaDA-MoE NLL ↓SDAR NLL ↓
MoNE4.0153.168
NAPE3.8202.675

NAPE also retains the lowest NLL, consistent with its advantage in summary generation.


W2 & Q2. Expert-choice routing

NAPE can in principle be extended to expert-choice routing [2]. Once each expert has selected its tokens, we can compute its score using the diffusion-loss gradient with respect to the scalar gate on its output and average the scores across noise levels. If the token capacity per expert is kept fixed, removing experts would also reduce expert computation. Our experiments focus on token-choice routing, which is widely used in MoE models, including LLaDA-MoE and SDAR. We hope future work will further explore and validate NAPE under expert-choice routing.

AuthorsOpenReview ↗

Response to Reviewer 5DYf (2/2)

W3 & Q3. Post-pruning recovery

Our work focuses on one-shot expert pruning, with the primary goal of reducing MoE memory requirements while preserving model performance without additional training. This makes it a practical approach to MoE compression when training resources are limited.

We agree that it is valuable to evaluate whether fine-tuning can further improve the pruned models. Following your suggestion, we fine-tuned all five 50%-pruned LLaDA-MoE models using the same data and training budget. Specifically, we used LoRA (rank 16) for 400 optimization steps on 3,200 C4 sequences disjoint from the calibration data, with a sequence length of 1,024 and a learning rate of 1e-4. The table reports all ten tasks after recovery, with the original average included for comparison.

MethodARC-cARC-eBoolQCOPAHellaSwagMMLUOBQAPIQARTEWinoGAvg. beforeAvg. after
Frequency29.2743.9473.2162.0048.6941.7127.6060.8361.0154.9346.9450.32
EAN31.9144.0768.1764.0051.9338.9829.6061.5965.7055.8048.8151.18
REAP32.5144.6573.2463.0051.3034.9529.0061.5959.9354.3847.3950.46
MoNE33.5344.5767.4963.0051.4534.8026.0061.2667.8754.5447.9550.45
NAPE31.2346.1371.0769.0054.9835.5429.8064.0460.2957.7049.0051.98

After recovery, NAPE improves from 49.00 to 51.98 and retains the highest average, exceeding EAN by 0.80 points. NAPE's lead therefore carries over to post-pruning fine-tuning, supporting its use as a starting point for recovery as well as for one-shot deployment.


W3. Inference throughput

For inference efficiency, we measured native-decoding throughput using 256-token prompts and outputs and report the median of three runs. Memory values are from Table 3 in our paper.

ModelSparsityPeak loading memoryGPUsTokens/s, batch 1 / 8 / 32
LLaDA-MoEFull15.55 GB13.88 / 27.25 / 52.03
LLaDA-MoE25%12.32 GB15.10 / 32.53 / 55.27
LLaDA-MoE50%9.10 GB17.38 / 41.05 / 59.76
SDARFull58.25 GB20.62 / 4.62 / 15.81
SDAR25%44.92 GB20.79 / 5.77 / 20.15
SDAR50%31.65 GB11.34 / 8.93 / 34.41

At 50% sparsity, batch-1 throughput improves by 1.90x on LLaDA-MoE and 2.14x on SDAR under these deployment configurations, with SDAR also moving from two GPUs to one. At this sparsity, NAPE retains approximately 93% of SDAR's full-model ten-task accuracy. These results show that NAPE reduces deployment costs while preserving most of the full model's accuracy.

References

[1] REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression. ICLR 2026.

[2] Mixture-of-Experts with Expert Choice Routing. NeurIPS 2022


Thank you again for the comments, which strengthened our empirical comparison and provided further insights into our method. We have included the new results in the revision, with the additions highlighted in blue. We hope our response resolves your concerns and would be very grateful if you could consider raising your score.

Thank you for the detailed response and the additional experiments. I appreciate the effort. However, after reading the revision carefully, my main concerns remain, and I will keep my current score.

  1. The gains are mostly within evaluation noise (W1). With MoNE included, NAPE’s lead in ten-task average over the strongest baseline is +0.54 (LLaDA 25%), +0.19 (LLaDA 50%), +0.29 (SDAR 25%), and +7.15 (SDAR 50%). The paper’s own analyses show larger noise than the first three margins: in Table 9, changing the Monte Carlo budget from 16 to 32 shifts per-task accuracy by up to about ±5 points, and in Table 10, NAPE’s average on LLaDA moves between 51.72, 47.19, and 50.89 across calibration-set sizes. No multi-seed results or confidence intervals are reported, so only the SDAR 50% setting is solid evidence of superiority.

  2. The MoNE baseline is not the full method (W1). Appendix C states that only the MoNE score is used to rank experts under the common deletion protocol. Its central component, replacing pruned experts with novice vectors, is not applied. This is a criterion-level comparison, not a comparison against MoNE as published, and it should be stated in the main text.

  3. Recovery does not resolve the usability concern (W3). After LoRA recovery, the 50%-pruned LLaDA reaches 51.98 versus 62.85 for the full model. More than half of NAPE’s +2.98 improvement comes from BoolQ alone, all five methods end within 50.3–52.0, and NAPE’s 0.80 lead over EAN is again within noise. Recovery is reported only for LLaDA.

Summary. As noted in my original review, given the limited methodological novelty, the contribution relies heavily on the empirical comparison. After the rebuttal, a clear advantage is established in only one of four settings. I therefore maintain my rating of 2 (weak rejection). I encourage the authors to add multi-seed results or confidence intervals and a full MoNE comparison in a future version.

Reviewer_k75TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies one-shot expert pruning for MoE diffusion language models. It argues that standard autoregressive pruning criteria are not ideal for dLLMs. They ignore the masked-prediction objective. The authors also show that expert importance changes with noise level. Rankings at low and high mask ratios differ. About 16 to 23 percent of selected experts change between extremes.

The proposed method is NAPE. It assigns a scalar gate to each expert. It scores experts by the absolute gradient of the masked-prediction loss with respect to the gate. Scores are averaged over calibration samples with mask ratios from U(0.1, 0.9). The lowest-scoring experts are removed.

NAPE is tested on LLaDA-MoE-7B-A1B and SDAR-30B-A3B at 25% and 50% sparsity. It is compared with Random, Frequency, EAN, and REAP. Tasks include ten likelihood-based zero-shot tasks, XSum, and masked-token NLL.

Critical strengths

S1: The paper addresses a relevant and underexplored problem.

MoE diffusion language models are still new. Keeping all experts in memory is costly. There is limited work on expert pruning for this model class. The paper makes a useful contribution by showing that expert rankings change with noise level. This effect appears across the network, not only in a few layers.

This point matters beyond NAPE. Calibration-based pruning for dLLMs should consider multiple noise levels. The paper gives a good reason to question a single mask ratio. The experimental setup is also fair. All methods use the same calibration data and pipeline. Monte Carlo masks are fixed across methods.

S2: NAPE is simple and gives a strong practical gain in the best setting.

NAPE is easy to implement. It needs one backward pass per calibration sample. It stores only scalar gate gradients. It does not require fine-tuning. The best result is on SDAR-30B-A3B at 50% sparsity. The pruned model retains 93% of dense-model accuracy. The best baseline retains about 83%. The pruned model also fits on a single 48 GB GPU.

The use of gate gradients is well motivated. They capture how an expert output relates to the loss gradient. Activation norms mostly reflect activation magnitude. Gate gradients therefore give a better estimate of contribution to the prediction objective. The SDAR result is the most convincing part of the paper.

Critical weaknesses

W1: The evidence for consistent gains is weaker than suggested.

NAPE is best in all four settings. However, the margins are small in three of them. On LLaDA at 25% and 50% sparsity, the gains are 0.77 and 0.19 points. On SDAR at 25% sparsity, the gain is 0.29 points. The main tables do not report error bars or significance tests. It is hard to know if these differences are meaningful.

The appendix increases this concern. Results are sensitive to calibration choices. For example, NAPE's eight-task average on LLaDA drops from 51.72 with 32 samples to 47.19 with 64 samples. It rises again to 50.89 with 128 samples. This variation is larger than some reported margins.

The ablation baselines are also problematic. Tables 4, 8, and 9 compare NAPE with masked-target versions of EAN and REAP. They do not use the original criteria. This matters because original EAN is competitive. On LLaDA at 50% sparsity, original EAN scores 50.96 on the eight-task subset. NAPE scores 50.89. Masked EAN scores only 46.19. Therefore, the ablation gap does not clearly show that NAPE beats the strongest baseline.

W2: The contribution of noise-aware calibration is not clear.

Figure 1 shows that expert rankings change with noise level. This is interesting. But the results do not clearly show that this variation explains NAPE's gains. Using a fixed mask ratio of 0.5 on SDAR gives 66.25. This is only 0.24 points below NAPE. On LLaDA, NAPE's advantage over the best fixed-ratio setting comes mostly from BoolQ. This is hard to interpret because NAPE's BoolQ score varies by 12 points across calibration sizes.

The validation is also somewhat circular. The bias argument and recovery experiment use a noise-marginal ranking as reference. NAPE is designed around this ranking. This gives limited independent evidence that noise-aware calibration is the key factor.

It is possible that gradient-based scoring explains more of the gain. This scoring rule is related to gate-gradient importance from Michel et al. (2019). The paper does not test it on an autoregressive MoE model. Therefore, it is unclear whether the benefit comes from the diffusion setting or from gradient-based scoring in general.

Questions for authors
  1. How sensitive are the results to calibration randomness? Please report means and standard deviations over multiple calibration seeds. Vary calibration samples, noise draws, and masks. Please include multiple Random-pruning runs. Paired significance tests would help.
  2. Why do the ablations use masked-target EAN and REAP instead of the original criteria? Original EAN is competitive on LLaDA. Including it would make the ablations clearer.
  3. How does gate-gradient scoring compare with EAN and REAP on an autoregressive MoE model? Qwen3-30B-A3B would be a natural test. If it wins there too, the claim about poor transfer to dLLMs needs revision.
  4. How stable are expert rankings across independent calibration sets at the same mask ratio? Comparing two disjoint calibration sets would help. It would separate noise-level effects from estimation noise.
  5. Do the gains transfer to full multi-step generation tasks? Examples include GSM8K and HumanEval. Most current tasks use likelihood scoring. Full generation tasks would better test the method in practice.
7 comments
AuthorsOpenReview ↗

Response to Reviewer k75T (1/3)

Dear Reviewer k75T,

Thank you for the constructive and careful review. We especially appreciate your recognition of noise-dependent expert importance, our controlled evaluation, and the practical benefit on SDAR. We are happy to address each concern below.


W1 & Q1. Calibration randomness and the magnitude of the gains

Thank you for this suggestion. We would like to clarify that our work focuses on task-agnostic one-shot expert pruning for MoE models. We evaluate a single pruned model across all ten tasks to assess how well it preserves broad capabilities. As the results show, no pruning method can lead on every benchmark in these comparisons, so the overall average matter most. We repeated Random pruning on LLaDA-MoE at 50% sparsity with three additional seeds (1, 2, and 3). The table reports mean ± standard deviation across these runs, alongside NAPE from Table 1.

MethodARC-cARC-eBoolQCOPAHellaSwagMMLUOBQAPIQARTEWinoGrandeAvg.
Random (3 seeds)29.64 ± 0.8138.86 ± 1.6151.72 ± 7.1760.33 ± 3.2137.84 ± 1.5331.54 ± 1.3226.87 ± 0.6457.40 ± 0.5258.36 ± 1.2751.25 ± 1.0444.38 ± 0.88
NAPE (Table 1)31.7443.9453.7970.0054.4928.4129.6063.3358.8455.8849.00

NAPE exceeds the Random mean on nine of ten tasks and improves the overall average by 4.62 points. Its ten-task average also exceeds that of all three Random runs. This supports the value of NAPE's expert scores for preserving broad capability at the same sparsity.

On SDAR at 50% sparsity, NAPE retains approximately 93% of full-model ten-task accuracy, compared with 83% for the strongest original baseline, while fitting on a single 48 GB GPU. NAPE thus preserves substantially more of the model's capability under the same compression budget.


W1 & Q2. Original EAN/REAP and controlled ablations

Thank you for raising this point. We includes comparisons with the original EAN and REAP in Tables 1–2, the XSum evaluation in Table 5, and Figure 3(b). The masked variants use the same corrupted inputs and target positions as NAPE to examine the contribution of the scoring criterion. For completeness, the table below presents both the original and masked variants on all ten LLaDA-MoE tasks at 50% sparsity, with the same calibration budget as the main experiments. The original EAN, REAP, and NAPE results are taken from Table 1.

MethodARC-cARC-eBoolQCOPAHellaSwagMMLUOBQAPIQARTEWinoGAvg.
EAN31.3140.9566.4567.0050.8229.5826.4059.9659.5756.0448.81
REAP31.4041.3352.2367.0050.1628.3627.4060.1259.9355.9647.39
Masked-EAN29.2735.4846.4269.0046.2324.9725.6057.2957.4051.7844.34
Masked-REAP30.8937.7944.8965.0048.8025.6026.6059.0954.8755.0144.85
NAPE31.7443.9453.7970.0054.4928.4129.6063.3358.8455.8849.00

With the corrupted inputs and target positions matched, NAPE outperforms both masked criteria on all ten tasks. These results show that measuring sensitivity to the diffusion loss improves expert selection beyond the use of masked inputs alone. NAPE also retains the highest ten-task average against the original EAN and REAP criteria.

AuthorsOpenReview ↗

Response to Reviewer k75T (2/3)

W2 & Q3. Gate-gradient scoring on autoregressive MoE models

To examine whether the gains can be attributed to gate gradients alone, we applied the same score with next-token loss to Qwen3-30B-A3B and OLMoE-1B-7B. All methods use the same calibration and evaluation settings with our main experiments.

Qwen3-30B-A3B

SparsityMethodARC-cARC-eBoolQCOPAHellaSwagMMLUOBQAPIQARTEWinoGAvg.
25%EAN62.8885.0688.6987.0079.5076.3244.0080.6376.9073.0975.41
25%REAP63.0582.6288.6288.0078.5374.6344.4081.9474.7373.1674.97
25%Gate gradient62.3784.0588.5689.0079.6975.6944.6080.4175.4572.9375.27
50%EAN57.3477.9088.3289.0078.2662.9645.4081.6179.0673.6473.35
50%REAP52.3074.6685.7580.0071.0361.9141.2077.6977.9870.0969.26
50%Gate gradient48.8970.6288.2087.0077.9758.9642.0081.2881.5973.1670.97

OLMoE-1B-7B

SparsityMethodARC-cARC-eBoolQCOPAHellaSwagMMLUOBQAPIQARTEWinoGAvg.
25%EAN43.9463.9371.6881.0075.4333.4842.6074.7664.6265.7561.72
25%REAP44.9768.5273.1281.0076.8238.4344.4076.8860.6563.3862.82
25%Gate gradient42.1565.1173.0679.0075.6843.1642.2076.1276.1762.9863.56
50%EAN25.9438.2657.6868.0050.7223.0932.4061.9754.8755.0146.79
50%REAP31.3145.4159.7274.0058.8424.2933.8067.0857.0455.4950.70
50%Gate gradient31.0646.0057.0369.0050.3930.9131.6066.0058.8456.8349.77

Gradient scoring does not show a clear overall advantage over EAN and REAP on these AR models. Together with the masked-criteria ablations, this suggests that NAPE's gains depend on measuring expert importance with the diffusion loss across noise levels, rather than using gradients alone.


W2 & Q4. Expert-ranking stability across calibration sets

To distinguish the effects of noise level and calibration sampling, we compared expert rankings across three disjoint C4 calibration sets on LLaDA-MoE. The noise-gap columns compare fixed-ratio scores across sets; the final column compares NAPE's multi-noise scores across the same sets.

Noise gap ∣ri−rj∣\lvert r_i-r_j\rvert0.00.20.40.60.8NAPE across sets
Median Spearman ρ\rho0.9450.9160.8700.7930.6920.945
Mean differing retained experts (of 32)3.024.025.236.598.093.35

As the noise gap increases, ranking correlation falls steadily from 0.945 to 0.692, while the mean number of differing retained experts rises from 3.02 to 8.09. NAPE's aggregated rankings remain consistent across calibration sets, with a median correlation of 0.945 and 3.35 of the 32 retained experts differing on average. These results show that noise level affects expert importance beyond calibration sampling and support aggregating importance across denoising stages.

The downstream results in Tables 13–14 of our paper also support this choice. Although the best fixed ratio differs between LLaDA-MoE and SDAR (0.1 versus 0.5), NAPE achieves higher averages on both with the same noise-sampling rule. Figure 4(b) evaluates estimation of the noise-marginal reference; Tables 13–14 provide the separate evidence on pruning quality.

AuthorsOpenReview ↗

Response to Reviewer k75T (3/3)

Q5. Full multi-step generation

We agree that evaluating open-ended generation is important. Our paper includes XSum, an open-ended summarization task in which the models generate complete summaries through multi-step denoising. NAPE achieves the best performance among the compared pruning methods. The table below compares the newly added MoNE baseline and NAPE at 50% sparsity under the same C4 calibration and evaluation settings as Table 5 in our paper.

MethodLLaDA-MoE ROUGE-1SDAR ROUGE-1LLaDA-MoE ROUGE-LSDAR ROUGE-L
MoNE20.4119.8714.2013.86
NAPE21.3324.1515.0518.43

NAPE remains the best pruned method on ROUGE-1 and ROUGE-L for both backbones, showing that its advantage extends to full generation under general-domain calibration.

Beyond summarization, following your suggestion, we evaluate NAPE on GSM8K for mathematical reasoning and MBPP for code generation. On LLaDA-MoE at 25% sparsity, all C4-calibrated pruning methods show substantial degradation on MBPP (pass@1 is 67.70 for the full model, while most methods score 0.78-5.45). Therefore, we use task-specific training data for calibration. Specifically, we use 32 sequences from the GSM8K training set and 16 from the MBPP training set, each containing 1,024 tokens. We evaluate on the corresponding test sets at 25% expert sparsity using OpenCompass harness [1] for evaluation, reporting accuracy for GSM8K and pass@1 for MBPP.

MethodLLaDA-MoE GSM8KSDAR GSM8KLLaDA-MoE MBPPSDAR MBPP
Full Model76.6588.1767.7066.93
NAPE76.6589.2367.7066.15

NAPE matches the full model on both LLaDA-MoE tasks, with MBPP pass@1 improving from 0.78 under C4 calibration to 67.70. On SDAR, it improves GSM8K accuracy by 1.06 points and retains MBPP pass@1 within 0.8 points of the full model. These results support NAPE's ability to preserve mathematical reasoning and code generation at 25% sparsity with task-specific calibration.

References

[1] OpenCompass Contributors. OpenCompass: A Universal Evaluation Platform for Foundation Models. 2023.


Thank you again for the comments, which strengthened our empirical comparison and provided further insights into our method. We have included the new results in the revision, with the additions highlighted in blue. We hope our response resolves your concerns and would be very grateful if you could consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.