- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a dynamic expert allocation and rectification framework for efficient inference of MoE-based diffusion models. To start with, this paper proposes three insights about the diffusion inference, and then introduces three solutions correspondingly. It consists of the similarity-based rescale, local-error aware allocation, and the global-error timestep weighting. Experiments show its efficacy on accelerating the inference while maintaining the performance.
Critical strengths
-
The framework is training-free and easy to implement. It does not require extensive training, and the extra costs are negligible.
-
Comprehensive experimental evaluation. The experimental settings are comprehensive, which include multiple SoTA models and various metrics.
-
The experiments show the proposed method can achieves a 2x speed-up while maintaining the performance well.
Critical weaknesses
-
To some extent, the proposed three insight are straightforward and naïve. Some are well-known. For example, the sensitivity/importance of different timesteps and layers vary, and the early steps control semantics.
-
The proposed timestep weighting schedule is heuristic. Although the paper empirically shows early-heavy expert allocation can be beneficial, it does not justify why the noise-level is a faithful proxy for importance, and why the adopted linear weighting schedule is appropriate.
-
The similarity threshold is also heuristic. It uses the fixed threshold of 0.9, and lack relevant ablation study or hyper-parameter sensitivity analysis. For different tasks and models, how to choose an appropriate threshold?
-
The method considers both timestep and layer simultaneously, but did not prove that this joint allocation is better than the simpler time-only and layer-only allocation strategy.
Questions for authors
Please see weaknesses
2 comments
Response to Reviewer pXuP (Part: 2/2)
W3: The similarity threshold is also heuristic. It uses the fixed threshold of 0.9, and lack relevant ablation study or hyper-parameter sensitivity analysis. For different tasks and models, how to choose an appropriate threshold?
Reply: The similarity threshold of 0.9 was selected through grid search. The complete threshold comparison on Instruct is shown below, with all layers and timesteps fixed at Top-4 and with neither DP allocation nor timestep weighting.
| Model | Threshold τ | ImageReward | PickScore | AestheticScore | OCR | GenEval | CLIP |
|---|---|---|---|---|---|---|---|
| Instruct | 0 | 0.9785 | 22.1368 | 5.6027 | 0.6379 | 0.8320 | 28.4033 |
| 0.3 | 0.9812 | 22.1294 | 5.6061 | 0.6358 | 0.8334 | 28.3962 | |
| 0.6 | 0.9956 | 22.1583 | 5.6009 | 0.6517 | 0.8316 | 28.5186 | |
| 0.85 | 1.0509 | 22.2298 | 5.6049 | 0.7209 | 0.8471 | 29.0517 | |
| 0.9 | 1.0532 | 22.2324 | 5.6063 | 0.7225 | 0.8466 | 29.0580 | |
| 0.95 | 1.0507 | 22.2301 | 5.6038 | 0.7218 | 0.8458 | 29.0552 | |
| 1 | 1.0511 | 22.2296 | 5.6080 | 0.7208 | 0.8459 | 29.0524 |
Performance remains stable for thresholds near 0.9, indicating robustness to small variations in the threshold. For different models, the threshold can be selected through grid search on a validation set.
W4: The method considers both timestep and layer simultaneously, but did not prove that this joint allocation is better than the simpler time-only and layer-only allocation strategy.
Reply: Following this suggestion, we compare joint allocation with time-only and layer-only strategies under the same total expert budget. The two strategies are defined as follows:
- Time-only: Expert counts vary across timesteps in proportion to , while all layers at the same timestep share the same count.
- Layer-only: Expert counts vary across layers but remain fixed across timesteps; DP allocates the budget using time-averaged local errors .
The results are shown below:
| Model | Method | ImageReward | PickScore | AestheticScore | OCR | GenEval | CLIP |
|---|---|---|---|---|---|---|---|
| Distil | Time-only | 0.9692 | 21.9657 | 5.5315 | 0.7111 | 0.8690 | 28.2525 |
| Distil | Layer-only | 0.9360 | 21.9323 | 5.5691 | 0.7139 | 0.8493 | 28.0260 |
| Distil | ESEA | 0.9887 | 21.9920 | 5.5938 | 0.7013 | 0.8748 | 28.3670 |
| Instruct | Time-only | 1.0633 | 22.2363 | 5.5881 | 0.7407 | 0.8633 | 29.0802 |
| Instruct | Layer-only | 1.0498 | 22.2427 | 5.5966 | 0.7373 | 0.8512 | 29.2176 |
| Instruct | ESEA | 1.0973 | 22.3053 | 5.6267 | 0.7440 | 0.8670 | 29.1966 |
ESEA achieves the highest scores on five of six metrics for both models. Compared with time-only and layer-only allocation, ImageReward increases to 1.0973 from 1.0633 and 1.0498 on Instruct, and to 0.9887 from 0.9692 and 0.9360 on Distil. These comparisons support jointly considering timestep and layer sensitivity when distributing a fixed expert budget.
Response to Reviewer pXuP (Part: 1/2)
W1: To some extent, the proposed three insight are straightforward and naïve. Some are well-known. For example, the sensitivity/importance of different timesteps and layers vary, and the early steps control semantics.
Reply: We thank the reviewer for the feedback on the novelty of our insights. Our emphasis is on the concrete quantification and method design for diffusion MoE expert compression: characterizing the approximation error caused by routing-weight redistribution, measuring local errors across timesteps and layers and their consistency across prompts, and using these measurements to allocate experts under a fixed total budget. These analyses directly motivate similarity-based weight adjustment, the calibrated local-error tensor, and timestep-aware joint allocation, rather than merely restating that different layers and timesteps have different importance.
In particular, Insight 1 is derived from theoretical analysis. Proposition 4.1 shows that renormalizing retained routing weights is equivalent to redistributing the discarded weights among retained experts. Equation 6 then expresses the resulting approximation error as a weighted sum of differences between discarded and retained expert outputs. This derivation establishes the connection between expert-output similarity and approximation error and provides the analytical basis for our similarity-guided weight adjustment.
W2: The proposed timestep weighting schedule is heuristic. Although the paper empirically shows early-heavy expert allocation can be beneficial, it does not justify why the noise-level is a faithful proxy for importance, and why the adopted linear weighting schedule is appropriate.
Reply: We appreciate this comment and agree that the current noise-based weighting is an empirical prior. Below, we clarify its motivation and compare alternative weighting functions.
Noise level identifies the denoising stage in a way that is consistent with the sampler: high-noise steps primarily establish global structure, while low-noise steps refine details. Our budget-swapping experiment in Appendix B shows that the timing of expert reduction affects final quality, motivating a stage-dependent weighting of local errors.
Furthermore, we compare noise-aware weighting wt=1+σt with linear wt=1+ut, quadratic wt=1+ut2, and reciprocal wt=1+1/ut weighting. Here, ut=1−t/T is the normalized reverse step index, where t=0,…,T−1 follows denoising order, with T=50 for Instruct and T=8 for Distil.
Noise-aware weighting achieves the highest scores on five of six metrics for both models, with particularly clear advantages in ImageReward,OCR and GenEval, suggesting that weighting consistent with the sampler’s noise schedule better captures the timestep-dependent impact of local errors on global error.