acceptodds
Under review as a conference paper at ICLR 2027

ESEA: Error-Aware Sparse Expert Allocation for Efficient Diffusion MoE Inference

Abstract

Mixture-of-Experts (MoE) has emerged as a promising paradigm for scaling diffusion models, yet activating a fixed number of experts for all tokens leaves computational redundancy. Existing expert skipping methods mainly target LLMs and overlook the iterative denoising nature of diffusion models, leading to degraded generation quality when directly applied to diffusion moe models. We analyze the underlying error mechanisms and identify three insights: (i) approximation error depends on expert similarity; (ii) local errors exhibit timestep–layer heterogeneity and cross-prompt consistency; (iii) the impacts of local errors on global error vary across timesteps. Based on these insights, we propose ESEA (Error-aware Sparse Expert Allocation), a training-free method for diffusion MoE inference. ESEA first adjusts routing weights based on expert similarity to reduce approximation error, then weights calibrated 3D local errors tensor to reflect their timestep-dependent impacts on global error. Finally, dynamic programming minimizes the weighted local error, jointly allocating experts across timesteps and layers under a fixed total expert budget. The routing weight adjustment strategy and expert allocation are determined offline, introducing negligible overhead during inference. Extensive experiments on HunyuanImage-3.0-Instruct, its distilled variant, and LingBot-Video demonstrate that ESEA achieves the best performance among the compared methods under the same expert budget.

Primary area:
generative models
Keywords:
Diffusion Models, Mixture-of-Experts, Dynamic Expert Skipping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pXuPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dynamic expert allocation and rectification framework for efficient inference of MoE-based diffusion models. To start with, this paper proposes three insights about the diffusion inference, and then introduces three solutions correspondingly. It consists of the similarity-based rescale, local-error aware allocation, and the global-error timestep weighting. Experiments show its efficacy on accelerating the inference while maintaining the performance.

Critical strengths
  1. The framework is training-free and easy to implement. It does not require extensive training, and the extra costs are negligible.

  2. Comprehensive experimental evaluation. The experimental settings are comprehensive, which include multiple SoTA models and various metrics.

  3. The experiments show the proposed method can achieves a 2x speed-up while maintaining the performance well.

Critical weaknesses
  1. To some extent, the proposed three insight are straightforward and naïve. Some are well-known. For example, the sensitivity/importance of different timesteps and layers vary, and the early steps control semantics.

  2. The proposed timestep weighting schedule is heuristic. Although the paper empirically shows early-heavy expert allocation can be beneficial, it does not justify why the noise-level is a faithful proxy for importance, and why the adopted linear weighting schedule is appropriate.

  3. The similarity threshold is also heuristic. It uses the fixed threshold of 0.9, and lack relevant ablation study or hyper-parameter sensitivity analysis. For different tasks and models, how to choose an appropriate threshold?

  4. The method considers both timestep and layer simultaneously, but did not prove that this joint allocation is better than the simpler time-only and layer-only allocation strategy.

Questions for authors

Please see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer pXuP (Part: 1/2)

W1: To some extent, the proposed three insight are straightforward and naïve. Some are well-known. For example, the sensitivity/importance of different timesteps and layers vary, and the early steps control semantics.

Reply: We thank the reviewer for the feedback on the novelty of our insights. Our emphasis is on the concrete quantification and method design for diffusion MoE expert compression: characterizing the approximation error caused by routing-weight redistribution, measuring local errors across timesteps and layers and their consistency across prompts, and using these measurements to allocate experts under a fixed total budget. These analyses directly motivate similarity-based weight adjustment, the calibrated local-error tensor, and timestep-aware joint allocation, rather than merely restating that different layers and timesteps have different importance.

In particular, Insight 1 is derived from theoretical analysis. Proposition 4.1 shows that renormalizing retained routing weights is equivalent to redistributing the discarded weights among retained experts. Equation 6 then expresses the resulting approximation error as a weighted sum of differences between discarded and retained expert outputs. This derivation establishes the connection between expert-output similarity and approximation error and provides the analytical basis for our similarity-guided weight adjustment.

W2: The proposed timestep weighting schedule is heuristic. Although the paper empirically shows early-heavy expert allocation can be beneficial, it does not justify why the noise-level is a faithful proxy for importance, and why the adopted linear weighting schedule is appropriate.

Reply: We appreciate this comment and agree that the current noise-based weighting is an empirical prior. Below, we clarify its motivation and compare alternative weighting functions.

Noise level identifies the denoising stage in a way that is consistent with the sampler: high-noise steps primarily establish global structure, while low-noise steps refine details. Our budget-swapping experiment in Appendix B shows that the timing of expert reduction affects final quality, motivating a stage-dependent weighting of local errors.

Furthermore, we compare noise-aware weighting wt=1+σtw_t=1+\sigma_t with linear wt=1+utw_t=1+u_t, quadratic wt=1+ut2w_t=1+u_t^2, and reciprocal wt=1+1/utw_t=1+1/u_t weighting. Here, ut=1−t/Tu_t=1-t/T is the normalized reverse step index, where t=0,…,T−1t=0,\ldots,T-1 follows denoising order, with T=50T=50 for Instruct and T=8T=8 for Distil.

ModelMethodImageRewardPickScoreAestheticScoreOCRGenEvalCLIP
InstructESEA1.097322.30535.62670.74400.867029.1966
InstructLinear1.072622.19005.50900.54670.537829.0922
InstructQuadratic1.043522.21035.53260.54770.570729.2384
InstructReciprocal0.987922.19615.61360.64830.836728.4430
DistilESEA0.988721.99205.59380.70130.874828.3670
DistilLinear0.962521.86185.47890.51130.588928.4076
DistilQuadratic0.972221.88725.47900.49090.589628.3769
DistilReciprocal0.850421.86395.56600.57130.827227.6949

Noise-aware weighting achieves the highest scores on five of six metrics for both models, with particularly clear advantages in ImageReward,OCR and GenEval, suggesting that weighting consistent with the sampler’s noise schedule better captures the timestep-dependent impact of local errors on global error.

AuthorsOpenReview ↗

Response to Reviewer pXuP (Part: 2/2)

W3: The similarity threshold is also heuristic. It uses the fixed threshold of 0.9, and lack relevant ablation study or hyper-parameter sensitivity analysis. For different tasks and models, how to choose an appropriate threshold?

Reply: The similarity threshold of 0.9 was selected through grid search. The complete threshold comparison on Instruct is shown below, with all layers and timesteps fixed at Top-4 and with neither DP allocation nor timestep weighting.

ModelThreshold τImageRewardPickScoreAestheticScoreOCRGenEvalCLIP
Instruct00.978522.13685.60270.63790.832028.4033
0.30.981222.12945.60610.63580.833428.3962
0.60.995622.15835.60090.65170.831628.5186
0.851.050922.22985.60490.72090.847129.0517
0.91.053222.23245.60630.72250.846629.0580
0.951.050722.23015.60380.72180.845829.0552
11.051122.22965.60800.72080.845929.0524

Performance remains stable for thresholds near 0.9, indicating robustness to small variations in the threshold. For different models, the threshold can be selected through grid search on a validation set.

W4: The method considers both timestep and layer simultaneously, but did not prove that this joint allocation is better than the simpler time-only and layer-only allocation strategy.

Reply: Following this suggestion, we compare joint allocation with time-only and layer-only strategies under the same total expert budget. The two strategies are defined as follows:

  • Time-only: Expert counts vary across timesteps in proportion to wt=1+σtw_t=1+\sigma_t, while all layers at the same timestep share the same count.
  • Layer-only: Expert counts vary across layers but remain fixed across timesteps; DP allocates the budget using time-averaged local errors Cl(k)=1T∑tEt,l,kC_l(k)=\frac{1}{T}\sum_t E_{t,l,k}.

The results are shown below:

ModelMethodImageRewardPickScoreAestheticScoreOCRGenEvalCLIP
DistilTime-only0.969221.96575.53150.71110.869028.2525
DistilLayer-only0.936021.93235.56910.71390.849328.0260
DistilESEA0.988721.99205.59380.70130.874828.3670
InstructTime-only1.063322.23635.58810.74070.863329.0802
InstructLayer-only1.049822.24275.59660.73730.851229.2176
InstructESEA1.097322.30535.62670.74400.867029.1966

ESEA achieves the highest scores on five of six metrics for both models. Compared with time-only and layer-only allocation, ImageReward increases to 1.0973 from 1.0633 and 1.0498 on Instruct, and to 0.9887 from 0.9692 and 0.9360 on Distil. These comparisons support jointly considering timestep and layer sensitivity when distributing a fixed expert budget.

Reviewer_1NtjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ESEA accelerates diffusion MoE models through routing-weight adjustment and expert allocation across layers and denoising steps. It calibrates approximation errors offline and uses dynamic programming to allocate a fixed expert budget. Experiments show improved generation quality over uniform expert reduction at similar speed.

Critical strengths
  • Clear presentation: The paper is well organized, clearly written, and easy to follow.
  • Focused contribution: The motivation, method, and experiments address one coherent problem.
  • Useful empirical results: Experiments across image and video models demonstrate a promising quality–speed trade-off, supported by component ablations.
  • Intuitive recursive formulation: The dynamic-programming recurrence turns a large expert-allocation problem into manageable subproblems. It provides a clear, systematic way to balance measured approximation error against a fixed computation budget across layers and denoising steps.
Critical weaknesses
  • Limited gains from Components 2 and 3: Local-error allocation and timestep weighting are well motivated, but their additional gains are relatively small, compared with routing-weight adjustment. Their practical contribution needs clearer justification.
  • Missing magnitude analysis: The cosine-based allocation metric measures output direction while ignoring magnitude. Since routing-weight rescaling changes magnitude, the error analysis leaves an important aspect unexplored.
  • Missing discussion of Alloc-MoE: Alloc-MoE already uses sensitivity profiling and dynamic programming for expert-budget allocation. The paper should explain its relationship to this closely related method and clarify the additional contribution.
Questions for authors
  • Would a magnitude-sensitive metric, such as normalized MSE, improve the allocation or better explain the observed quality gains?
  • How does ESEA differ from Alloc-MoE beyond extending allocation across diffusion timesteps?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1Ntj (Part: 1/2)

W1: Limited gains from Components 2 and 3: Local-error allocation and timestep weighting are well motivated, but their additional gains are relatively small, compared with routing-weight adjustment. Their practical contribution needs clearer justification.

Reply: We appreciate the suggestion to clarify the practical contribution of Components 2 and 3. The table below reports the incremental ablation on Instruct at an average Top-k of 4, with the original Top-8 model as a reference.

MethodImageRewardPickScoreAestheticScoreOCRGenEvalCLIP
Original model (Top-8)1.111422.37865.69990.79320.861929.5973
Similarity-based weight adjustment1.053222.23245.60630.72250.846629.0580
+ Local-error-based DP allocation1.089522.29715.61670.74110.870229.1875
+ Timestep weighting1.097322.30535.62670.74400.867029.1966
  • Each component improves most benchmark scores. Together, Components 2 and 3 raise ImageReward from 1.0532 to 1.0973 and OCR from 0.7225 to 0.7440 under the same expert budget.
  • The relatively small numerical gains reflect the limited room for further quality recovery as performance approaches the original Top-8 baseline. For example, ESEA achieves an ImageReward of 1.0973 and a PickScore of 22.3053, compared with 1.1114 and 22.3786 for Top-8.
  • The benefits are also visible in the generated images, beyond the aggregate benchmark scores. Qualitative comparisons are available here: https://anonymous.4open.science/r/ICLR2027-ESEA-64DD/component_ablation_qualitative.pdf. For example, for the prompt “An orange colored sandwich,” Component 1 alone generates an orange fruit and misses the sandwich structure, whereas the full method produces a sandwich with visible fillings between orange-colored halves. This illustrates how Components 2 and 3 can improve compositional fidelity even when the average score gains are modest.

Q1&W2: Would a magnitude-sensitive metric, such as normalized MSE, improve the allocation or better explain the observed quality gains?

Reply: We thank the reviewer for suggesting a magnitude-sensitive metric. We additionally evaluated NMSE to examine both its relationship with cosine error and its effect on expert allocation.

1. Correlation between cosine error and NMSE

Using identical layer inputs, we measure the error of the reduced routed-MoE output relative to the original Top-8 output:

NMSE=∥yk−y8∥22∥y8∥22.\mathrm{NMSE}=\frac{\lVert y_k-y_8\rVert_2^2}{\lVert y_8\rVert_2^2}.

For each Top-k from 1 to 7, we compute the Pearson correlation between NMSE and cosine error.

ModelTop-1Top-2Top-3Top-4Top-5Top-6Top-7AVG
Instruct0.99550.99730.99770.99790.99820.99880.99950.9978
Distil0.99460.99690.99740.99780.99820.99880.99950.9976

The average correlations are 0.9978 on Instruct and 0.9976 on Distil. Although cosine error does not explicitly measure output magnitude, it closely tracks magnitude-sensitive NMSE .

2. Generation quality with NMSE-based allocation

We further replace cosine error with NMSE in the allocation objective and evaluate the resulting DP allocation.

ModelMethodImageRewardPickScoreAestheticScoreOCRGenEvalCLIP
InstructESEA1.097322.30535.62670.74400.867029.1966
InstructNMSE1.085322.29155.60940.74610.866229.1639
DistilESEA0.988721.99205.59380.70130.874828.3670
DistilNMSE0.984921.98865.58990.70100.875628.3577

NMSE-based allocation does not consistently improve generation quality: cosine-based ESEA achieves higher scores on five of the six metrics for each model. Together with the strong error correlations, these results support cosine error as an effective allocation metric .

AuthorsOpenReview ↗

Response to Reviewer 1Ntj (Part: 2/2)

Q2&W3: How does ESEA differ from Alloc-MoE beyond extending allocation across diffusion timesteps?

Reply: We thank the reviewer for highlighting Alloc-MoE, a closely related work that uses sensitivity profiling and DP for expert-budget allocation. ESEA’s contribution lies in its diffusion-specific error modeling and weight adjustment, with the following differences:

  • Calibration signal: Alloc-L profiles layer sensitivity using end-to-end text perplexity. This metric is not directly available for diffusion image generation. ESEA instead measures local MoE output errors relative to the original model on identical layer inputs, constructing a timestep–layer–Top-k error tensor.
  • Allocation objective: Alloc-MoE allocates budgets across layers and redistributes activations across tokens using routing scores. ESEA jointly allocates experts across denoising timesteps and layers to better align with the multi-step iterative denoising process of diffusion models.
  • Weight adjustment: ESEA additionally analyzes the approximation error induced by redistributing discarded routing weights and uses expert similarity to decide whether to rescale the retained weights. This provides a compensation mechanism for discarded expert contributions that is absent from Alloc-MoE.

For a direct comparison, we adapted Alloc-Moe to HunyuanImage-3.0-Instruct and Distil. We replaced text perplexity in sensitivity profiling with final denoised-latent NMSE. The results are reported below.

ModelMethodImageRewardPickScoreAestheticScoreOCRGenEvalCLIP
InstructAlloc-MoE1.036022.11215.60680.63710.855828.7403
InstructESEA1.097322.30535.62670.74400.867029.1966
DistilAlloc-MoE0.901721.75975.55800.64260.875527.9648
DistilESEA0.988721.99205.59380.70130.874828.3670

ESEA achieves higher scores on all six metrics on Instruct and five of six on Distil. ImageReward increases from 1.0360 to 1.0973 on Instruct and from 0.9017 to 0.9887 on Distil; OCR increases from 0.6371 to 0.7440 and from 0.6426 to 0.7013, respectively. These results support the benefit of ESEA’s diffusion-specific design.

Reviewer_j49gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to reduce expert computation while preserving generation quality during the inference of diffusion MoE models. Based on the similarity of expert outputs, the authors determine whether to renormalize routing weights after pruning experts. Through offline calibration, they quantify local errors corresponding to different denoising steps, network layers and expert counts. Combined with noise-aware temporal weights, dynamic programming is adopted to allocate a fixed expert budget. Precomputed configurations are directly applied at inference time without retraining. Experiments are conducted on Hunyuan image model, its distilled variant and LingBot-Video. Under the same average expert budget, the overall generation quality outperforms fixed Top-k and two expert-skip baseline methods.

Critical strengths
  1. The idea of allocating the expert budget to both denoising steps and network layers simultaneously is interesting. The authors also provide the rationale for adopting an offline configuration through experiments on error distribution and cross-prompt correlation, and the method design is coherent.
  2. The quality gain under the same expert budget is notable. The ImageReward of the non-distilled Hunyuan model increases from 0.9785 with fixed Top-4 to 1.0973, which is close to 1.1114 of the original Top-8; gains are also observed on the distilled model and video model. Experiments with different budgets and incremental module addition further validate the practicality of the method.
Critical weaknesses
  1. There is no control group for the setting of "retaining original weights and skipping renormalization after experts are pruned", so it remains unclear how much additional benefit similarity judgment can bring compared with this simple scheme.
  2. The timing range for the speedup ratio is not clearly specified, and the absolute generation latency as well as the offline calibration cost are missing.
Questions for authors
  1. Could you add the control group of "no re-normalization at all" to compare generation quality and local errors?
  2. Could you specify the timing range for the speedup ratio, and provide the actual generation time as well as offline calibration time of the original model and ESEA?
2 comments
AuthorsOpenReview ↗

Response to Reviewer j49g (Part: 1/2)

Q1&W1: Could you add the control group of "no re-normalization at all" to compare generation quality and local errors?

Reply: We thank the reviewer for pointing out this missing control. For generation quality, we added a Top-4 control group that retains the original Top-8 routing weights without renormalization. The table below compares it with the original model, renormalized Top-4, and ESEA at an average Top-k of 4.

MethodImageRewardPickScoreAestheticScoreOCRGenEvalCLIP
(a) HunyuanImage-3.0-Instruct
Original model (Top-8)1.111422.37865.69990.79320.861929.5973
Top-4 (with renormalization)0.978522.13685.60270.63790.832028.4033
Top-4 (no renormalization)0.994722.16235.60410.65860.836528.5579
ESEA (average Top-k = 4)1.097322.30535.62670.74400.867029.1966
(b) HunyuanImage-3.0-Instruct-Distil
Original model (Top-8)1.024822.12055.67070.74500.888928.5997
Top-4 (with renormalization)0.843821.83495.55650.56320.819727.5620
Top-4 (no renormalization)0.867121.85625.55120.59840.831627.7018
ESEA (average Top-k = 4)0.988721.99205.59380.70130.874828.3670

ESEA consistently outperforms both Top-4 baselines, with and without renormalization, across all six metrics on both models. For example, on Instruct, ESEA achieves an ImageReward of 1.0973, compared with 0.9785 and 0.9947 for the two baselines; on Distil, the corresponding scores are 0.9887, 0.8438, and 0.8671. These results demonstrate that ESEA maintains a clear quality advantage over both baselines under the same average expert budget.

For local error, we additionally profiled the output MSE relative to the original Top-8 model. ESEA also achieves lower local errors than both Top-4 baselines, with and without renormalization, across the sampled denoising steps, with more pronounced reductions in deeper layers. The detailed results are available here: https://anonymous.4open.science/r/ICLR2027-ESEA-64DD/local-erroe.pdf

AuthorsOpenReview ↗

Response to Reviewer j49g (Part: 2/2)

Q2&W2: Could you specify the timing range for the speedup ratio, and provide the actual generation time as well as offline calibration time of the original model and ESEA?

Reply: Thank you for raising this point. We clarify the timing scope and provide the absolute generation latency and offline calibration costs below.

1. Timing scope

The speedup ratios in the paper measure MoE-module latency, including routing, expert computation, output aggregation, and the overhead of each skipping strategy, under the same hardware and inference settings.

2. Actual generation time

We additionally report end-to-end generation latency on 8 NVIDIA A800-SXM4-40GB GPUs at 1024 × 1024 resolution, using seeds 42, 43, and 44, with 10 warm-up runs followed by 50 generations.

Hunyuan modelMethodEnd to End Latency (s)MoE Latency (s)End to End Speedup
InstructOriginal Top-882.19550.1461.000×
InstructESEA (Top-k=4)59.24027.9181.387×
DistilOriginal Top-87.3344.5681.000×
DistilESEA (Top-k=4)5.2902.5631.386×

ESEA reduces end-to-end latency from 82.195 s to 59.240 s on Instruct and from 7.334 s to 5.290 s on Distil, yielding 1.387× and 1.386× speedups, respectively.

3. Offline calibration cost

All methods use the same 10 calibration prompts per model. NAEE collects routing statistics and uses genetic search to determine layer-specific skipping thresholds; MoDES estimates layer importance and searches for a skipping threshold that meets the target expert budget; ESEA computes expert similarities and local errors.

  • NAEE: 10 reference generations plus 38 threshold configurations evaluated on 10 prompts each, totaling 390 generations.
  • MoDES: 10 reference generations, 32 layer ablations on 10 prompts each, and 12 threshold evaluations on 10 prompts each, totaling 450 generations.
  • ESEA: 10 similarity-profiling trajectories and 10 local-error-profiling trajectories. Local-error profiling evaluates Top-1 through Top-8 within each trajectory.
MethodInstruct calibration time (h)Distil calibration time (h)
NAEE9.7841.177
MoDES11.8721.384
ESEA2.6130.277

ESEA achieves offline calibration speedups of 3.74× over NAEE and 4.54× over MoDES on Instruct, and 4.25× and 5.00×, respectively, on Distil.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We appreciate the reviewers’ time and constructive feedback, and we have provided detailed responses and additional experiments addressing the concerns raised; we summarize the main points below to facilitate further discussion.

Reviewer j49g

  • No-renormalization control: We added the requested control and local-error profiling, showing that ESEA maintains better generation quality than both renormalized and non-renormalized Top-4 baselines under the same average expert budget.
  • Inference and calibration costs: We clarified the MoE timing scope and reported absolute generation latency and offline calibration costs, with approximately 1.39× end-to-end acceleration and 3.74×–5.00× faster calibration than the adapted NAEE and MoDES baselines.

Reviewer 1Ntj

  • Contributions of Components 2 and 3: We provided incremental ablations and qualitative comparisons showing improvements in most benchmark scores and compositional fidelity under the same expert budget.
  • Magnitude-sensitive error analysis: We evaluated NMSE, found strong correlations with cosine error, and showed that replacing cosine error with NMSE does not consistently improve allocation quality.
  • Relationship to Alloc-MoE: We clarified the differences in calibration signals, timestep–layer allocation, and weight compensation, and added an adapted Alloc-MoE comparison in which ESEA performs better on most metrics.

Reviewer pXuP

  • Novelty of the insights: We clarified that our contribution lies in the concrete quantification and method design for diffusion MoE expert compression, with Insight 1 derived through the theoretical analysis in Proposition 4.1 and Equation 6.
  • Noise-level weighting: We clarified its role as an empirical prior and compared it with linear, quadratic, and reciprocal weighting, with noise-aware weighting achieving the highest scores on five of six metrics for both models.
  • Similarity threshold: We provided the threshold grid-search results, demonstrating stable performance near 0.9 and explaining how the threshold can be selected for different models using a validation set.
  • Joint allocation: We added time-only and layer-only comparisons under the same total expert budget, with joint allocation achieving the highest scores on five of six metrics for both models.

We would welcome further feedback from the reviewers and are happy to clarify any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.