acceptodds
Under review as a conference paper at ICLR 2027

E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models

Abstract

Masked diffusion models (MDMs) generate sequences by progressively unmasking several tokens per denoising step, but their reverse process is typically factorized over positions, limiting sample quality in the few-step regime where diffusion's speed advantage over autoregressive decoding matters most. A recent line of work introduces a continuous Gaussian latent, trained as a variational autoencoder, to capture correlations across positions, but such approaches are prone to posterior collapse, where the latent is silently ignored. We propose **Enhanced Mixture-of-Experts (E-MoE)**, which builds the reverse process as a mixture of factorized distributions over a *discrete shared latent* given by the expert-routing decisions of a mixture-of-experts backbone, without increasing the number of active parameters relative to the factorized baseline. On synthetic multi-modal benchmarks, binarized MNIST and LM1B, E-MoE substantially improves few-step generation over factorized MDM baselines.

Primary area:
generative models
Keywords:
Discrete diffusion language models, Masked diffusion models, Mixture-of-experts, Non-factorized generation, Few-step generation, Latent variable models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mLcQOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes E-MoE, a non-factorized masked diffusion language model that uses MoE routing decisions as discrete shared latent variables to capture dependencies among tokens generated within the same denoising step. The method avoids additional inference cost while substantially improving few-step generation. Experiments on synthetic data, Binarized-MNIST, and LM1B consistently demonstrate the effectiveness of the proposed approach.

Critical strengths

The paper identifies an important limitation of masked diffusion models—the factorization error when multiple tokens are generated simultaneously—and proposes a simple and well-motivated solution by reusing MoE routing decisions as discrete shared latent variables.

The empirical results are strong and convincing, particularly in the few-step regime where the proposed method substantially outperforms both factorized MDMs and the continuous-latent baseline without additional inference cost. The ablation on routing further provides clear evidence that the improvement comes from the proposed mixture formulation.

Critical weaknesses

It would be useful to understand whether the advantage of E-MoE also transfers to conditional generation and downstream language modeling tasks.

Questions for authors

How sensitive is E-MoE to the number of experts and the routing strategy? In particular, is the improvement mainly due to having a discrete shared latent, or does it depend strongly on the specific MoE configuration?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mLcQ (1/2)

Dear Reviewer mLcQ, thank you for your thoughtful and positive assessment. We are glad that you found the idea simple and well motivated and the results convincing. Your comments raise two important directions: conditional generation and the sensitivity of E-MoE to its MoE configuration. We address them below. Throughout, we also compare with MDLM-MoE, which has the same backbone, experts and training budget as E-MoE, but deterministic routing and no latent.

[W.1]It would be useful to understand whether the advantage of E-MoE also transfers to conditional generation and downstream language modeling tasks.

Conditional generation. Let x0=(x01,…,x0L)\mathbf{x}_0 = (\mathbf{x}_0^1, \dots, \mathbf{x}_0^L) be a sequence from the LM1B test set with L=128L = 128. We use its first P=32P = 32 tokens as the prompt and start the reverse process from

xtT=(x01,…,x0P⏟prefix prompt,m,…,m⏟L−P masks)\mathbf{x}_{t_T} = (\underbrace{\mathbf{x}_0^1, \dots, \mathbf{x}_0^{P}}_{\text{prefix prompt}}, \underbrace{\mathbf{m}, \dots, \mathbf{m}}_{L-P \text{ masks}})

instead of the fully masked sequence of Algorithm 2 in the paper, with all other steps unchanged. Unmasked tokens are copied by the reverse transition (Eq. (2) in the paper), so the prompt stays fixed and the model generates the continuation x^P+1:L\hat{\mathbf{x}}^{P+1:L} conditioned on it. Masked diffusion models are trained to fill in arbitrary subsets of masked positions, so none of the compared models requires retraining. The setup is the same as for unconditional generation: we use 2000 test sequences, and all models receive the same prompts and the same sampling seed. We report the Gen-PPL of the continuation given the prompt, the entropy HH of the generated tokens, and MAUVE between the generated and the true continuations (Table 1).

Table 1: Conditional generation on LM1B. We use the first 32 tokens as a prefix prompt and generate 96 tokens. Bold: best per row within each metric.

Gen-PPL (↓) / HH│MAUVE (↑, %)
NFEMDLMMDLM-MoEVADDE-MoE│MDLMMDLM-MoEVADDE-MoE
11006.0 / 4.14975.7 / 4.14860.1 / 4.13455.3 / 4.15│0.48 ± 0.010.47 ± 0.000.51 ± 0.021.06 ± 0.08
2586.0 / 4.14563.8 / 4.14516.5 / 4.14269.2 / 4.15│0.83 ± 0.050.89 ± 0.061.17 ± 0.097.14 ± 0.37
4282.2 / 4.13274.6 / 4.13261.0 / 4.14169.4 / 4.14│5.80 ± 0.456.59 ± 0.428.41 ± 0.4233.56 ± 1.33
8162.1 / 4.13155.6 / 4.13163.5 / 4.14129.2 / 4.14│33.86 ± 1.9434.38 ± 1.0334.52 ± 1.9956.65 ± 0.64
16117.8 / 4.13111.9 / 4.13125.9 / 4.13112.0 / 4.14│66.56 ± 1.4170.55 ± 0.9258.07 ± 1.0670.87 ± 1.90

The few-step advantage of E-MoE carries over to conditional generation (Table 1). E-MoE targets the few-step regime, where many tokens are revealed at once and the factorization barrier is strongest. This is where the gain is largest: at NFE ≤4\le 4, E-MoE lowers the Gen-PPL of the continuation by 1.51.5–2.2×2.2\times relative to all baselines and raises MAUVE by 22–9×9\times, at no loss of entropy. MDLM-MoE stays close to MDLM, as in unconditional generation. At NFE =16= 16, E-MoE still attains the highest MAUVE and matches the best Gen-PPL, so the latent brings its gain in the few-step regime and costs nothing in quality elsewhere. Moreover, E-MoE uses the prompt not only in the denoiser but also in the latent: its prior pθ(z∣xt)p_\theta(\mathbf{z} \mid \mathbf{x}_t) is computed from the noisy sequence, which contains the prompt, so the route is sampled conditionally on the context. In contrast, the latent of VADD is drawn from N(0,I)\mathcal{N}(0, I) independently of the prompt. The discrete latent of E-MoE is therefore conditioned on the context by construction, which makes it naturally suited to conditional generation without any change to the model. We have added these results to the revised paper (Section 5.3, Table 6).

Downstream tasks. Following prior work on discrete diffusion at this scale [0, 1], every model in our comparison is a full pretraining run from scratch on LM1B, with the same budget for all methods, which already requires substantial compute. E-MoE uses 139M active parameters, as MDLM. Standard downstream benchmarks target models trained on much larger corpora and are not informative at this scale for any of the compared models. Scaling E-MoE to such regimes is a next step that we leave for future work.

AuthorsOpenReview ↗

Response to Reviewer mLcQ (2/2)

[Q.1] How sensitive is E-MoE to the number of experts and the routing strategy? In particular, is the improvement mainly due to having a discrete shared latent, or does it depend strongly on the specific MoE configuration?

We answer this question with controls that separate the latent from the MoE architecture and with the expert usage of the trained model.

1) Where the improvement comes from. (1) MDLM-MoE has exactly the MoE configuration of E-MoE (backbone, experts, top-1 dispatch and training budget), but deterministic routing and no latent. It stays close to MDLM in the few-step regime, both in unconditional generation (see our response to Reviewer eAT7, [W.2 / Q.2]) and in the conditional setting above. The MoE architecture and its extra parameters alone therefore do not remove the factorization error. (2) The greedy-routing ablation in Table 9 of the submission keeps the trained E-MoE weights and only replaces the sampled route by arg⁡max⁡\arg\max, which collapses the mixture to a single factorized component. With the same configuration, and even with the same weights, the gain disappears once the route is no longer a sampled latent. The improvement therefore comes from the discrete shared latent, i.e., from the mixture of factorized distributions.

2) Sensitivity to the MoE configuration. E-MoE does not rely on a specific number of experts or routing scheme.

Number of experts. The number of experts KK sets how many choices the route has in each layer, but the model does not have to use all of them. The KL term charges the model for every routing decision that the prior cannot predict, so the model itself selects how many components it uses. KK therefore acts as an upper limit on the capacity of the latent rather than as a hyperparameter to tune. To measure how much of this capacity carries information about the data, we compute the mutual information between the data and the latent codes zℓ=(z1ℓ,…,zDℓ)z^\ell = (z^\ell_1, \dots, z^\ell_D) of a position ℓ\ell across all DD layers,

I(x0;zℓ)=H(qˉℓ)−Epdata(x0)H(qθ(zℓ∣x0)),qˉℓ(zℓ)=Epdata(x0)qθ(zℓ∣x0),I(x_0; z^\ell) = H(\bar q^\ell) - \mathbb{E}_{p_{\text{data}}(x_0)} H\big(q_\theta(z^\ell \mid x_0)\big), \qquad \bar q^\ell(z^\ell) = \mathbb{E}_{p_{\text{data}}(x_0)} q_\theta(z^\ell \mid x_0),

averaged over positions ℓ\ell. It is zero if the routes ignore x0x_0, and it grows as the routes become more diverse across sequences and more certain for a given sequence. With K=8K = 8, on 2000 LM1B test sequences, I(x0;zℓ)=1.69I(x_0; z^\ell) = 1.69 nats per token, so the latent carries substantial information about the data.

Routing strategy. E-MoE uses the standard top-1 routing of sparse MoE. What matters is only whether the route is sampled or deterministic, and part 1 shows that the gain comes from sampling the route as a latent.

To summarize, the improvement is due to the discrete shared latent and does not depend strongly on the specific MoE configuration.


References

[0] Sahoo et al. Simple and Effective Masked Diffusion Language Models. NeurIPS, 2024.
[1] Xie et al. Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling. ICLR, 2026.


Thank you,

The Authors

Thanks for the detailed response. I have no further questions.

Reviewer_C31xOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the factorization error in masked diffusion models, where assuming independent token distributions across simultaneously unmasked positions degrades generation quality in the few-step regime. While prior work like VADD uses continuous Gaussian latents, this paper proposes E-MoE. The key idea is to use the expert-routing decisions of a MoE backbone as a discrete shared latent. By letting the router on clean data serve as the posterior and the router on noisy data serve as the prior, the model aligns routing distributions via categorical KL under a derived ELBO bound. Crucially, this requires zero extra active parameters and preserves the standard 1-forward-pass per sampling step. Evaluated on 2D synthetic manifolds, binarized MNIST, and LM1B, E-MoE cuts generative perplexity by over 2 at NFE=1 and 2, and boosts MAUVE by 8 at NFE=2 compared to factorized baselines.

Critical strengths

1. Clever, parameter-free latent formulation. Reusing an MoE’s native routing decisions as a discrete latent variable is conceptually elegant. It circumvents the posterior collapse and fixed-prior limitations of continuous VAE approaches (like VADD) without introducing any auxiliary networks.

2. Substantial few-step improvements with zero inference penalty. The method retains the same active parameter count and sampling latency (1 forward pass per NFE) as the factorized baseline, while achieving massive gains in the few-step regime where diffusion speed matters most. The toy 2D visualizations also provide crystal-clear intuition for why the method works.

Critical weaknesses

1. Limited experimental scale and sequence length. The text experiments are restricted to LM1B with a short sequence length of 128 on a small DiT. It remains unclear whether the discrete routing alignment scales effectively to larger backbones, longer sequence horizons, or conditional generation / downstream benchmarks.

2. Unfair comparison regarding total parameter footprint. While active parameter counts are strictly matched, an MoE backbone inherently has a larger total parameter count and memory footprint than a dense MDLM. The paper lacks a comparison against a dense baseline matched for total parameter capacity or equivalent training FLOPs.

Questions for authors
  1. How does E-MoE perform against a dense MDLM scaled up to match the total parameter count (or total training FLOPs) of the MoE model, rather than just the active parameters?

  2. Does the routing alignment become more difficult as sequence length scales up (e.g., 512 or 1024 tokens)? Have you observed any routing degradation or mode collapse on longer sequences?

  3. Did you encounter any expert load-imbalance issues during training, and was any auxiliary routing loss required to prevent only a few experts from dominating?

4 comments
AuthorsOpenReview ↗

Response to Reviewer C31x (1/3)

Dear Reviewer C31x, thank you for your thoughtful review and positive assessment. We are glad that you found the latent formulation elegant and the few-step gains substantial at zero inference cost. Below we address each of your questions.

W.1 / Q.2 Limited experimental scale and sequence length. Experiments use LM1B, sequence length 128, and a small DiT. It remains unclear whether routing alignment scales to larger models, longer sequences, or conditional generation / downstream benchmarks.

Model scale and sequence length. LM1B with sequence length 128 and a small DiT is a standard benchmark for discrete diffusion language models [MDLM [0], SEDD [1], BD3-LM [2], DUO [3], VADD [4]], enabling direct comparison with published results, and every model in our comparison is a full pretraining run on it. E-MoE is designed to scale to larger backbones: it introduces no additional modules and reuses the routing mechanism of MoE architectures, which are already used in large diffusion models such as LLaDA2.0 / LLaDA-MoE [5, 6]. Longer sequences also pose no architectural difficulty: each token has its own route, chosen per layer from its hidden state, exactly as in a standard MoE, so the prior and posterior routers face the same task at any sequence length.

Conditional generation. To address the remaining part of this question, we additionally evaluated E-MoE in a conditional setting. Let x0=(x01,…,x0L)\mathbf{x}_0 = (\mathbf{x}_0^1, \dots, \mathbf{x}_0^L) be a sequence from the LM1B test set with L=128L = 128. We use its first P=32P = 32 tokens as the prompt and start the reverse process from

xtT=(x01,…,x0P⏟prefix prompt,m,…,m⏟L−P masks)\mathbf{x}_{t_T} = (\underbrace{\mathbf{x}_0^1, \dots, \mathbf{x}_0^{P}}_{\text{prefix prompt}}, \underbrace{\mathbf{m}, \dots, \mathbf{m}}_{L-P \text{ masks}})

instead of the fully masked sequence of Algorithm 2 in the paper, with all other steps unchanged. The reverse transition (Eq. (2) in the paper) preserves unmasked tokens, keeping the prompt fixed while generating the continuation x^P+1:L\hat{\mathbf{x}}^{P+1:L}. Since masked diffusion models can fill arbitrary subsets of masked positions, no retraining is needed. Following the unconditional setup, we use 2000 test sequences with identical prompts and sampling seeds across models. We report conditional Gen-PPL / entropy HH, and MAUVE against the true continuations (Table 1).

Table 1: Conditional generation on LM1B. We use the first 32 tokens as a prefix prompt and generate 96 tokens. True continuations: Gen-PPL 38.45, H=4.12H = 4.12. Bold: best per row within each metric.

Gen-PPL (↓) / HH│MAUVE (↑, %)
NFEMDLMMDLM-MoEVADDE-MoE│MDLMMDLM-MoEVADDE-MoE
11006.0 / 4.14975.7 / 4.14860.1 / 4.13455.3 / 4.15│0.48 ± 0.010.47 ± 0.000.51 ± 0.021.06 ± 0.08
2586.0 / 4.14563.8 / 4.14516.5 / 4.14269.2 / 4.15│0.83 ± 0.050.89 ± 0.061.17 ± 0.097.14 ± 0.37
4282.2 / 4.13274.6 / 4.13261.0 / 4.14169.4 / 4.14│5.80 ± 0.456.59 ± 0.428.41 ± 0.4233.56 ± 1.33
8162.1 / 4.13155.6 / 4.13163.5 / 4.14129.2 / 4.14│33.86 ± 1.9434.38 ± 1.0334.52 ± 1.9956.65 ± 0.64
16117.8 / 4.13111.9 / 4.13125.9 / 4.13112.0 / 4.14│66.56 ± 1.4170.55 ± 0.9258.07 ± 1.0670.87 ± 1.90

The few-step advantage of E-MoE carries over to conditional generation (Table 1). At NFE ≤4\le 4, E-MoE lowers the Gen-PPL of the continuation by 1.51.5–2.2×2.2\times relative to all baselines and raises MAUVE by 22–9×9\times. MDLM-MoE again stays close to MDLM. At NFE =16= 16, E-MoE still attains the highest MAUVE and matches the best Gen-PPL. Moreover, E-MoE uses the prompt not only in the denoiser but also in the latent: its prior pθ(z∣xt)p_\theta(\mathbf{z} \mid \mathbf{x}_t) is computed from the noisy sequence, which contains the prompt, so the route is sampled conditionally on the context. In contrast, the latent of VADD is drawn from N(0,I)\mathcal{N}(0, I) independently of the prompt. The discrete latent of E-MoE is therefore conditioned on the context by construction, which makes it naturally suited to conditional generation without any change to the model. We have added these results to the revised paper (Section 5.3, Table 6).

Downstream benchmarks. Standard downstream benchmarks target models trained on much larger corpora and are not informative at the scale of LM1B for any of the compared models. Evaluating E-MoE on them requires the larger-scale models discussed above, which we leave for future work.

AuthorsOpenReview ↗

Response to Reviewer C31x (2/3)

[W.2 / Q.1] Unfair comparison regarding total parameter footprint. While active parameter counts are matched, an MoE backbone has a larger total parameter count and memory footprint than a dense MDLM. The paper lacks a dense baseline matched for total parameter capacity.

To show that the gain of E-MoE comes from the latent and not from its total parameter count, we additionally train MDLM-MoE and MDLM-dense with the same batch size and the same number of steps as E-MoE. MDLM-MoE is E-MoE with deterministic routing and no latent (see our response to Reviewer eAT7, [W.2 / Q.2]). MDLM-dense is MDLM with 8×8\times wider MLPs, so all of its 536M parameters are active for every token. Table 2 reports the inference cost and sample quality, and Table 3 the training cost.

Table 2: Inference cost and sample quality on LM1B against baselines of the same total size. All models are trained with the same batch size for the same number of steps and evaluated with one protocol. Gen-PPL, entropy and MAUVE are computed on 2000 samples per NFE, and MAUVE uses 2000 LM1B test sentences as reference. Bold: best per row within each metric.

GFLOPs / seq. (↓)│Gen-PPL (↓) / HH│MAUVE (↑, %)
NFEMDLMMDLM-denseMDLM-MoEE-MoE│MDLMMDLM-denseMDLM-MoEE-MoE│MDLMMDLM-denseMDLM-MoEE-MoE
156.7259.756.856.8│1433.8 / 4.371385.5 / 4.361281.8 / 4.33643.8 / 4.35│0.931.000.954.67
285.1389.585.285.1│997.5 / 4.37890.1 / 4.36813.4 / 4.33383.7 / 4.35│2.952.942.9324.58
4141.8649.2141.9141.9│477.8 / 4.36383.9 / 4.34383.9 / 4.32235.4 / 4.34│17.0925.0423.4558.25
8255.31168.5255.4255.4│260.5 / 4.35209.5 / 4.33213.6 / 4.32174.9 / 4.34│57.0466.0465.2178.68
16482.22207.1482.5482.5│179.4 / 4.35144.3 / 4.33152.4 / 4.33146.4 / 4.34│83.7586.3884.2488.37
32936.04284.4936.6936.6│148.6 / 4.35119.0 / 4.33124.6 / 4.32135.1 / 4.34│90.5390.8188.9690.59
641843.68439.01844.91844.8│134.1 / 4.35107.0 / 4.33112.7 / 4.33128.1 / 4.33│91.1791.8192.2288.53
1283658.816748.33661.43661.2│126.3 / 4.35101.1 / 4.32105.6 / 4.33126.3 / 4.34│91.2992.3390.7089.02

Inference. At few steps, where many tokens are revealed at once and the factorization barrier is strongest, scaling the backbone barely helps: at NFE =2= 2 MDLM, MDLM-MoE and MDLM-dense all reach a MAUVE of 2.9%, against 24.6% for E-MoE, which also lowers Gen-PPL by 2.22.2–2.3×2.3\times relative to MDLM-dense at NFE ≤2\le 2. As NFE grows, the barrier fades and the extra compute of MDLM-dense lowers its Gen-PPL, but each of its steps costs 4.6×4.6\times more than a step of E-MoE, which keeps the per-step cost of MDLM. We have added MDLM-MoE and MDLM-dense to the revised paper (Section 5.3, Tables 4 and 5, Figure 5), with the full NFE sweep and the FLOP count in Appendix C.3.2.

AuthorsOpenReview ↗

Response to Reviewer C31x (3/3)

Table 3: Parameters and training cost.

ModelParams (total)Params (active)Training GFLOPs / seq.Training vs. MDLMInference FLOPs vs. MDLM
MDLM139.3M139.3M85.11.0×1.0×
MDLM-dense535.9M535.9M389.54.6×4.6×
MDLM-MoE536.1M139.4M99.61.2×1.0×
E-MoE536.1M139.4M184.92.2×1.0×

Training. Training FLOPs are counted for sparse expert implementation, which gives the same outputs and gradients as the dense expert evaluation of the released code. The clean pass for the posterior routes makes a training step of E-MoE 1.9×1.9\times as costly as that of MDLM-MoE, which is still 2.1×2.1\times cheaper than MDLM-dense. Scaling a factorized model, in active width or as a deterministic mixture, therefore leaves its few-step quality almost unchanged, while the latent of E-MoE improves it at the inference cost of MDLM.

[Q.3] Did you encounter any expert load-imbalance issues during training, and was any auxiliary routing loss required to prevent only a few experts from dominating?

No auxiliary loss was required. E-MoE is trained without any load-balancing loss. In a standard MoE the routes allocate compute, and a balancing loss keeps all experts trained and the load even. In E-MoE the routes are a latent variable, and the objective itself decides how much information about x0x_0 they carry, since every route the prior cannot predict is charged by the KL term. The model therefore selects how many mixture components it uses, as a VAE selects its active latent units [7]. Uneven usage of single experts is thus expected, and with sparse dispatch it does not change the compute per token. Since a route is chosen at every layer, the mixture components of E-MoE are the latent codes zℓ=(z1ℓ,…,zDℓ)z^\ell = (z^\ell_1, \dots, z^\ell_D) of a position ℓ\ell across all DD layers, not single experts. We measure how much information they carry about the data by

I(x0;zℓ)=H(qˉℓ)−Epdata(x0)H(qθ(zℓ∣x0)),qˉℓ(zℓ)=Epdata(x0)qθ(zℓ∣x0),I(x_0; z^\ell) = H(\bar q^\ell) - \mathbb{E}_{p_{\text{data}}(x_0)} H\big(q_\theta(z^\ell \mid x_0)\big), \qquad \bar q^\ell(z^\ell) = \mathbb{E}_{p_{\text{data}}(x_0)} q_\theta(z^\ell \mid x_0),

averaged over positions ℓ\ell. I(x0;zℓ)=0I(x_0; z^\ell) = 0 if the latent codes ignore x0x_0. On 2000 LM1B test sequences the codes zℓz^\ell take 19 distinct values and I(x0;zℓ)=1.69I(x_0; z^\ell) = 1.69 nats per token, so they carry substantial information about x0x_0, which the few-step gains rely on (mixture ablation in Appendix C.3). MDLM-MoE, whose routing is deterministic and carries no latent, is trained with the standard load-balancing loss of weight 0.01.


References

[0] Sahoo et al. Simple and Effective Masked Diffusion Language Models. NeurIPS, 2024.
[1] Lou et al. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. ICML, 2024.
[2] Arriola et al. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. ICLR, 2025.
[3] Sahoo et al. The Diffusion Duality. ICML, 2025.
[4] Xie et al. Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling. ICLR, 2026.
[5] Bie et al. LLaDA2.0: Scaling Up Diffusion Language Models to 100B. arXiv, 2025.
[6] Zhu et al. LLaDA-MoE: A Sparse MoE Diffusion Language Model. arXiv, 2025.
[7] Burda et al. Importance Weighted Autoencoders. ICLR, 2016.


Concluding remarks: We hope these clarifications and additional experiments address your concerns. We would be happy to provide further clarification if needed, and would appreciate it if you could let us know whether this changes your assessment.

Thank you,

The Authors

AuthorsOpenReview ↗

Final remarks before the end of the discussion period

Dear Reviewer C31x,

We thank you for your review and appreciate the time you have dedicated to evaluating our paper.

As the discussion period is coming to a close, we would be grateful for any final feedback you may have regarding our responses to your questions and concerns. The general response above summarizes the additions, and the revised paper has all changes marked in blue. We would be happy to address any remaining points.

Thank you in advance.

Best regards,

The Authors

Reviewer_eAT7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes E-MoE, a masked diffusion model that uses stochastic Mixture-of-Experts routing decisions as discrete latent variables to capture dependencies between tokens generated in the same denoising step. The same router is used to construct a prior from the corrupted input and a variational posterior from the clean input, avoiding a separate recognition network. Experiments on synthetic distributions, binarized MNIST, and LM1B show substantial improvements in few-step generation, particularly in terms of generative perplexity and MAUVE.

Critical strengths

[S1] Interesting and potentially useful way of addressing the factorization problem. The inability to model dependencies between simultaneously generated tokens is an important limitation of masked diffusion models, especially at low NFE. Using the routing decisions of an MoE backbone as discrete latents is a neat idea. I particularly like that the prior and posterior share the same router, rather than requiring an additional encoder as in VADD. This is probably the most interesting aspect of the paper, although the general idea of using latent mixtures to address factorization is not new.

[S2] Promising improvements in few-step generation. The LM1B results are quite strong, particularly at 1–4 sampling steps. I also found the toy examples to be quite useful for illustrating the proposed mechanism. The greedy-routing ablation provides some evidence that stochastic routing matters, rather than simply having a larger network. If these improvements hold under more controlled comparisons, I think the approach could be impactful.

Critical weaknesses

I like the core idea, and the few-step improvements are genuinely interesting. However, the mismatch between the derivation and implementation is a substantial concern, and the lack of a matched MoE baseline makes it hard to tell how much of the improvement should be attributed to the proposed mechanism. The efficiency claims also need to be revisited. I would be open to increasing my score if the authors convincingly resolve the objective issue and provide stronger controlled comparisons.

[W1] There appears to be a serious discrepancy between the theoretical objective and the actual implementation. This is my biggest concern. Eq. (9) derives an objective where the route KL is weighted by T, and App A.3 explicitly emphasizes this. However, the released LM code uses continuous time (T=0) and weights the KL by 1/t, effectively optimizing (CE + KL)/t. The MNIST code also applies the diffusion weighting to both terms. These are NOT equivalent objectives, and I don't see how the claimed likelihood upper bound follows for the implemented loss. This is particularly problematic for the MNIST BPD comparison, which is presented as a negative ELBO. There are also some inconsistencies in the mathematical formulation of the routing distributions, since later-layer routing depends on routes at other positions through attention. I think the authors need to clarify what exactly is being optimized and provide a derivation corresponding to the code.

[W2] The experiments don't convincingly isolate the contribution of the proposed routing mechanism, and some efficiency claims are questionable. E-MoE has 536M total parameters compared to 139M for MDLM. While the authors match nominal active parameters, a factorized MoE with the same expert architecture and capacity seems like an essential missing baseline. The greedy-routing ablation is useful, but changing the sampling behavior of an already-trained stochastic model is not equivalent to training a factorized MoE.

More concerning, the provided code actually computes all experts and combines their outputs using routing weights, rather than executing only the selected expert. Thus, the claim of no additional inference cost is not supported by the implementation. The sampling code also performs an additional final forward pass beyond the nominal NFE, and I noticed some differences in numerical precision between the VADD and E-MoE sampling paths. Actual runtime/FLOPs measurements and a consistent evaluation protocol would help a lot here.

[W3] Novelty is somewhat overstated. VADD already motivates latent mixtures and discusses conditional priors, while LADD includes discrete latent variants. The main novelty is therefore the particular MoE-based parameterization, rather than discrete latents or learned priors themselves. Finally, the claim that the approach avoids KL warmup and auxiliary stabilization is not generally true (the MNIST setup uses both KL warmup and a load-balancing loss).

Questions for authors
  1. Can the authors clarify the objective used in the experiments? In particular, why is the route KL divided by time in the implementation, instead of having the coefficient derived in Eq. (9)? Does the resulting objective still upper-bound NLL? This would materially affect my soundness assessment and interpretation of Table 3.

  2. How much improvement remains against a matched factorized MoE? I would really like to see a baseline with the same number of experts, total parameters, and training budget, but without stochastic latent-variable training. This seems necessary to establish that the gains come from the proposed method rather than increased capacity.

  3. Can the authors clarify the actual inference cost? The released implementation evaluates all expert MLPs, despite the claimed sparse routing. What are the measured latency and FLOPs relative to MDLM/VADD? Also, are the additional final denoising pass and differences in sampling precision included in the reported comparisons?

  4. Could the authors clarify the posterior construction and collapse claims? Fig3 suggests conditioning the posterior on both clean and noisy inputs, whereas the code appears to use the clean input alone. Also, positive routing KL does not necessarily establish meaningful latent usage, and MNIST requires KL warmup and load balancing. How should these observations be reconciled with the claims?

7 comments
AuthorsOpenReview ↗

Response to Reviewer eAT7 (1/6)

Dear Reviewer, thanks for your efforts and your thoughtful review. We are delighted that you found our improvements in few-step generation and way of addressing the factorization problem to be promising and useful. Below we respond to your specific questions.

[W.1 / Q.1] The connection between theoretical objective Eq. (9) and implementation. Mathematical formulation of the routing distributions.

We presented Eq. (9) to make the comparison with VADD explicit. It applies the variational argument to the E-MoE reverse transition using a clean-conditioned proposal for all routes at every step, which gives the full routing KL a coefficient of TT in the uniform step average. Applying the same argument to the same reverse transition with a reveal-dependent proposal gives a finite-grid NLL upper bound with a shared 1/ti1/t_i weight for reconstruction and routing. This derivation explains the time weighting used in our training loss. The NLL guarantee applies when both terms are evaluated under the mixed proposal.

The bound in equation 9. The derivation of Eq. (9) uses the clean-conditioned proposal ri(z)=qθ(z∣xti,x0)r_i(\mathbf{z})=q_\theta(\mathbf{z}\mid\mathbf{x}_{t_i},\mathbf{x}_0) for the entire route z\mathbf{z}, including every position and layer. This proposal is the same for every possible outcome of the reverse step. Consequently, the full route KL enters the bound even when the step reveals a single token. Writing the sum over TT steps as a uniform average gives this full KL a coefficient of TT.

The reveal-dependent NLL bound. To derive a bound that gives both terms the same time weight in the code, we condition the proposal on the set of newly revealed positions Ri=M(xti)∖M(xti−1)\mathcal{R}_i = \mathcal{M}(\mathbf{x}_{t_i}) \setminus \mathcal{M}(\mathbf{x}_{t_{i-1}}) at the (i−1)(i-1)-th step. Let πi(S)=Pr⁡(Ri=S∣xti,x0)\pi_i(S) = \Pr(\mathcal{R}_i = S \mid \mathbf{x}_{t_i}, \mathbf{x}_0) and πiℓ=πi(⋅∣ℓ∈Ri)\pi_i^\ell = \pi_i(\cdot \mid \ell \in \mathcal{R}_i) for ℓ∈M(xti)\ell \in \mathcal{M}(\mathbf{x}_{t_i}). The generative route prior is pi(z)=pθ(z∣xti)p_i(\mathbf{z}) = p_\theta(\mathbf{z} \mid \mathbf{x}_{t_i}), and z<d\mathbf{z}_{\lt d} contains the routes at all positions in earlier layers. With conditional independence across positions within a layer given this history, the joint laws and the mixed proposal's conditionals are:

ri,Ri(zdℓ∣z<d)={qθ(zdℓ∣z<d,xti,x0),ℓ∈Ri,pi(zdℓ∣z<d),ℓ∉Ri.r_{i,\mathcal{R}_i}(\mathbf{z}_d^\ell \mid \mathbf{z}_{\lt d}) = \begin{cases} q_\theta(\mathbf{z}_d^\ell \mid \mathbf{z}_{\lt d}, \mathbf{x}_{t_i}, \mathbf{x}_0), & \ell \in \mathcal{R}_i, \\\\ p_i(\mathbf{z}_d^\ell \mid \mathbf{z}_{\lt d}), & \ell \notin \mathcal{R}_i. \end{cases}

As for mathematical formulation of the routing distributions, we correct misprint and revise posterior and prior notations in the main text to condition each routing factor on z<d\mathbf{z}_{ < d}, the expert assignments at all positions in earlier layers. Both terms in the conditional KL are evaluated on the same sampled history.

The Jensen's inequality applies separately to each possible outcome xti−1\mathbf{x}_{t_{i-1}} of the reverse step. Applying Jensen's inequality with ri,Rir_{i,\mathcal{R}_i} bounds the negative log transition probability by the expected negative log transition probability given the route plus KL(ri,Ri∥pi)\mathrm{KL}(r_{i,\mathcal{R}_i}\Vert p_i). The latter stays inside the expectation over reverse outcomes because Ri\mathcal{R}_i depends on the outcome. Define the per-position terms

\mathrm{KL}_{i,\ell}(\mathcal{R}_i, \mathbf{z}) := \sum_{d=1}^{D} \mathrm{KL}\left( r_{i,\mathcal{R}_i}(\mathbf{z}_d^\ell \mid \mathbf{z}_{\lt d}) \\, \middle\\| \\, p_i(\mathbf{z}_d^\ell \mid \mathbf{z}_{\lt d}) \right), \qquad \mathrm{CE}_{i,\ell}(\mathbf{z}) := -\log p_\theta(\mathbf{x}_0^\ell \mid \mathbf{x}_{t_i}, \mathbf{z}).

Each conditional KL averages the log-ratio from Proposition 3.1 over the current expert assignment zdℓ\mathbf{z}_d^\ell given its history. For ℓ∉Ri\ell \notin \mathcal{R}_i, the two conditional laws coincide at every sampled history, so KLi,ℓ=0\mathrm{KL}_{i,\ell} = 0. The chain rule therefore gives, at fixed Ri\mathcal{R}_i,

KL(ri,Ri∣pi)=Ez∼ri,Ri[∑ℓ∈RiKLi,ℓ(Ri,z)].\mathrm{KL}(r_{i,\mathcal{R}_i} \\| p_i) = \mathbb{E}_{\mathbf{z} \sim r_{i,\mathcal{R}_i}} \left[ \sum_{\ell \in \mathcal{R}_i} \mathrm{KL}_{i,\ell}(\mathcal{R}_i, \mathbf{z}) \right].

AuthorsOpenReview ↗

Response to Reviewer eAT7 (2/6)

The true reverse posterior and the model use the same reveal probabilities, so the probability of the reveal set cancels in the transition KL. Both reconstruction and routing contribute only at ℓ∈Ri\ell\in\mathcal{R}_i. With ti=i/Tt_i=i/T and αt=1−t\alpha_t=1-t, a masked position enters Ri\mathcal{R}_i with probability 1/(Tti)1/(Tt_i). Its expected contribution is this probability times its expectation conditional on being revealed. Rewriting the step sum as a uniform average multiplies that factor by TT, giving 1/ti1/t_i. Including the reconstruction step at i=1i=1, we get the following finite-grid NLL upper bound under the mixed proposal:

\sum_{\ell\in\mathcal{M}(\mathbf{x}_{t_i})} \mathbb{E}_{\substack{\mathcal{R}_i\sim\pi_i^\ell \\\\ \mathbf{z}\sim r_{i,\mathcal{R}_i}}} \frac{1}{t_i}\Bigl[ \mathrm{KL}_{i,\ell}(\mathcal{R}_i,\mathbf{z}) +\mathrm{CE}_{i,\ell}(\mathbf{z}) \Bigr] \Biggr].$$ Here $i\sim\operatorname{Unif}\\{1,\ldots,T\\}$ and $\mathbf{x}_{t_i}\sim q(\cdot\mid\mathbf{x}_0)$. The inner expectation includes the other positions revealed on the same step. Averaging the sampled log-ratios in derived estimation over the current assignments yields the equivalent form above that is still an upper-bound on NLL. We add the full derivation to the appendix A4. **Why the weight $1/t_i$.** The two bounds charge the routes differently. Under Eq. (9) the KL of the full route is paid at every step, when the token is revealed. With $T$ steps the routes cost $T$ times their information, and the objective favors switching the latent off. The reveal-dependent bound charges the routes of a token once, at the step that reveals it, as in a standard VAE. **Binarized MNIST.** The original E-MoE BPD in Table 3 was computed using the time-weighted surrogate. We replace it with an estimate of the finite-grid NLL upper bound in bits per dimension. We train E-MoE on binarized MNIST directly on the reveal-dependent bound, with $T$ drawn from a fixed grid and **without** KL warmup or load balancing. Since E-MoE resamples its routes at every step, its bound depends on the number of steps, and the relevant quantity is the bound of the $T$-step sampler used for generation. The revised Table 3 reports this bound for every model on the 10,000 test images, with the same random draws for all models. For MDLM it is the discrete-time NELBO, and for E-MoE the reveal-dependent bound above. For VADD, whose latent is shared by all pixels, the latent KL is charged once if the latent is kept fixed during sampling and at every step if it is resampled. **Revised Table 3: NLL upper bound (BPD, ↓) of the $T$-step sampler on binarized MNIST.** All models are evaluated on the 10,000 test images with the same random draws. Standard deviations over 4 draws are at most 0.001. **Bold:** best per column. | Model | $T=1$ | $T=2$ | $T=4$ | $T=8$ | $T=16$ | |:-|:-:|:-:|:-:|:-:|:-:| | MDLM | 0.290 | 0.175 | 0.120 | 0.096 | **0.086** | | VADD, latent fixed | **0.110** | 0.107 | 0.110 | 0.112 | 0.113 | | VADD, latent per step | **0.110** | 0.105 | 0.113 | 0.141 | 0.205 | | **E-MoE**, trained for $T \in \\{1,2,4\\}$ | 0.125 | **0.104** | 0.098 | 0.107 | 0.121 | | **E-MoE**, trained for $T \in \\{1,2,4,8,16\\}$ | 0.155 | 0.114 | **0.097** | **0.094** | 0.097 | E-MoE attains the lowest bound for $2 \le T \le 8$ and lowers the bound of the factorized MDLM by up to $2.3\times$ at $T=1$. The grid of $T$ used in training sets where the bound is lowest, as expected from an objective defined for a $T$-step sampler. VADD with a fixed latent pays its latent KL once and is best at $T=1$, but its bound does not improve with more steps. With the latent resampled at every step, its bound grows with $T$. As $T$ grows, each step reveals few pixels, the factorization barrier fades, and MDLM approaches its continuous-time bound. **LM1B.** On LM1B we train Eq. (9) in its $\beta$-VAE form [3]. The reconstruction term is unchanged, and the routing KL is scaled by $\beta_i = 1/(Tt_i) = 1/i \le 1$, which gives reconstruction and routing the same $1/t_i$ weight as in the reveal-dependent bound. We use this form on LM1B because it gives a single model for all numbers of steps, while the exact bound is tied to the grid of $T$ used in training, as the two E-MoE rows of the revised Table 3 show. The released code draws $t$ continuously ($T=0$), as MDLM and VADD do, which applies the same weight $1/t$ without fixing the number of sampling steps. Weighted objectives of this kind are standard practice: DDPM trains with a reweighted ELBO [4], text VAEs anneal the KL weight [5], and VADD anneals its latent KL over the first 100k training steps. We therefore evaluate the LM1B model by sample quality, and report likelihood on binarized MNIST, where the training objective is the exact bound.
AuthorsOpenReview ↗

Response to Reviewer eAT7 (3/6)

[W.2 / Q.2]. [W.2] The experiments don't convincingly isolate the contribution of the proposed routing mechanism. E-MoE has 536M total parameters compared to 139M for MDLM. [...] [Q.2] How much improvement remains against a matched factorized MoE? [...]

To check whether the gain comes from the latent routing or from the extra parameters, we trained the requested baseline, MDLM-MoE, from scratch. In contrast to the greedy-routing ablation (Table 9 in the paper), which applies arg⁡max⁡\arg\max routing to an already trained E-MoE, MDLM-MoE is trained with deterministic routing from scratch.

Setup. All models are trained on LM1B using the same DiT-small backbone, sequence packing, BERT-base-uncased tokenizer, sequence length L=128L=128, and training budget of 1M steps with global batch size 512. MDLM-MoE uses the same E-MoE architecture, with K=8K=8 experts per MLP and top-1 routing, matching its 536M total and 139M active parameters. The only difference is deterministic arg⁡max⁡\arg\max routing, without Gumbel sampling, latent prior/posterior or KL regularization. We use a standard load-balancing loss (weight 0.010.01). Sampling follows the paper's protocol (ancestral sampling, fp64 categorical sampling, 2000 samples per NFE). MDLM and E-MoE results are taken from Tables 4 and 8 and reproduced using the released code.

Table 1: Capacity control on LM1B. Left: Gen-PPL (↓\downarrow) / Entropy (H). Right: MAUVE (↑\uparrow, %). Bold: best per row within each metric.

Gen-PPL (↓) / HH│MAUVE (↑, %)
MDLMMDLM-MoEE-MoE│MDLMMDLM-MoEE-MoE
Trained params139M536M536M│139M536M536M
Active params139M139M139M│139M139M139M
NFE 11433.8 / 4.371281.8 / 4.33643.8 / 4.35│0.93 ± 0.040.95 ± 0.094.67 ± 0.31
NFE 2997.5 / 4.37813.4 / 4.33383.7 / 4.35│2.95 ± 0.292.93 ± 0.4224.58 ± 0.62
NFE 4477.8 / 4.36383.9 / 4.32235.4 / 4.34│17.09 ± 0.9423.45 ± 1.5858.25 ± 1.69
NFE 8260.5 / 4.35213.6 / 4.32174.9 / 4.34│57.04 ± 1.7265.21 ± 2.0678.68 ± 0.55
NFE 16179.4 / 4.35152.4 / 4.33146.4 / 4.34│83.75 ± 1.9184.24 ± 1.6488.37 ± 0.41
NFE 32148.6 / 4.35124.6 / 4.32135.1 / 4.34│90.53 ± 0.8688.96 ± 1.3690.59 ± 0.68
NFE 64134.1 / 4.35112.7 / 4.33128.1 / 4.33│91.17 ± 1.0692.22 ± 0.3488.53 ± 1.52
NFE 128126.3 / 4.35105.6 / 4.33126.3 / 4.34│91.29 ± 0.5690.70 ± 0.5189.02 ± 1.94

Results. Table 1 shows that extra parameters do not remove the factorization barrier. MDLM-MoE is still a factorized model of the form of Eq. (3) in the paper, in which all tokens revealed in one step are sampled independently. Thus, increasing its capacity can improve individual marginals but cannot capture dependencies between them. In contrast, E-MoE samples a shared route z∼pθ(z∣xt)\mathbf{z} \sim p_\theta(\mathbf{z} \mid \mathbf{x}_t) and yields a mixture of factorized distributions (Eq. (5) in the paper), which can represent such dependencies. This is why, with the same backbone and parameter count, E-MoE is far ahead at NFE ≤\le 16, where many tokens are revealed per step, while MDLM-MoE stays close to MDLM. At NFE ≥\ge 32 only a few tokens are revealed per step, the factorization error is small, and the extra capacity of MDLM-MoE gives lower Gen-PPL. We have added MDLM-MoE to Table 4, Figure 5 and the MAUVE table (Appendix C.3.1) of the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General answer

Dear Reviewers, thank you for your thoughtful feedback. We appreciate that you found the use of MoE routing decisions as a discrete shared latent a neat, elegant and well-motivated idea (eAT7, C31x, mLcQ) that needs no additional encoder (eAT7, C31x), the few-step improvements strong (eAT7, C31x, mLcQ) at no additional inference cost (C31x, mLcQ), and the toy examples (eAT7, C31x) and the routing ablation (eAT7, mLcQ) useful evidence for the proposed mechanism. Below we summarize the main additions.

(1) Baselines of the same total size. We trained two factorized models with the 536M total parameters of E-MoE, with the same batch size and the same number of steps: MDLM-MoE, which is E-MoE with deterministic routing and no latent, and MDLM-dense, which is MDLM with 8×8\times wider MLPs. At NFE =2= 2 MDLM, MDLM-MoE and MDLM-dense all stay at a MAUVE of about 3%, against 24.6% for E-MoE, which also lowers Gen-PPL by 2.02.0–2.3×2.3\times relative to both at NFE ≤2\le 2. Extra capacity does not remove the factorization barrier, so the few-step gain comes from the latent.

(2) Training objective and likelihood bound. Appendix A.4 (new) derives an NLL upper bound with a reveal-dependent proposal, which takes posterior routes only for the tokens revealed at the current step, so that reconstruction and routing share the weight 1/ti1/t_i. On binarized MNIST, E-MoE is now trained directly on this bound, without KL warmup or load balancing, and the revised Table 3 reports the bound of the TT-step sampler for all models.

(3) Inference and training cost. At inference E-MoE uses standard top-1 sparse MoE dispatch, so a single expert per token is evaluated. FLOPs measured with FlopCounterMode match a layer-wise analytic count: one denoising step costs 28.38 GFLOPs per sequence for E-MoE and 28.36 for MDLM. All models use the same sampler, including the final noise-removal pass of MDLM, which is counted in all FLOPs and metrics, and VADD is now evaluated with float64 categorical sampling as all other models. A training step of E-MoE costs 2.2×2.2\times that of MDLM and is 2.1×2.1\times cheaper than that of MDLM-dense.

(4) Conditional generation. With prefix prompts from the LM1B test set, E-MoE lowers the Gen-PPL of the continuation by 1.51.5–2.2×2.2\times relative to all baselines at NFE ≤4\le 4 and raises MAUVE by 22–9×9\times (Table 6). Its prior routes are computed from the noisy sequence, which contains the prompt, so the latent is conditioned on the context by construction.

(5) Latent usage. On LM1B the routes carry I(x0;zℓ)=1.69I(x_0; z^\ell) = 1.69 nats per token of information about the data, where a collapsed latent would give 00. Replacing the sampled route by arg⁡max⁡\arg\max on the same trained checkpoint collapses the mixture to one factorized component and raises Gen-PPL by 1.7×1.7\times and 2.1×2.1\times at NFE =1= 1 and 22 (Appendix C.3.3). The latent is therefore in use, and the few-step gain comes from sampling it.


Revised paper. Changes are highlighted in blue.

  • Sections 2 and 3.2, Figure 3. We corrected the notation of the prior and posterior routing distributions. In Section 3.2 each routing factor is conditioned on the routes at all positions in earlier layers.
  • Section 3.3. We refer to the reveal-dependent bound of Appendix A.4.
  • Section 5. We state the matched baselines and the two factorized models of the same total size as E-MoE.
  • Section 5.2. Table 3 reports the NLL upper bound of the TT-step sampler for all models on the test images. We revised the results paragraph.
  • Section 5.3. We describe MDLM-MoE and MDLM-dense, add MDLM-MoE to Table 4 and Figure 5, and evaluate VADD with float64 sampling. The new Table 5 gives the capacity control and the new Table 6 the conditional generation.
  • Appendix A.4 (new). The reveal-dependent likelihood bound (Proposition 3.2) and its proof.
  • Appendix B. The final noise-removal pass is counted for every model. B.2 gives the E-MoE setup on binarized MNIST, trained on the bound without KL warmup or load balancing, and the evaluation of the bound. B.3 describes MDLM-MoE and MDLM-dense.
  • Appendix C.3.1. The MAUVE table includes MDLM-MoE and VADD with float64 sampling.
  • Appendix C.3.2 (new). Sparse inference, FLOPs by layer, inference cost and sample quality against baselines of the same total size, and training cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.