- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes E-MoE, a non-factorized masked diffusion language model that uses MoE routing decisions as discrete shared latent variables to capture dependencies among tokens generated within the same denoising step. The method avoids additional inference cost while substantially improving few-step generation. Experiments on synthetic data, Binarized-MNIST, and LM1B consistently demonstrate the effectiveness of the proposed approach.
Critical strengths
The paper identifies an important limitation of masked diffusion models—the factorization error when multiple tokens are generated simultaneously—and proposes a simple and well-motivated solution by reusing MoE routing decisions as discrete shared latent variables.
The empirical results are strong and convincing, particularly in the few-step regime where the proposed method substantially outperforms both factorized MDMs and the continuous-latent baseline without additional inference cost. The ablation on routing further provides clear evidence that the improvement comes from the proposed mixture formulation.
Critical weaknesses
It would be useful to understand whether the advantage of E-MoE also transfers to conditional generation and downstream language modeling tasks.
Questions for authors
How sensitive is E-MoE to the number of experts and the routing strategy? In particular, is the improvement mainly due to having a discrete shared latent, or does it depend strongly on the specific MoE configuration?
3 comments
Response to Reviewer mLcQ (2/2)
[Q.1] How sensitive is E-MoE to the number of experts and the routing strategy? In particular, is the improvement mainly due to having a discrete shared latent, or does it depend strongly on the specific MoE configuration?
We answer this question with controls that separate the latent from the MoE architecture and with the expert usage of the trained model.
1) Where the improvement comes from. (1) MDLM-MoE has exactly the MoE configuration of E-MoE (backbone, experts, top-1 dispatch and training budget), but deterministic routing and no latent. It stays close to MDLM in the few-step regime, both in unconditional generation (see our response to Reviewer eAT7, [W.2 / Q.2]) and in the conditional setting above. The MoE architecture and its extra parameters alone therefore do not remove the factorization error. (2) The greedy-routing ablation in Table 9 of the submission keeps the trained E-MoE weights and only replaces the sampled route by , which collapses the mixture to a single factorized component. With the same configuration, and even with the same weights, the gain disappears once the route is no longer a sampled latent. The improvement therefore comes from the discrete shared latent, i.e., from the mixture of factorized distributions.
2) Sensitivity to the MoE configuration. E-MoE does not rely on a specific number of experts or routing scheme.
Number of experts. The number of experts sets how many choices the route has in each layer, but the model does not have to use all of them. The KL term charges the model for every routing decision that the prior cannot predict, so the model itself selects how many components it uses. therefore acts as an upper limit on the capacity of the latent rather than as a hyperparameter to tune. To measure how much of this capacity carries information about the data, we compute the mutual information between the data and the latent codes of a position across all layers,
averaged over positions . It is zero if the routes ignore , and it grows as the routes become more diverse across sequences and more certain for a given sequence. With , on 2000 LM1B test sequences, nats per token, so the latent carries substantial information about the data.
Routing strategy. E-MoE uses the standard top-1 routing of sparse MoE. What matters is only whether the route is sampled or deterministic, and part 1 shows that the gain comes from sampling the route as a latent.
To summarize, the improvement is due to the discrete shared latent and does not depend strongly on the specific MoE configuration.
References
[0] Sahoo et al. Simple and Effective Masked Diffusion Language Models. NeurIPS, 2024.
[1] Xie et al. Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling. ICLR, 2026.
Thank you,
The Authors
Thanks for the detailed response. I have no further questions.
Response to Reviewer mLcQ (1/2)
Dear Reviewer mLcQ, thank you for your thoughtful and positive assessment. We are glad that you found the idea simple and well motivated and the results convincing. Your comments raise two important directions: conditional generation and the sensitivity of E-MoE to its MoE configuration. We address them below. Throughout, we also compare with MDLM-MoE, which has the same backbone, experts and training budget as E-MoE, but deterministic routing and no latent.
[W.1]It would be useful to understand whether the advantage of E-MoE also transfers to conditional generation and downstream language modeling tasks.
Conditional generation. Let x0=(x01,…,x0L) be a sequence from the LM1B test set with L=128. We use its first P=32 tokens as the prompt and start the reverse process from
xtT=(prefix promptx01,…,x0P,L−P masksm,…,m)
instead of the fully masked sequence of Algorithm 2 in the paper, with all other steps unchanged. Unmasked tokens are copied by the reverse transition (Eq. (2) in the paper), so the prompt stays fixed and the model generates the continuation x^P+1:L conditioned on it. Masked diffusion models are trained to fill in arbitrary subsets of masked positions, so none of the compared models requires retraining. The setup is the same as for unconditional generation: we use 2000 test sequences, and all models receive the same prompts and the same sampling seed. We report the Gen-PPL of the continuation given the prompt, the entropy H of the generated tokens, and MAUVE between the generated and the true continuations (Table 1).
Table 1: Conditional generation on LM1B. We use the first 32 tokens as a prefix prompt and generate 96 tokens. Bold: best per row within each metric.
The few-step advantage of E-MoE carries over to conditional generation (Table 1). E-MoE targets the few-step regime, where many tokens are revealed at once and the factorization barrier is strongest. This is where the gain is largest: at NFE ≤4, E-MoE lowers the Gen-PPL of the continuation by 1.5–2.2× relative to all baselines and raises MAUVE by 2–9×, at no loss of entropy. MDLM-MoE stays close to MDLM, as in unconditional generation. At NFE =16, E-MoE still attains the highest MAUVE and matches the best Gen-PPL, so the latent brings its gain in the few-step regime and costs nothing in quality elsewhere. Moreover, E-MoE uses the prompt not only in the denoiser but also in the latent: its prior pθ(z∣xt) is computed from the noisy sequence, which contains the prompt, so the route is sampled conditionally on the context. In contrast, the latent of VADD is drawn from N(0,I) independently of the prompt. The discrete latent of E-MoE is therefore conditioned on the context by construction, which makes it naturally suited to conditional generation without any change to the model. We have added these results to the revised paper (Section 5.3, Table 6).
Downstream tasks. Following prior work on discrete diffusion at this scale [0, 1], every model in our comparison is a full pretraining run from scratch on LM1B, with the same budget for all methods, which already requires substantial compute. E-MoE uses 139M active parameters, as MDLM. Standard downstream benchmarks target models trained on much larger corpora and are not informative at this scale for any of the compared models. Scaling E-MoE to such regimes is a next step that we leave for future work.