- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes the Equilibrium Diffusion Language Model, which approaches generation as a process of finding and preserving valid solutions. It is trained with a simple cross-entropy loss. The authors also provide a theoretical analysis of why this model works.
Critical strengths
-
The distinction between distribution matching and finding any valid solution is insightful and provides an interesting motivation for the proposed approach.
-
The method uses a single cross-entropy training objective, with the same predictor supporting both token generation and adaptive stopping without requiring a separate verifier at inference.
-
The theoretical analysis provides useful insights into the proposed generation process, clarifying the conditions under which it can recognize, reach, and preserve valid solutions.
Critical weaknesses
-
The paper would benefit from clearer positioning relative to prior equilibrium-based language models, including Equilibrium Language Models (Jiang et al., 2026).
-
The relationship to uniform-state diffusion language models deserves a more detailed discussion. Although these models are briefly discussed, a direct comparison of the training objectives, update rules, and stopping mechanisms would help clarify the methodological contribution. It would also be useful to discuss whether the validity-recognition analysis extends to existing uniform-state diffusion denoisers.
-
The empirical evaluation is limited in scope. Experiments with more diverse benchmarks would strengthen the empirical evidence. Considering the potential resource limitation, this is a minor issue.
Questions for authors
See weakness.
2 comments
I appreciate the author's detailed response. I found the comparison between EqDLM and UDLM very helpful. The limitation on the scope of the evaluation is also reasonable given the short rebuttal period; I will maintain my positive score.
Rebuttal
Dear Reviewer nNMA, thank you for your comments and for taking the time to review our work. Please find our responses below.
[W.1] The paper would benefit from clearer positioning relative to prior equilibrium-based language models, including Equilibrium Language Models (Jiang et al., 2026).
We have expanded Related Work (Appendix I, “Equilibrium-based language models”) to discuss Deep Equilibrium Models [1], including DEQ-Transformers for language modeling, their stabilization by Jacobian regularization [2], and Equilibrium Language Models [3]. These methods compute a fixed point of hidden representations within a network evaluation. Jiang et al. use this mechanism to replace groups of Transformer layers for compression. In EqDLM, equilibrium instead concerns the generated sequence across denoising calls: we study conditions under which valid solutions become absorbing states and derive a stopping criterion from the probability of reproducing the current candidate. Our denoiser can be an ordinary Transformer, without a hidden-state fixed-point solver.
[W.2] The relationship to uniform-state diffusion language models deserves a more detailed discussion. Although these models are briefly discussed, a direct comparison of the training objectives, update rules, and stopping mechanisms would help clarify the methodological contribution. It would also be useful to discuss whether the validity-recognition analysis extends to existing uniform-state diffusion denoisers.
The main distinction is what the objective requires these predictions to represent. EqDLM learns a factorized distribution fθ(x∣z)=∏ifθ,i(xi∣z) for reconstructing a clean solution x from a corrupted candidate z. UDLM instead optimizes reverse diffusion transitions Pθ(zs∣zt) between noise levels s<t. This leads to different training, updates, and stopping rules:
Let q(zs∣zt,x) denote the forward process's conditional reverse transition given a clean sequence x, and let gθ(zt,t) be the network's predicted clean-token probabilities. UDLM ([4], Section 4.1, Eq. 15) substitutes these probabilities for the one-hot clean-token indicators in the transition formula:
Pθ(zs∣zt)=q!(zs∣zt,;x=gθ(zt,t)).
This is the plug-in parameterization. The ELBO evaluates the resulting transition, whereas EqDLM's CE directly evaluates the predicted clean solution. Our recognition proof requires the CE population optimum: each prediction must equal the corresponding clean-token posterior probability. The UDLM objective does not directly impose this requirement on its network output, so our e−1 guarantee does not directly apply. Appendix D of the revised paper compares the shared features, corruption and time conditioning, training objectives, sampling updates, and stopping rules.
[W.3] The empirical evaluation is limited in scope. Experiments with more diverse benchmarks would strengthen the empirical evidence. Considering the potential resource limitation, this is a minor issue.
We use established problem-solving setups from recent diffusion language modeling work, with comparisons to MDLM, Duo, CANDI, FLM, S-FLM, FMLM+, and autoregressive models. We agree that more diverse benchmarks would strengthen the evaluation and appreciate your acknowledgment of the resource constraints. Extending to broader mathematical reasoning and code generation typically requires larger models with substantial pretraining and task-specific fine-tuning, making this difficult to complete within the rebuttal period. We consider this an important direction for future work.
Concluding remarks. We would be grateful if you could let us know whether our explanations have addressed your concerns. If so, we kindly ask you to consider increasing your rating. We are happy to discuss any remaining questions.
References
[1] Bai et al. Deep Equilibrium Models. NeurIPS, 2019.
[2] Bai et al. Stabilizing Equilibrium Models by Jacobian Regularization. ICML, 2021.
[3] Jiang et al. Equilibrium Language Models. ICLR, 2026.
[4] Schiff et al. Simple Guidance Mechanisms for Discrete Diffusion Models. ICLR, 2025.