acceptodds
Under review as a conference paper at ICLR 2027

Equilibrium Diffusion Language Models

Abstract

Diffusion language models generate text through iterative, parallel token updates. When applying these models to reasoning tasks, we often seek any correct solution, rather than reproduce the distribution of answers in the training data. We introduce *Equilibrium Diffusion Language Models*, which approach generation as a process of finding and preserving valid solutions. We train the model with a single cross-entropy objective to reconstruct valid solutions from corrupted inputs. Our theoretical analysis identifies conditions on the corruption process under which valid solutions become absorbing states of the population-optimal model and repeated sampling reaches them almost surely, with exponentially fast convergence. The analysis also establishes a probability gap between preserving valid and invalid states, suggesting a criterion for stopping generation. We evaluate the approach on unconditional generation of valid Sudoku grids and program synthesis with training on TinyGSM and evaluation on GSM8K. On GSM8K, our model achieves 63.6% accuracy, comparable to 63.3% for the autoregressive greedy baseline.

Primary area:
generative models
Keywords:
Discrete diffusion, Diffusion language models, Equilibrium models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nNMAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the Equilibrium Diffusion Language Model, which approaches generation as a process of finding and preserving valid solutions. It is trained with a simple cross-entropy loss. The authors also provide a theoretical analysis of why this model works.

Critical strengths
  1. The distinction between distribution matching and finding any valid solution is insightful and provides an interesting motivation for the proposed approach.

  2. The method uses a single cross-entropy training objective, with the same predictor supporting both token generation and adaptive stopping without requiring a separate verifier at inference.

  3. The theoretical analysis provides useful insights into the proposed generation process, clarifying the conditions under which it can recognize, reach, and preserve valid solutions.

Critical weaknesses
  1. The paper would benefit from clearer positioning relative to prior equilibrium-based language models, including Equilibrium Language Models (Jiang et al., 2026).

  2. The relationship to uniform-state diffusion language models deserves a more detailed discussion. Although these models are briefly discussed, a direct comparison of the training objectives, update rules, and stopping mechanisms would help clarify the methodological contribution. It would also be useful to discuss whether the validity-recognition analysis extends to existing uniform-state diffusion denoisers.

  3. The empirical evaluation is limited in scope. Experiments with more diverse benchmarks would strengthen the empirical evidence. Considering the potential resource limitation, this is a minor issue.

Questions for authors

See weakness.

2 comments

Dear Reviewer nNMA, thank you for your comments and for taking the time to review our work. Please find our responses below.

[W.1] The paper would benefit from clearer positioning relative to prior equilibrium-based language models, including Equilibrium Language Models (Jiang et al., 2026).

We have expanded Related Work (Appendix I, “Equilibrium-based language models”) to discuss Deep Equilibrium Models [1], including DEQ-Transformers for language modeling, their stabilization by Jacobian regularization [2], and Equilibrium Language Models [3]. These methods compute a fixed point of hidden representations within a network evaluation. Jiang et al. use this mechanism to replace groups of Transformer layers for compression. In EqDLM, equilibrium instead concerns the generated sequence across denoising calls: we study conditions under which valid solutions become absorbing states and derive a stopping criterion from the probability of reproducing the current candidate. Our denoiser can be an ordinary Transformer, without a hidden-state fixed-point solver.

[W.2] The relationship to uniform-state diffusion language models deserves a more detailed discussion. Although these models are briefly discussed, a direct comparison of the training objectives, update rules, and stopping mechanisms would help clarify the methodological contribution. It would also be useful to discuss whether the validity-recognition analysis extends to existing uniform-state diffusion denoisers.

The main distinction is what the objective requires these predictions to represent. EqDLM learns a factorized distribution fθ(x∣z)=∏ifθ,i(xi∣z)f_\theta(x\mid z)=\prod_i f_{\theta,i}(x_i\mid z) for reconstructing a clean solution xx from a corrupted candidate zz. UDLM instead optimizes reverse diffusion transitions Pθ(zs∣zt)P_\theta(z_s\mid z_t) between noise levels s<ts<t. This leads to different training, updates, and stopping rules:

EqDLMUDLM
CorruptionAny corruption satisfying our coverage assumption, without a time input.Uniform diffusion with a time-conditioned network.
TrainingMinimize reconstruction cross-entropy (CE): E[−log⁡fθ(x∣z)]\mathbb E[-\log f_\theta(x\mid z)].Minimize the negative continuous-time evidence lower bound (ELBO) for the reverse process.
UpdateCategorical sampling from fθ(⋅∣z)f_\theta(\cdot\mid z), or annealed categorical sampling.Sample zs∼Pθ(⋅∣zt)z_s\sim P_\theta(\cdot\mid z_t) along the noise schedule.
StoppingAccept when fθ(z∣z)>e−1f_\theta(z\mid z)>e^{-1}.Complete the reverse noise schedule.

Let q(zs∣zt,x)q(z_s\mid z_t,x) denote the forward process's conditional reverse transition given a clean sequence xx, and let gθ(zt,t)g_\theta(z_t,t) be the network's predicted clean-token probabilities. UDLM ([4], Section 4.1, Eq. 15) substitutes these probabilities for the one-hot clean-token indicators in the transition formula:

Pθ(zs∣zt)=q!(zs∣zt,;x=gθ(zt,t)).P_\theta(z_s\mid z_t) =q\\!\left(z_s\mid z_t,\\;x=g_\theta(z_t,t)\right).

This is the plug-in parameterization. The ELBO evaluates the resulting transition, whereas EqDLM's CE directly evaluates the predicted clean solution. Our recognition proof requires the CE population optimum: each prediction must equal the corresponding clean-token posterior probability. The UDLM objective does not directly impose this requirement on its network output, so our e−1e^{-1} guarantee does not directly apply. Appendix D of the revised paper compares the shared features, corruption and time conditioning, training objectives, sampling updates, and stopping rules.

[W.3] The empirical evaluation is limited in scope. Experiments with more diverse benchmarks would strengthen the empirical evidence. Considering the potential resource limitation, this is a minor issue.

We use established problem-solving setups from recent diffusion language modeling work, with comparisons to MDLM, Duo, CANDI, FLM, S-FLM, FMLM+, and autoregressive models. We agree that more diverse benchmarks would strengthen the evaluation and appreciate your acknowledgment of the resource constraints. Extending to broader mathematical reasoning and code generation typically requires larger models with substantial pretraining and task-specific fine-tuning, making this difficult to complete within the rebuttal period. We consider this an important direction for future work.

Concluding remarks. We would be grateful if you could let us know whether our explanations have addressed your concerns. If so, we kindly ask you to consider increasing your rating. We are happy to discuss any remaining questions.

References

[1] Bai et al. Deep Equilibrium Models. NeurIPS, 2019.

[2] Bai et al. Stabilizing Equilibrium Models by Jacobian Regularization. ICML, 2021.

[3] Jiang et al. Equilibrium Language Models. ICLR, 2026.

[4] Schiff et al. Simple Guidance Mechanisms for Discrete Diffusion Models. ICLR, 2025.

I appreciate the author's detailed response. I found the comparison between EqDLM and UDLM very helpful. The limitation on the scope of the evaluation is also reasonable given the short rebuttal period; I will maintain my positive score.

Reviewer_3vHzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Equilibrium Diffusion Language Models, which frame the generation process as the task of finding and retaining valid solutions. The approach employs a time-independent model trained via standard denoising cross-entropy, using the self-retention probability of a complete candidate solution as the criterion for termination. The authors prove that, under conditions of global optimality and appropriate coverage, the self-retention probability for an invalid candidate is strictly less than e−1e^{-1}. Furthermore, they establish the convergence properties of this threshold-based stopping mechanism, as well as the absorption properties under more rigorous corruption conditions. Experimental evaluations cover the generation of discrete "Swiss roll" structures, complete Sudoku grids, and program generation for the GSM8K dataset using a model trained on TinyGSM.

Critical strengths
  1. The theoretical contributions are clear and insightful. Linking standard token-level denoising cross-entropy (CE) to the identification of valid candidates is the most valuable aspect of this paper. Furthermore, the paper provides a clear distinction between identification, reachability, and strict absorption, avoiding any conflation of these properties.

  2. The experimental evaluation of the theoretical mechanisms is robust. The "Swiss roll" task is used for validation across the full state space, while the Sudoku experiments employ independent calibration and test trajectories, with metrics covering erroneous termination, retention of valid states, and sample diversity. By increasing the stopping threshold, the model achieves a 99.99% validity rate with an average of 10.43 calls, demonstrating the tangible benefits of the stopping criterion.

  3. The results on program generation tasks are competitive, and the limitations are disclosed transparently. EqDLM achieves an accuracy of 63.61% on the GSM8K benchmark, which is comparable to the cited autoregressive (AR) greedy baseline. The authors also document the failure modes of standard categorical sampling—such as the acceptance of invalid outputs—and discuss the limitations regarding NFE (number of function evaluations) comparisons, which facilitates an accurate understanding of the method's scope of applicability.

Critical weaknesses
  1. The effectiveness of correctness identification and adaptive stopping on GSM8K remains insufficiently demonstrated. At the maximum budget setting, 99.92% of outputs pass the stopping threshold, yet the accuracy of these accepted outputs is only 63.58%. Furthermore, no trajectories terminate early during the annealing phase, and continued sampling after an extended annealing period typically adds only about one additional model call. Given that standard categorical sampling essentially fails, the current evidence primarily highlights the generative capability under annealing sampling; the independent contribution of "stay probability" to correctness identification and computational allocation has not been clearly quantified. While the authors acknowledge this limitation, it remains closely tied to the core mechanisms proposed in the paper.

  2. Evidence regarding actual computational efficiency is limited. Experiments involving a shared budget are compared based on the number of model calls; however, autoregressive (AR) models utilize a KV cache, whereas EqDLM processes the entire sequence in each call—meaning the cost per call differs between the two. Additionally, since MDLM continues for a fixed number of rounds even after all masks have been filled, its actual call-based efficiency may be underestimated. Furthermore, while the metrics for "multiple attempts" assess whether a correct answer exists within the pool of generated candidates, the model's ability to select the final, correct answer from this pool has not been evaluated. Consequently, the current results offer only limited support regarding actual runtime performance and the ability to deliver a single, accurate final answer.

Questions for authors
  1. Under the same checkpoint and annealing trajectory, how do the accuracy and average number of function calls compare across the following strategies: returning the candidate at the end of the annealing process, performing one or two additional fixed steps, and using the current threshold-based stopping criterion? Furthermore, could you provide a plot showing acceptance accuracy and coverage as a function of the threshold on the GSM8K dataset, to demonstrate whether the "stay score" effectively enables selective identification?

  2. Were the training targets for TinyGSM subjected to execution filtering and verification of answer correctness? Given that only 96,850 out of the 100,000 source programs mentioned in the appendix could be executed to produce a finite value, what is the relationship between these validated samples and the actual training data used?

  3. Could you provide a comparison of FLOPs or actual runtime under a few representative computational budgets—accounting for the KV cache in AR (Autoregressive) models and the batch execution behavior of EqDLM? For MDLM, it would also be useful to report the actual average number of function calls required when all mask positions are filled.

2 comments
AuthorsOpenReview ↗

Rebuttal part 1

Dear Reviewer 3vHz, thank you for your comments and for taking the time to review our work. Please find our responses below.

[W.1] The effectiveness of correctness identification and adaptive stopping on GSM8K remains insufficiently demonstrated. ... .

Annealed sampling reduces the mismatch between corrupted training inputs and model-generated states. Alternatively, we can train on model outputs. Our stopping analysis allows this change. For any corruption distribution covering all candidates, the population-optimal CE denoiser assigns every invalid candidate a stay probability below e−1e^{-1} (Theorem 1).

We change the distribution of training inputs while keeping the same CE objective and the same clean answers as targets. Given a clean answer xx, we first draw its corrupted version z0z_0 and sample one model update z1∼pθ(⋅∣z0)z_1\sim p_\theta(\cdot\mid z_0). The training input is now drawn with equal probability from these original corruptions and model-generated states:

LFT=12,E![−log⁡pθ(x∣z0)−log⁡pθ(x∣z1)].\mathcal L_{\mathrm{FT}}=\tfrac12\\,\mathbb E\\!\left[ -\log p_\theta(x\mid z_0)-\log p_\theta(x\mid z_1) \right].

We stop gradients through the sampled update and train both predictions to reconstruct xx.

Although this input distribution changes with the model, fixing the generator at any training step defines a particular corruption distribution. Including the original corruptions preserves coverage of all candidates. Theorem 1 therefore gives the same e−1e^{-1} bound for the population-optimal CE denoiser under each such distribution.

This follows SUNDAE’s one-step unrolling [1], discussed in Related Work. We introduce these inputs after 200k steps of denoising training. The resulting EqDLM model uses categorical sampling without annealing and stops at e−1e^{-1}. It reaches 58.30% accuracy at 46.97 mean calls with self-conditioning and 53.53% at 73.03 calls without it. With SC, 99.55% of trajectories stop within the 1,024-call limit. Stopping now determines the generation length throughout sampling, rather than adding a final check after a prescribed annealing schedule.

We also trained from scratch for 250k updates with the same CE loss on both corrupted and model-generated inputs. Under the same categorical sampler and e−1e^{-1} stopping rule, these models reach 27.98% accuracy with self-conditioning and 50.80% without it, compared with 58.30% and 53.53% when this training is applied only during fine-tuning, and require more model calls in both settings. Thus, adding model-generated states after denoising training works better than using them from the start. The analysis also covers other noising distributions with full input coverage.

See Section 4.3 for the method, Table 1 and Figures 6 and 10 for results, and Appendix G for the derivation and training details. Q.1 examines the stopping threshold.

[W.2 + Q.3] Evidence regarding actual computational efficiency is limited. ... Could you provide a comparison of FLOPs or actual runtime under a few representative computational budgets ... For MDLM, it would also be useful to report the actual average number of function calls ... Furthermore, while the metrics for "multiple attempts" assess whether a correct answer exists within the pool of generated candidates ... .

[Cost per call and latency] We measured H100 forward latency at batch sizes 1, 2, 4, and 8. AR uses KV caching, while EqDLM processes the full sequence. Times below are milliseconds per call.

Batch size1248
EqDLM + one-step FT (SC)15.8116.3417.4619.65
AR greedy, KV cache15.6415.8715.9015.89
AR sampling, KV cache14.8615.0915.0715.05

EqDLM with one-step unrolling needs fewer calls to produce an answer, offsetting its slightly higher cost per call. At batch size 1, it reaches 58.61% accuracy in 0.94 seconds on average, compared with 53.22% in 2.44 seconds for AR sampling. AR greedy achieves higher accuracy (63.46%), but takes 2.69 seconds. Thus, EqDLM offers a faster answer with higher accuracy than AR sampling, while greedy decoding trades additional time for higher accuracy. Figure 12 in Appendix G shows the full latency–accuracy curves for batch sizes 1, 2, 4, and 8.

[MDLM sampler efficiency] We now stop MDLM immediately when all masks are filled and use the resulting mean NFE throughout the GSM8K comparisons, with and without SC. At a configured limit of 1,024 calls, mean NFE is 66.01 with SC and 76.28 without SC. Every decoded answer matches the original fixed-round evaluation, so accuracy is unchanged. See Appendix J of the revised paper (Table 12 and Figure 14) for the measurements.

References

[1] Savinov et al. Step-unrolled Denoising Autoencoders for Text Generation. ICLR, 2022.

AuthorsOpenReview ↗

Rebuttal part 2

[Final-answer selection] On all 1,319 GSM8K questions, we use EqDLM to select by highest stay probability. AR sampling (T=1T=1) has no built-in selection rule, so we use maximum sequence log probability without length normalization. Neither uses reference answers or a verifier. Only attempts completed within the shared budget are eligible. EqDLM completes at its stopping threshold or call limit. Unfinished attempts consume budget, and empty pools count as incorrect.

We compare original EqDLM with annealed categorical sampling and EqDLM + one-step FT (W.1) with categorical sampling. Both use e−1e^{-1} stopping and are evaluated with and without SC.

Total generation NFE per problem

Selected-answer accuracy (%)642561,0242,048
EqDLM, no SC (Annealed Categorical)35.4140.6443.2941.32
EqDLM, SC (Annealed Categorical)52.0854.2854.0652.69
EqDLM + one-step FT, no SC48.6757.6260.8860.05
EqDLM + one-step FT, SC53.8362.9364.0664.06
AR sampling, max log probability0.0053.9063.2364.90
AR greedy (T=0T=0)0.0061.9463.0863.08

EqDLM with one-step unrolling and SC leads at 64–1,024 NFE, and AR sampling at 2,048. Neither AR variant completes a correct program within 64 calls. Budgets exclude AR's offline rescoring, so this is not a latency comparison.

[Q.1] Under the same checkpoint and annealing trajectory, how do the accuracy and average number of function calls compare across the following strategies: returning the candidate at the end of the annealing process, performing one or two additional fixed steps, and using the current threshold-based stopping criterion? Furthermore, could you provide a plot showing acceptance accuracy and coverage as a function of the threshold on the GSM8K dataset, to demonstrate whether the "stay score" effectively enables selective identification?

(1) Fixed steps versus threshold stopping after annealing. All four requested strategies use the same checkpoint and shared 1,022-update annealing trajectories on all 1,319 GSM8K questions.

StrategyAccuracy (%)Mean NFE
End of annealing46.471022
One additional categorical step47.381023
Two additional categorical steps47.691024
Categorical updates until stay >e−1>e^{-1}47.611023.07

After annealing, threshold stopping performs almost identically to two fixed updates (47.61% versus 47.69%), confirming your observation in W.1. For EqDLM with one-step unrolling, however, categorical sampling needs no annealing and reaches 54.36% accuracy at 72.45 mean calls with the same e−1e^{-1} threshold.

(2) Acceptance accuracy and coverage. Figure 15 in Appendix K shows the threshold curves. Coverage is the fraction of questions with accepted answers, and acceptance accuracy is their correctness rate. Overall accuracy includes answers returned at the call limit. Both models are evaluated without self-conditioning: the original EqDLM uses annealing, and EqDLM with one-step unrolling uses categorical sampling. Each sweep uses the same trajectories and accepts the first threshold crossing. Appendix K gives the protocol. These runs are separate from part (1) and Table 1.

Model and samplerThresholdOverall accuracy (%)Acceptance accuracy (%)Coverage (%)
Original, annealede−1e^{-1}46.0246.02100.00
0.990.9946.3246.6397.88
EqDLM + one-step FT, categoricale−1e^{-1}54.3654.7698.71
0.70.756.6360.7391.51
0.80.856.1068.6172.71
0.90.952.0181.6028.43
0.950.9550.2792.115.76

Raising the threshold barely improves acceptance accuracy for the original model. For EqDLM with one-step unrolling, acceptance accuracy increases from 54.76% at e−1e^{-1} to 81.60% at 0.90.9, while coverage falls from 98.71% to 28.43%. Thus, stay probability allows more selective acceptance for this variant. Overall accuracy peaks at 56.63% at 0.70.7 (full sweep: Appendix K, Table 14).

[Q.2] Were the training targets for TinyGSM subjected to execution filtering and verification of answer correctness? Given that only 96,850 out of the 100,000 source programs mentioned in the appendix could be executed to produce a finite value, what is the relationship between these validated samples and the actual training data used?

We train on the public TinyGSM release, which is the standard corpus for this TinyGSM + GSM8K setup. The public files still contain a small fraction of targets that do not produce to a finite number we use that standard published data without filtering.

Concluding remarks. We would be grateful if you could let us know whether our explanations have addressed your concerns. If so, we kindly ask you to consider increasing your rating. We are happy to discuss any remaining questions.

Reviewer_nLwGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Equlibrium diffusion language models which formulate generation as reaching a particular valid solution or set of valid solutions instead of modelling the likelihood of the training data. The paper formulates and provides theoretical conditons for which the process recognizes these valid solutions and eventually reaches them.The results seem to be strong on Sudoku and GSM8K, where it reaches comparable performance to an autoregressive model.

Critical strengths
  1. The formulation is novel and and the focus for finding a correct solution is a natural goal for things like program synthesis.

  2. The loss proposed is simple within a single denoising loss providing both a signal to revise the candidate solutions and for when to stop

  3. The results are strong on Sudoku and GSM8K.

Critical weaknesses
  1. The paper's evaluation scope is quite limited and narrow, with Sudoku being somewhat of a toy setting. Results on more general mathematical reasoning and code generation would strengthen the paper.

  2. The method does match autoregressive perfromnace but requires significant computation with about 1024 model calls, which makes it unclear whether this method does have an efficiency advantage over autoregressive models.

  3. Direct categorical sampling fails quite badly on GSM8k, and additional tools like noise annealing are required in practice to get good results, which puts into question whether this method can generalise to harder settings.

  4. Stopping probability is not necessarily a good signal for correctness, as the paper shows that incorrect programs can bypass this stop check.

Questions for authors

Please address weakenesses, would be happy to increase my score based on the author's response.

After Rebuttal

My questions have been addressed through both the additional results and clarifications provided. I still believe that results on mathematical reasoning, and coding would strengthen the paper, even with a model without extensive pretrainig, though I do recognize that other papers also lack this evaluation. I am raising my score from 2 to 3.

3 comments
AuthorsOpenReview ↗

Rebuttal part 1

Dear Reviewer nLwG, thank you for your comments and for taking the time to review our work. Please find our responses below.

[W.1] The paper's evaluation scope is quite limited and narrow, with Sudoku being somewhat of a toy setting. Results on more general mathematical reasoning and code generation would strengthen the paper.

We use established problem-solving setups from recent diffusion language modeling work, with comparisons to MDLM, Duo, CANDI, FLM, S-FLM, FMLM+, and autoregressive models. We agree that broader mathematical reasoning and code generation would strengthen the evaluation. However, extending to these tasks typically requires larger models with substantial pretraining and task-specific fine-tuning, making this a significant additional training effort beyond our current experimental scope. We consider this an important direction for future work.

[W.3] Direct categorical sampling fails quite badly on GSM8k, and additional tools like noise annealing are required in practice to get good results, which puts into question whether this method can generalise to harder settings.

We proposed annealed sampling to reduce mismatch between training distribution of inputs and inference distribution produced by the model itself. Another option to fix this mismatch is to train on model outputs. Our stopping analysis allows this change in training. For any corruption distribution covering all candidates, the population-optimal CE denoiser assigns every invalid candidate a stay probability below e−1e^{-1} (Theorem 1).

We change the distribution of training inputs while keeping the same CE objective and the same clean answers as targets. Given a clean answer xx, we first draw its corrupted version z0z_0 and sample one model update z1∼pθ(⋅∣z0)z_1\sim p_\theta(\cdot\mid z_0). The training input is now drawn with equal probability from these original corruptions and model-generated states:

LFT=12,E![−log⁡pθ(x∣z0)−log⁡pθ(x∣z1)].\mathcal L_{\mathrm{FT}}=\tfrac12\\,\mathbb E\\!\left[ -\log p_\theta(x\mid z_0)-\log p_\theta(x\mid z_1) \right].

We stop gradients through the sampled update, so both terms train the denoiser to reconstruct xx using the same CE loss.

Although this input distribution changes with the model, fixing the generator at any training step defines a particular corruption distribution. Including the original corruptions preserves coverage of all candidates. Theorem 1 therefore gives the same e−1e^{-1} bound for the population-optimal CE denoiser under each such distribution.

Training on model-generated states in this way follows the one-step unrolling approach of SUNDAE [1], discussed in Related Work. We apply this change of input distribution after 200k steps of our original denoising training. The resulting EqDLM model uses categorical sampling without annealing and stops at e−1e^{-1}. It reaches 58.30% accuracy at 46.97 mean calls with self-conditioning and 53.53% at 73.03 calls without it. Thus, annealing is not required for effective generation on GSM8K, and the model can use only categorical sampling together with the stopping rule.

We also tested whether to include model-generated inputs from the start: we trained models from scratch for 250k updates using the same CE loss on both original corruptions and model-generated states throughout. Under the same categorical sampler and e−1e^{-1} stopping rule, these models reach 27.98% accuracy with self-conditioning and 50.80% without it, compared with 58.30% and 53.53% when this training is applied only during fine-tuning, and require more model calls in both settings. In these experiments, learning from corrupted data first and then adding model-generated states works better than using the latter from the beginning. Our theoretical analysis is not limited to these two cases, one can also use other noising distributions if they cover all possible inputs.

In the revised version, Section 4.3 describes the training procedure of the added experiments, Table 1 and Figures 6 and 10 compare the results, and Appendix G gives the derivation and training details.

References

[1] Savinov et al. Step-unrolled Denoising Autoencoders for Text Generation. ICLR, 2022.

AuthorsOpenReview ↗

Rebuttal part 2

[W.4] Stopping probability is not necessarily a good signal for correctness, as the paper shows that incorrect programs can bypass this stop check.

Under the assumptions of Theorem 1, the population-optimal CE denoiser assigns every invalid candidate a stay probability below e−1e^{-1}. Incorrect programs passing the threshold therefore demonstrate that a particular model is not ideally trained (under the assumption that train and test data distributions are the same). In the previous answer, with a new model trained also on its outputs, we show that it is possible to make the model capable enough to distinguish correct and invalid answers to compete with and outperform other diffusion and autoregressive baselines. In this sense, our framework does not differ from diffusion or autoregressive models, which also guarantee 100% only if they are population-optimally trained.

[W.2] The method does match autoregressive perfromnace but requires significant computation with about 1024 model calls, which makes it unclear whether this method does have an efficiency advantage over autoregressive models.

Thank you for raising this point. The submitted paper compared accuracy with published AR results without reporting the number of calls AR needs. We have now evaluated accuracy and mean NFE under the same maximum number of calls.

(1) Accuracy under a hard call limit. The table reports one answer per question on all 1,319 GSM8K questions. Both EqDLM variants use SC. Annealed uses the original model, while EqDLM with one-step unrolling uses categorical sampling without annealing, which we discussed in previous answers. All methods return an answer by the stated limit, including any stopping checks in the call count. EqDLM may stop earlier at e−1e^{-1}, and AR at EOS. We score the returned output even if it is truncated, rather than discarding outputs without EOS.

Max. NFEEqDLM, Annealed: Acc. (%)EqDLM, Annealed: Mean NFEEqDLM + one-step FT: Acc. (%)EqDLM + one-step FT: Mean NFEAR greedy: Acc. (%)AR greedy: Mean NFEAR sampling: Acc. (%)AR sampling: Mean NFE
3249.6631.0741.3225.380.0032.000.0032.00
6457.1663.0353.3733.970.0064.000.0064.00
12860.12127.0256.8638.7124.11122.2421.15122.02
25661.33255.0257.9241.8662.40156.0553.68157.36

At limits of 32–128 calls, both EqDLM variants outperform both AR variants. EqDLM with one-step unrolling gives lower accuracy than Annealed at the same maximum NFE but uses substantially fewer calls on average through earlier stopping. AR generates one token per call and often cannot complete a program under these limits, whereas EqDLM updates the entire answer in parallel. AR greedy becomes more accurate at the limit of 256 calls.

For a concrete comparison, Annealed EqDLM reaches 60.12% accuracy at 127.02 mean NFE under a 128-call limit, while AR greedy reaches 62.40% at 156.05 mean NFE under a 256-call limit.

(2) Cost per call and latency. Model calls have different costs for diffusion and AR due to KV caching. On H100 at batch size 1, a forward pass takes 15.81 ms for EqDLM with one-step unrolling versus 14.86–15.64 ms for AR. At batch size 8, EqDLM takes 19.65 ms per call, compared with 15.89 ms for AR greedy and 15.05 ms for AR sampling, an overhead of 23.6% and 30.5%, respectively.

The smaller number of calls allows this variant to achieve a better latency–accuracy trade-off than AR sampling across the tested batch sizes and than AR greedy at low latency. AR greedy reaches a higher maximum accuracy. See Figure 12, Appendix G of the revised paper for the measured curves at batch sizes 1, 2, 4, and 8, and our combined response to reviewer 3vHz, W.2 + Q.3, for further details.

Concluding remarks. We would be grateful if you could let us know whether our explanations have addressed your concerns. If so, we kindly ask you to consider increasing your rating. We are happy to discuss any remaining questions.

Thank you for the response. My questions have been addressed through both the additional results and clarifications provided. I still believe that results on mathematical reasoning, and coding would strengthen the paper, even with a model without extensive pretrainig, though I do recognize that other papers also lack this evaluation. I am raising my score to a 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General answer

Dear reviewers, thank you for your thoughtful feedback. We appreciate your recognition of the motivation to find valid solutions rather than match their frequencies (nLwG, nNMA), the simplicity of using one denoiser for both generation and stopping (nLwG, nNMA), and the theoretical conditions for recognizing, reaching, and preserving valid solutions (3vHz, nNMA). We are also glad that reviewers nLwG and 3vHz highlighted the strong Sudoku and competitive GSM8K results. Please find the answers to your shared questions below.

(1) Generation without annealing on GSM8K. We used an annealing sampler to reduce mismatch between the input distributions during training and inference. An alternative way to reduce this mismatch is to train the model also on samples produced by the model itself. Such a change to the training distribution is still covered by our stopping guarantee at the population optimum, provided the corruption distribution covers all states. EqDLM with one-step unrolling and self-conditioning reaches 58.30% GSM8K accuracy at 46.97 mean calls without annealing, exceeding AR sampling (53.9%) and the evaluated diffusion baselines. See Table 1 and Figures 6 and 10 of the revised paper for the results, Section 4.3 for the method, and Appendix G for details.

(2) Computational efficiency. We added latency-accuracy curves for batch sizes 1, 2, 4, and 8, comparing EqDLM with AR sampling and greedy decoding using KV caching. EqDLM achieves a better latency-accuracy trade-off than AR sampling across all tested batch sizes and outperforms AR greedy at low latency. See Figure 12 in Appendix G of the revised paper for the curves and evaluation settings.

Revised paper. Changes are highlighted in blue.

  • Training on model-generated states. We added the objective and its connection to the stopping analysis in Section 4.3, results in Table 1 and Figures 6 and 10, and the derivation, training details, and comparison with training from scratch in Appendix G.
  • Latency–accuracy comparison. We added curves for batch sizes 1, 2, 4, and 8 against AR sampling and greedy decoding with KV caching. The results and evaluation settings are provided in Appendix G, Figure 12 of the revised paper.
  • Positioning relative to equilibrium-based language models. We expanded Related Work (Appendix I) with a discussion of Deep Equilibrium Models, Jacobian regularization, and Equilibrium Language Models, distinguishing hidden-state fixed points from equilibrium and stopping in generated sequences.
  • Comparison with uniform-state diffusion. Appendix D compares EqDLM with UDLM: their shared features, training objectives, sampling updates, and stopping rules. It also explains why our CE-based recognition guarantee does not directly apply to UDLM predictions.
  • MDLM early stopping. We updated all GSM8K comparisons to count calls until all masks are filled, with and without SC. Appendix J reports the measurements and verifies that accuracy is unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.