acceptodds
Under review as a conference paper at ICLR 2027

Unified Energy for Invariant and Independent Decoding in Diffusion Language Models

Abstract

Diffusion Language Models (DLMs) enable parallel text generation by iteratively denoising a full sequence, offering attractive flexibility compared to auto-regressive (AR) decoding. However, existing methods fail to fully capture token relationships, leading to a performance gap relative to AR baselines, especially as the degree of parallelism increases. In this paper, we give a systematic analysis of the gap, identifying three key factors: (i) model capacity, (ii) dependency, and (iii) invariance. To address these issues, we first propose an invariant energy (Inv-E) together with an effective sampling-based estimator to handle the invariance issue. By further combining with the independent energy (Ind-E), we obtain a unified energy (Uni-E), that accounts for all these factors. Uni-E enjoys a unique advantage: it can be computed exactly without sampling-based partition estimation. Besides, Uni-E is model agnostic and can therefore be scaled to models of arbitrary size. We further prove that Uni-E can correct the distribution shift caused by dependency and invariance. Extensive experiments across Diffusion Language Models (DLMs) and Diffusion Large Language Models (DLLMs) demonstrate the effectiveness of the proposed Uni-E.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Model, Energy-based Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_77yyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Unified Energy (Uni-E) to address the degradation in generation quality caused by insufficient modeling of token relationships during parallel decoding in diffusion language models. Through a KL divergence analysis, the authors identify three factors contributing to distribution shifts: model capacity, token dependency, and invariance across decoding steps. Based on this analysis, they introduce an Invariant Energy (Inv-E) that uses an AR proxy model to assess the influence of future context, and combine it with the existing Independent Energy (Ind-E) to improve parallel decoding through energy-based candidate reweighting. The authors also derive an analytical expression for the partition function. Experiments on both conventional and large-scale diffusion language models demonstrate improvements in generation quality, reduced performance degradation at higher levels of parallelism, and competitive inference efficiency.

Critical strengths

I really appreciate the paper's clear and insightful analysis of parallel decoding limitation with corresponding solutions of two factors. this has inspired me a lot, And I find the following two aspects particularly compelling.

  1. The paper decomposes the distribution shift in DLM parallel decoding into three factors through a KL divergence analysis: model capacity, token dependency, and invariance across decoding steps. This provides a valuable perspective for understanding generation quality degradation during parallel decoding. Building on this analysis, the authors introduce Inv-E and combine it with the existing Ind-E to construct a unified energy-based correction framework. The method is well motivated, and the ablation studies demonstrate the complementary benefits of the two energy components in improving generation quality.
  2. The paper provides comprehensive experimental evaluations, demonstrating improvements in language modeling and generation quality on conventional DLMs and extending Uni-E to large-scale models such as LLaDA-8B and Dream-7B. The evaluation covers diverse tasks, including mathematical reasoning, code generation, and knowledge-based benchmarks. Furthermore, component ablations, hyperparameter analyses, and accuracy–throughput comparisons provide substantial empirical evidence of the method's effectiveness across different model scales and decoding configurations.
Critical weaknesses

There are still weakness in this paper, and I hope this can be addressed by the authors.

  1. Although the paper discusses and compares against existing methods such as EDLM, it lacks sufficient discussion of closely related work, particularly Diffusion Language Model Parallel Decoding via Product-of-Experts Bridge (PoE-Bridge). These methods similarly leverage probabilistic information from AR models to correct the parallel generation distribution of DLMs. The authors should more clearly distinguish Uni-E from existing distribution correction approaches, particularly regarding the unique contributions of the proposed invariance modeling. This omission makes the novelty and positioning of the paper less convincing.
  2. While the proposed KL divergence decomposition provides an insightful perspective, several key assumptions in the subsequent theoretical analysis are not sufficiently justified. Specifically, the sampling estimation in Lemma 1 relies on the additivity of joint probability contributions, which does not necessarily hold when complex token dependencies exist. Furthermore, Lemma 2 claims that Uni-E strictly reduces the KL divergence from the ground-truth distribution, but the assumed correct energy ranking is insufficient to guarantee such a reduction. The paper also does not clearly establish how these theoretical guarantees extend to the finite-candidate, energy-based selection strategy used in practice. Consequently, the current proofs do not fully support the general theoretical guarantees claimed by the authors.
Questions for authors

After reading this paper, I have following questions:

  1. Could the authors clarify the key differences between Uni-E and closely related AR-guided distribution correction methods, particularly PoE-Bridge? What specific advantages does the proposed invariance modeling offer beyond existing approaches?
  2. Could the authors clarify the assumptions underlying Lemmas 1 and 2? Specifically, how is the additivity assumption in the invariance sampling estimation justified when token dependencies exist? Furthermore, why does correct energy ranking necessarily guarantee a strict reduction in KL divergence?
  3. Could the authors provide more direct experimental evidence that Inv-E specifically mitigates invariance violations across decoding steps? For instance, analyzing inconsistencies between early-decoded tokens and subsequently revealed context would help substantiate the proposed mechanism beyond the overall improvements reported in the ablation studies.
  4. The theoretical analysis considers an energy-reweighted probability distribution, while the actual inference algorithm selects the lowest-energy candidate from a finite set. Could the authors clarify whether the theoretical guarantees remain valid under this practical selection strategy and, if so, under what conditions? Helpful explanations will help me to update my rate.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 77yy

Response to Q1/W1: Relation to PoE-Bridge and other AR-guided corrections

We thank the reviewer for pointing to PoE-Bridge (Shi et al., ICML 2026), a recent work that we will discuss in the revision. Both methods use an AR model and multiple DLM drafts, but they differ fundamentally in goal, in the error they model, and in mechanism.

PoE-BridgeUni-E (ours)
Target distributionAR model (DLM accelerates AR)DLM, with AR as a small proxy
Error modeledwithin-step dependencydependency and cross-step invariance
Mechanismrejection sampling + importance samplingone batched energy scoring per step

1. Target distribution. PoE-Bridge drafts with the DLM and corrects the drafts toward the AR target. Its quality is therefore bounded by the AR model, and it reports reaching at least 95% of the AR model's performance. In Uni-E, the DLM remains the target model, and the AR model is only a lightweight proxy (Qwen2.5-0.5B for LLaDA-8B). As a result, Uni-E can exceed its own proxy: over multiple seeds, LLaDA-8B + Uni-E reaches 39.6 vs. 33.6 on MATH500, 81.6 vs. 41.7 on GSM8K, and 65.9 vs. 46.9 on MMLU (Uni-E vs. Qwen2.5-0.5B). This shows that Uni-E combines the strengths of both models rather than reproducing the AR model.

2. Invariance modeling (our unique contribution). Existing AR-guided corrections, including EDLM and PoE-Bridge, correct the joint distribution of the tokens decoded in the current step, i.e., the dependency factor. To our knowledge, they do not model whether a committed token will remain valid once the future context is revealed. Inv-E addresses exactly this question. It scores each candidate together with its lookahead xs\mathbf{x}_s and penalizes commitments whose plausibility changes once more context is revealed. Part 3 gives direct evidence that this reduces invariance violations.

3. Mechanism and cost. PoE-Bridge relies on rejection sampling with token-level verification. Uni-E needs one batched energy evaluation over kk candidates per step. Its partition function is available in closed form, so no MCMC or sampling-based estimation is required.

AuthorsOpenReview ↗

Response to Reviewer 77yy

Response to Q2/W2: Assumptions of Lemmas 1 and 2

We thank the reviewer for these precise questions. We give rigorous versions of both lemmas, which make the assumptions explicit. These revision will be further updated in the paper.

Lemma 1: additivity is not required. Let c=xt+1c=\mathbf{x}_{t+1}, let Vi−\mathcal{V}_i^- be the not-yet-decoded invariant part of token ii, and let ZsZ_s be a uniformly random subset of size rr of the n=x+yn=x+y future positions. Stated in expectation,

EZsEπ[log⁡π(xZs∣xi,c)π(xZs∣c)]=βr Iπ(xi;xVi−∣c),βr=∑k≤rak∑k≤nak∈[0,1],\mathbb{E}_{Z_s}\mathbb{E}_\pi\Big[\log\frac{\pi(\mathbf{x}^{Z_s}\mid\mathbf{x}^i,c)}{\pi(\mathbf{x}^{Z_s}\mid c)}\Big]=\beta_r\thinspace I_\pi(\mathbf{x}^i;\mathbf{x}^{\mathcal{V}_i^-}\mid c),\qquad\beta_r=\frac{\sum_{k\le r}a_k}{\sum_{k\le n}a_k}\in[0,1],

where ak=EσI(xi;xσ(k)∣xσ(<k),c)≥0a_k=\mathbb{E}_\sigma I(\mathbf{x}^i;\mathbf{x}^{\sigma(k)}\mid\mathbf{x}^{\sigma(\lt k)},c)\ge 0 and σ\sigma is a random ordering of the future positions.

Proof.

  1. The left-hand side is EZsI(xi;xZs∣c)\mathbb{E}_{Z_s}I(\mathbf{x}^i;\mathbf{x}^{Z_s}\mid c).
  2. Taking ZsZ_s as the first rr elements of σ\sigma, the chain rule of mutual information gives ∑k≤rak\sum_{k\le r}a_k.
  3. Tokens outside Vi−\mathcal{V}_i^- contribute zero by the definition of the invariant part. □\square

This holds for arbitrary token dependencies. Dependencies only change the value of βr\beta_r; βr\beta_r is always non-decreasing in rr, with βn=1\beta_n=1. Additivity is needed only for the specific value βr=r/(x+y)\beta_r=r/(x+y). Hence the estimator remains a valid, monotone proxy of the invariance term in general.

Lemma 2: an exact identity replaces the ranking argument. We agree that a correct energy ranking alone does not control probability magnitudes. We therefore replace the ranking argument with an exact identity. The Uni-E corrected kernel has the closed form pϕ,θ(x0∣xt,xs)=pAR(x0∣xs)p_{\phi,\theta}(\mathbf{x}_0\mid\mathbf{x}_t,\mathbf{x}_s)=p_{AR}(\mathbf{x}_0\mid\mathbf{x}_s). Let MsM_s be the positions still masked in xs\mathbf{x}_s and Δ\Delta those revealed between tt and ss. Then

E KL(π(⋅∣xs)∥∏i∈Mspθ(xi∣xt))=E TCπ(xMs∣xs)⏟dependency+∑i∈Ms[Iπ(xi;xΔ∣xt)⏟invariance+E KL(π(xi∣xt)∥pθ(xi∣xt))⏟capacity],\mathbb{E}\thinspace \mathrm{KL}\Big(\pi(\cdot\mid\mathbf{x}_s)\Big\Vert\prod_{i\in M_s}p_\theta(\mathbf{x}^i\mid\mathbf{x}_t)\Big)=\underbrace{\mathbb{E}\thinspace \mathrm{TC}_\pi(\mathbf{x}^{M_s}\mid\mathbf{x}_s)}_{\text{dependency}}+\sum_{i\in M_s}\Big[\underbrace{I_\pi(\mathbf{x}^i;\mathbf{x}^{\Delta}\mid\mathbf{x}_t)}_{\text{invariance}}+\underbrace{\mathbb{E}\thinspace \mathrm{KL}\big(\pi(\mathbf{x}^i\mid\mathbf{x}_t)\Vert p_\theta(\mathbf{x}^i\mid\mathbf{x}_t)\big)}_{\text{capacity}}\Big],

while Uni-EDLM incurs only εAR=E KL(π(⋅∣xs)∥pAR(⋅∣xs))\varepsilon_{AR}=\mathbb{E}\thinspace \mathrm{KL}(\pi(\cdot\mid\mathbf{x}_s)\Vert p_{AR}(\cdot\mid\mathbf{x}_s)). Here TC denotes total correlation.

Proof. The KL to a product distribution equals TC plus the sum of the marginal KLs. Splitting each marginal log-ratio through π(xi∣xt)\pi(\mathbf{x}^i\mid\mathbf{x}_t) gives the mutual-information term. □\square

Revised Lemma 2. The KL is strictly reduced if and only if εAR\varepsilon_{AR} is smaller than the right-hand side. In particular, the reduction is strict in non-independent or non-invariant scenarios whenever the proxy is accurate. The required assumption, an accurate proxy, is now explicit.

AuthorsOpenReview ↗

Response to Reviewer 77yy

Response to Q3/Q4: Direct evidence of invariance, and finite-candidate selection

Q3: Direct evidence. Following the reviewer's suggestion, we directly measured invariance violations across decoding steps. After generation, we checked each early-decoded token against the fully revealed context. The flip rate is the fraction of these tokens that an oracle AR model would change given the complete sequence. A lower flip rate means that committed tokens remain consistent with the context revealed later, i.e., fewer invariance violations.

MethodFlip rate ↓
MDLM0.40
MDLM + Ind-E0.32
MDLM + Inv-E0.28
MDLM + Uni-E0.24
  • Inv-E alone reduces the flip rate more than Ind-E (0.28 vs. 0.32), although Ind-E corrects the joint distribution within each step. This directly confirms that Inv-E targets cross-step invariance rather than only within-step dependency.
  • The two energies are complementary: Uni-E reduces the flip rate by 40% relative to MDLM (0.40 → 0.24). This mirrors their complementary effect on Gen PPL in our ablation (Fig. 2).

We will add this analysis to the revision, together with the qualitative cases in Appendix A.12 (Figs. 5 and 7).

Q4: Finite-candidate selection. The guarantee in Lemma 2 concerns the energy-reweighted distribution pϕ,θp_{\phi,\theta}, and the practical algorithm approximates it as follows.

  • Resampling variant (SNIS). Drawing one of kk candidates with weights ∝exp⁡(−E)\propto\exp(-E) is sampling-importance-resampling. Its output distribution converges to pϕ,θp_{\phi,\theta} as k→∞k\to\infty, so the guarantee holds in the limit and is approached as kk grows.
  • Arg-min variant (our default). This is the zero-temperature limit, which targets the mode of pϕ,θp_{\phi,\theta}. It satisfies a simple exact property: the expected energy of the selected candidate, E[min⁡j≤kEj]\mathbb{E}[\min_{j\le k}E_j], is non-increasing in kk.
  • Empirical support. We added a comparison of the two variants. Both clearly outperform EDLM and the backbone, and they differ mostly within one standard deviation on LLaDA. For example, on MATH500 at k=3k=3, arg-min gives 39.6 ± 0.9 and resampling gives 39.4 ± 1.1. Increasing kk from 3 to 6 further improves performance (40.0 and 40.2, respectively), consistent with the approximation tightening as kk grows.
Reviewer_VmVvOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper extends EDLM with an additional energy term intended to improve consistency as more tokens are revealed. It generates several candidate completions, scores them using diffusion and autoregressive models, and selects the lowest-energy candidate. The method supports both training-free inference and NCE fine-tuning. The authors claim theoretical guarantees and report improvements in language modeling.

Critical strengths

Reported improvements over the closest baseline. Evaluations across model scales, together with runtime comparisons and ablations, provide encouraging evidence of practical usefulness. The proposed decoding heuristic shows meaningful empirical promise beyond the existing EDLM framework.

Critical weaknesses
  1. Invalid KL decomposition Equation (3) and in line 146. As far as I undersatand the first equality replaces chain-rule conditionals with conditionals given all other tokens. Their product generally does not equal the joint probability. Averaging over decoding orders also yields an upper bound, rather than the asserted equality. Consequently, if I am not mistaken the proposed three-factor decomposition does not establish a separate “invariance” error.

  2. Unjustified sampling estimator in Lemma 1. The additional independence and unchanged-marginal claims in lines 856 do not follow from Eq. (19). Furthermore, Eqs. (20)–(21) replace pointwise joint log-ratios with sums of marginal KL divergences without the required expectation and factorization. Consequently, the conclusion in Eq. (25) is not established. The claimed equality can fail undermining the justification for the invariance estimator.

  3. Unsupported improvement guarantee in Lemma 2. The crucial transition from the ranking in Equation (31) and normalization in Equations (32)–(33) to the weighted log-likelihood inequality in Equation (34) is unjustified. Correct ranking and normalization do not control probability magnitudes: reweighting can overconcentrate probabilities and increase KL divergence. Consequently, the strict KL reduction claimed in Eq. (35) does not follow. Moreover, the energy ordering in Eq. (30) is asserted rather than derived from the proposed energy. Additional assumptions and a different proof are needed to establish an improvement guarantee.

These errors undermine the central theoretical claims. Moreover, with consistent AR conditionals, the unnormalized Inv-E simplifies to

Einv=log⁡pθ(xs∣xt).E_{\mathrm{inv}}=\log p_{\theta}(x_s\mid x_t).

Thus, the unnormalized Uni-E becomes

EUni=EEDLM+log⁡pθ(xs∣xt).E_{\mathrm{Uni}}=E_{\mathrm{EDLM}}+\log p_{\theta}(x_s\mid x_t).

What remains is an additional DLM score whose interpretation as an invariance correction remains unproven.

Questions for authors

Clarification of the theoretical claims. I may be misunderstanding the notation or overlooking an assumption. Could the authors identify any errors in my reasoning concerning Eq. (3), the sampling argument in Lemma 1, and the transition from Eqs. (31)–(33) to Eq. (34) in Lemma 2? In particular, are there additional assumptions that justify these steps?

If these concerns are valid, could the authors provide corrected statements and proofs, and clarify whether the invariance estimator and KL-improvement guarantee still hold? A rigorous resolution of these points would materially affect my assessment.

5 comments
AuthorsOpenReview ↗

Response to Reviewer VmVv

Response to W1: The KL decomposition (Eq. 3)

We sincerely thank the reviewer for the careful check of our derivations and for the explicit invitation to clarify them. We address each point below. None of the clarifications change the algorithm or the empirical results.

Theoretical basis. Our analysis follows the framework of Lavenant & Zanella (2025) [1], as stated in Sec. 3 (L126). We adopt from [1]:

  • the modeling of a DLM as a decoding planner μϕ\mu_\phi plus a distribution learner pθp_\theta;
  • the split of the sampling error into a learning error and a factorization error caused by decoding several tokens independently in one step.

Our capacity factor (Eq. 4) is the learning error of [1], and our dependency factor (Eq. 5) is the factorization error of [1].

First equality of Eq. (3). We thank the reviewer for pointing this out. As in [1], the decomposition should use chain-rule conditionals π(xtZt∣xt+1)\pi(\mathbf{x}_t^{Z_t}\mid\mathbf{x}_{t+1}), not full conditionals; the product of full conditionals is a pseudo-likelihood. Since the DLM factorizes within a step, the correct form of Eq. (3), for a fixed order Ψ\Psi, is

KL(π∥PΨ)=∑t[E TCπ(xZt∣xt+1)⏟dependency (Eq. 5)+∑i∈ZtE KL(π(xi∣xt+1)∥pθ(xi∣xt+1))⏟capacity (Eq. 4)].\mathrm{KL}(\pi\Vert P_\Psi)=\sum_t\Big[\underbrace{\mathbb{E}\thinspace \mathrm{TC}_\pi(\mathbf{x}^{Z_t}\mid\mathbf{x}_{t+1})}_{\text{dependency (Eq. 5)}}+\sum_{i\in Z_t}\underbrace{\mathbb{E}\thinspace \mathrm{KL}\big(\pi(\mathbf{x}^i\mid\mathbf{x}_{t+1})\Vert p_\theta(\mathbf{x}^i\mid\mathbf{x}_{t+1})\big)}_{\text{capacity (Eq. 4)}}\Big].

Here TCπ(xZ∣c)=E[log⁡π(xZ∣c)∏i∈Zπ(xi∣c)]\mathrm{TC}_\pi(\mathbf{x}^{Z}\mid c)=\mathbb{E}\Big[\log\frac{\pi(\mathbf{x}^{Z}\mid c)}{\prod_{i\in Z}\pi(\mathbf{x}^i\mid c)}\Big] is the total correlation of the co-decoded tokens, i.e., the expectation of the first term of Eq. (5). We will fix the notation.

Averaging over orders (L146). We agree. By the log-sum inequality, KL(π∥Pdlm)≤EΨ KL(π∥PΨ)\mathrm{KL}(\pi\Vert P_{dlm})\le\mathbb{E}_\Psi\thinspace \mathrm{KL}(\pi\Vert P_\Psi). We will state L146 as an upper bound, and all subsequent results hold per order Ψ\Psi.

Role of invariance. Invariance governs the dependency error of parallel decoding, in the following precise sense. Let MtM_t be the masked positions at step tt, and define the non-invariance of token ii as

Iti:=Iπ(xi;xMt∖i∣xt+1)=Eπ[log⁡π(xi∣x0−xi)π(xi∣xt+1)],\mathcal{I}_t^i:=I_\pi(\mathbf{x}^i;\mathbf{x}^{M_t\setminus i}\mid\mathbf{x}_{t+1})=\mathbb{E}_\pi\Big[\log\frac{\pi(\mathbf{x}^i\mid\mathbf{x}_0-\mathbf{x}^i)}{\pi(\mathbf{x}^i\mid\mathbf{x}_{t+1})}\Big],

the expectation of the token-level invariance ratio in Eq. (6). Iti=0\mathcal{I}_t^i=0 means exactly that xi\mathbf{x}^i is invariant.

  • Proposition A (invariance bounds the dependency error). For any set Zt⊆MtZ_t\subseteq M_t decoded in parallel, TCπ(xZt∣xt+1)≤∑i∈ZtIti\mathrm{TC}_\pi(\mathbf{x}^{Z_t}\mid\mathbf{x}_{t+1})\le\sum_{i\in Z_t}\mathcal{I}_t^i. Proof. By the chain rule, H(xZt∣⋅)≥∑iH(xi∣xZt∖i,⋅)H(\mathbf{x}^{Z_t}\mid\cdot)\ge\sum_i H(\mathbf{x}^i\mid\mathbf{x}^{Z_t\setminus i},\cdot). Hence TC≤∑iI(xi;xZt∖i∣⋅)≤∑iIti\mathrm{TC}\le\sum_i I(\mathbf{x}^i;\mathbf{x}^{Z_t\setminus i}\mid\cdot)\le\sum_i\mathcal{I}_t^i. □\square Thus invariant tokens can be decoded in parallel with any other tokens without factorization error.
  • Non-invariant commitments induce later errors. Let π(010)=π(111)=14\pi(010)=\pi(111)=\tfrac14 and π(100)=π(101)=π(000)=π(001)=18\pi(100)=\pi(101)=\pi(000)=\pi(001)=\tfrac18. Here x2,x3x^2,x^3 are independent, while x1x^1 is non-invariant, since π(x1∣x2x3)\pi(x^1\mid x^2x^3) depends on x2x3x^2x^3. Assume exact marginals:
    • Committing x1x^1 first gives π(x2x3∣x1=1)\pi(x^2x^3\mid x^1{=}1) with mass (0.5,0.25,0.25)(0.5,0.25,0.25) on (11,00,01)(11,00,01), so x2,x3x^2,x^3 become dependent. Parallel decoding then outputs the invalid 110110, and KL=0.216\mathrm{KL}=0.216 nats.
    • Decoding x2,x3x^2,x^3 first gives KL=0\mathrm{KL}=0.
  • An exact additive term under lookahead (Proposition B, Part 3).

[1] Lavenant, H., & Zanella, G. (2025). Error bounds and optimal schedules for masked diffusions with factorized approximations. arXiv preprint arXiv:2510.25544.

AuthorsOpenReview ↗

Response to Reviewer VmVv

Response to W2: Lemma 1 and the derivation at L856

Where L856 comes from. L856 follows from the definition of the invariant part (L208–212) rather than from Eq. (19) alone.

  • The invariant part Vi\mathcal{V}_i of token ii is the set of tokens that xi\mathbf{x}^i interacts with.
  • With context c=xt+1c=\mathbf{x}_{t+1}, this set splits into Vi+=Vi∩Z>t\mathcal{V}_i^+=\mathcal{V}_i\cap Z_{>t} (already decoded, contained in cc) and Vi−=Vi−Z>t\mathcal{V}_i^-=\mathcal{V}_i-Z_{>t} (not yet decoded).
  • All other masked tokens, R:=Z<t∖Vi−R:=Z_{\lt t}\setminus\mathcal{V}_i^-, do not interact with xi\mathbf{x}^i or its invariant part.

Formally, this is the separation

(\mathbf{x}^i,\mathbf{x}^{\mathcal{V}_i^-})\ \perp\ \mathbf{x}^{R}\mid c, \tag{D}

which is exactly the structure shown in Table 4: decoding xi\mathbf{x}^i changes only the invariant tokens (p→qp\to q) and leaves the rest unchanged (g→gg\to g). From (D), the two statements at L856 follow by standard properties of conditional independence:

  1. Weak union gives π(xVi−∣xi,c,xR)=π(xVi−∣xi,c)\pi(\mathbf{x}^{\mathcal{V}_i^-}\mid\mathbf{x}^i,c,\mathbf{x}^R)=\pi(\mathbf{x}^{\mathcal{V}_i^-}\mid\mathbf{x}^i,c): the invariant tokens of xi\mathbf{x}^i are invariant to the rest.
  2. Decomposition gives π(xR∣xi,c)=π(xR∣c)\pi(\mathbf{x}^R\mid\mathbf{x}^i,c)=\pi(\mathbf{x}^R\mid c): the marginals of non-invariant tokens are unchanged.

We agree that Eq. (19) alone (xi⊥xR∣c,xVi−\mathbf{x}^i\perp\mathbf{x}^R\mid c,\mathbf{x}^{\mathcal{V}_i^-}) is weaker than (D). We will state (D) explicitly as the definition at L210 and refer to it at L856.

Eqs. (20)–(21). We agree these should hold in expectation rather than pointwise. In expectation, the log-ratio becomes a mutual information, and Lemma 1 holds as follows. Let ZsZ_s be a uniformly random subset of size rr of the n=x+yn=x+y future positions.

Lemma 1 (restated).

EZsEπ[log⁡π(xZs∣xi,c)π(xZs∣c)]=βr Iπ(xi;xVi−∣c),βr=∑k≤rak∑k≤nak∈[0,1].\mathbb{E}_{Z_s}\mathbb{E}_\pi\Big[\log\frac{\pi(\mathbf{x}^{Z_s}\mid\mathbf{x}^i,c)}{\pi(\mathbf{x}^{Z_s}\mid c)}\Big]=\beta_r\thinspace I_\pi(\mathbf{x}^i;\mathbf{x}^{\mathcal{V}_i^-}\mid c),\qquad\beta_r=\frac{\sum_{k\le r}a_k}{\sum_{k\le n}a_k}\in[0,1].

Here ak=EσI(xi;xσ(k)∣xσ(<k),c)≥0a_k=\mathbb{E}_\sigma I(\mathbf{x}^i;\mathbf{x}^{\sigma(k)}\mid\mathbf{x}^{\sigma(\lt k)},c)\ge0, with σ\sigma a random ordering of the future positions. In addition, βr\beta_r is non-decreasing in rr with βn=1\beta_n=1. When the invariant tokens contribute additively (the per-token KLs in Table 4), βr=r/(x+y)\beta_r=r/(x+y), as stated in the paper.

Proof.

  1. The left-hand side equals EZsI(xi;xZs∣c)\mathbb{E}_{Z_s}I(\mathbf{x}^i;\mathbf{x}^{Z_s}\mid c).
  2. Taking ZsZ_s as the first rr positions of σ\sigma, the chain rule gives ∑k≤rak\sum_{k\le r}a_k.
  3. By (D), positions in RR contribute zero, so the total over all nn positions equals I(xi;xVi−∣c)I(\mathbf{x}^i;\mathbf{x}^{\mathcal{V}_i^-}\mid c). □\square

With the AR proxy, the identity holds up to the proxy error γ\gamma. The conclusion of Lemma 1 is therefore preserved, and the corrected proof makes it slightly stronger, since βr\beta_r is monotone in rr. We will rewrite A.4 accordingly.

AuthorsOpenReview ↗

Response to Reviewer VmVv

Response to W3: Lemma 2 and the Uni-E simplification

Lemma 2. We agree that Eqs. (31)–(33) do not by themselves imply Eq. (34), and we replace that proof with an exact identity. By Eqs. (13)–(15), the corrected kernel is available in closed form: pϕ,θ(x0∣xt,xs)=pAR(x0∣xs)p_{\phi,\theta}(\mathbf{x}_0\mid\mathbf{x}_t,\mathbf{x}_s)=p_{AR}(\mathbf{x}_0\mid\mathbf{x}_s), with partition function F(xt,xs)F(\mathbf{x}_t,\mathbf{x}_s). Let MsM_s be the positions still masked in xs\mathbf{x}_s and Δ\Delta the positions revealed between tt and ss.

Proposition B (exact error identity).

E KL(π(⋅∣xs)∥∏i∈Mspθ(xi∣xt))=E TCπ(xMs∣xs)⏟dependency+∑i∈Ms[Iπ(xi;xΔ∣xt)⏟invariance+E KL(π(xi∣xt)∥pθ(xi∣xt))⏟capacity],\mathbb{E}\thinspace \mathrm{KL}\Big(\pi(\cdot\mid\mathbf{x}_s)\Big\Vert\prod_{i\in M_s}p_\theta(\mathbf{x}^i\mid\mathbf{x}_t)\Big)=\underbrace{\mathbb{E}\thinspace \mathrm{TC}_\pi(\mathbf{x}^{M_s}\mid\mathbf{x}_s)}_{\text{dependency}}+\sum_{i\in M_s}\Big[\underbrace{I_\pi(\mathbf{x}^i;\mathbf{x}^{\Delta}\mid\mathbf{x}_t)}_{\text{invariance}}+\underbrace{\mathbb{E}\thinspace \mathrm{KL}\big(\pi(\mathbf{x}^i\mid\mathbf{x}_t)\Vert p_\theta(\mathbf{x}^i\mid\mathbf{x}_t)\big)}_{\text{capacity}}\Big],

whereas Uni-EDLM incurs only εAR:=E KL(π(⋅∣xs)∥pAR(⋅∣xs))\varepsilon_{AR}:=\mathbb{E}\thinspace \mathrm{KL}\big(\pi(\cdot\mid\mathbf{x}_s)\Vert p_{AR}(\cdot\mid\mathbf{x}_s)\big).

Proof. The KL to a product distribution equals TC plus the sum of the marginal KLs. Splitting each marginal log-ratio through π(xi∣xt)\pi(\mathbf{x}^i\mid\mathbf{x}_t) produces the mutual-information term. □\square

All three factors (dependency, invariance, capacity) therefore appear as exact additive terms, and Uni-E replaces their sum with the single proxy error εAR\varepsilon_{AR}. Lemma 2 becomes: the KL is strictly reduced if and only if εAR\varepsilon_{AR} is smaller than the right-hand side. This holds in non-independent or non-invariant scenarios with an accurate proxy (εAR≤γ\varepsilon_{AR}\le\gamma), which makes explicit the assumption the reviewer asked for. It also replaces the asserted ordering in Eq. (30).

Uni-E simplification. We thank the reviewer for this observation, which is correct. Since pAR(x0∣xt)=pAR(x0)/pAR(xt)p_{AR}(\mathbf{x}_0\mid\mathbf{x}_t)=p_{AR}(\mathbf{x}_0)/p_{AR}(\mathbf{x}_t) is consistent, Euni=Eedlm+log⁡pθ(xs∣xt)E_{uni}=E_{edlm}+\log p_\theta(\mathbf{x}_s\mid\mathbf{x}_t). This identity is precisely why the partition function of Uni-E is tractable: the intractable AR marginals cancel. Its interpretation as an invariance correction is as follows.

  1. For a fixed lookahead xs\mathbf{x}_s, the extra term is constant. The Uni-E kernel is therefore the lookahead-conditioned posterior pAR(x0∣xs)p_{AR}(\mathbf{x}_0\mid\mathbf{x}_s), which by Prop. B removes the invariance term the DLM incurs when it predicts from xt\mathbf{x}_t.
  2. Across candidates, log⁡pθ(xs∣xt)\log p_\theta(\mathbf{x}_s\mid\mathbf{x}_t) scores the tokens committed at this step. It favors commitments supported by AR consistency over those driven by the DLM's own confidence.
  3. The term is not redundant empirically. Uni-E vs. Ind-E (= EDLM) gives 13.4 vs. 19.0 Gen PPL at 1024 steps (Table 2) and, on LLaDA, 66.5 vs. 62.7 on MMLU and 10.9 vs. 8.6 on LiveBench (Table 3).

We will add these corrected statements and proofs to the revision.

Reviewer_tcSuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes energy-based decoding strategies for improving sampling quality in diffusion language models. It combines a new invariance energy with an existing independence energy, and uses an autoregressive proxy model to select among candidate completions. The authors motivate the approach through a sampling error decomposition and argue that the energy correction reduces the KL divergence from the target distribution. Experiments with MDLM, LLaDA, and Dream report improvements in generation quality and downstream task performance.

Critical strengths
  1. The paper targets an important problem: improving the quality and efficiency of parallel decoding in diffusion language models.
  2. The proposed framework can combine pretrained diffusion and autoregressive models without additional training, making it potentially useful across existing model backbones.
  3. The paper evaluates the proposed methods on both smaller DLMs and larger models with real-world datasets, and reports empirical improvements.
Critical weaknesses
  1. The theoretical analysis in Section 4.1 that motivates the main results and methods is incorrect.
  • First, it is already established in prior work [1] that sampling error arises from imperfect model learning and factorization during parallel sampling. In addition, Equation (5) does not capture the factorization bias due to parallel sampling. It becomes zero when the learned distribution pθp_{\theta} equals the target π\pi, even though independently sampling from the true marginals can still introduce factorization bias.
  • More fundamentally, the stated “invariance” condition as an additional source of sampling error is incorrect. Consider a target distribution assigning equal probability to (0,0) and (1,1). Sampling the first bit uniformly and then setting the second bit equal to the first recovers the target exactly. However, the first bit’s conditional distribution changes from uniform to deterministic when the second bit is revealed, which violates the stated invariance condition. Thus, non-invariance does not itself imply sampling error.
  1. As Sections 4.2–4.3 use the “invariance” factor to motivate Inv-E and Uni-E, the above analysis issue makes it doubtful whether the proposed method can reduce sampling error. A valid sampling error decomposition and a rigorous connection between energy reweighting and output distribution accuracy are needed to support the theoretical claims.
  2. The empirical advantage over competitive decoding strategies is not fully established. Section 5.1 does not clearly compare against the same MDLM backbone using SOTA confidence-based decoding. Section 5.2 includes stronger baselines, but improvements on several benchmarks are modest. Also, Table 3 reports no uncertainty estimates, making the reliability of these small gains difficult to assess.

[1] Lavenant, H., & Zanella, G. (2025). Error bounds and optimal schedules for masked diffusions with factorized approximations. arXiv preprint arXiv:2510.25544.

Questions for authors
  1. See Weakness.
  2. In "Inference of Uni-EDLM", the authors state that the self-normalized importance sampling strategy is used. However, Algorithms 2–3 select the minimum energy candidate rather than resampling according to normalized importance weights. Can the authors clarify this inconsistency?
4 comments
AuthorsOpenReview ↗

Response to Reviewer tcSu

Response to Q1/W1

W1(a): Relation to [1], and Eq. (5) captures the factorization bias.

We agree that the decomposition into learning error and factorization error was established in [1]. Our analysis is built on that framework, as stated in Sec. 3 (L126: "Based on the framework proposed in Lavenant & Zanella (2025)"). Our capacity and dependency factors correspond to the learning error and factorization error of [1], respectively. Regarding the concern that Eq. (5) "becomes zero when the learned distribution pθp_\theta equals the target π\pi", we would like to respectfully clarify a subtle point. By design, a DLM decodes the tokens in ZtZ_t under the independence assumption (Sec. 3): pθ(xtZt∣xt+1)=∏i∈Ztpθ(xti∣xt+1).p_\theta(\mathbf{x}_t^{Z_t}\mid\mathbf{x}_{t+1})=\prod_{i\in Z_t}p_\theta(\mathbf{x}_t^{i}\mid\mathbf{x}_{t+1}).

Therefore, the premise "pθp_\theta equals π\pi" implicitly requires the same factorization for the target: π(xtZt∣xt+1)=∏i∈Ztπ(xti∣xt+1),\pi(\mathbf{x}_t^{Z_t}\mid\mathbf{x}_{t+1})=\prod_{i\in Z_t}\pi(\mathbf{x}_t^{i}\mid\mathbf{x}_{t+1}),

i.e., the co-decoded tokens are truly independent. In that case there is no factorization bias, and Ldp=0L_{dp}=0 is correct.

In practice, this independence usually does not hold for parallel decoding. Then pθp_\theta cannot equal π\pi, and Eq. (5) is not zero. Even if every learned marginal were exact, pθ(xti∣xt+1)=π(xti∣xt+1)p_\theta(\mathbf{x}_t^{i}\mid\mathbf{x}_{t+1})=\pi(\mathbf{x}_t^{i}\mid\mathbf{x}_{t+1}) for all ii, the second log-term of Eq. (5) is identically zero, and the first term remains: Eπ[Ldp]=Eπ[log⁡π(xtZt∣xt+1)∏i∈Ztπ(xti∣xt+1)]=TCπ(xZt∣xt+1)>0.\mathbb{E}_\pi[L_{dp}]=\mathbb{E}_\pi\Big[\log\frac{\pi(\mathbf{x}_t^{Z_t}\mid\mathbf{x}_{t+1})}{\prod_{i\in Z_t}\pi(\mathbf{x}_t^{i}\mid\mathbf{x}_{t+1})}\Big]=\mathrm{TC}_\pi(\mathbf{x}^{Z_t}\mid\mathbf{x}_{t+1})>0.

This is the total correlation of the co-decoded tokens, i.e., exactly the factorization bias of independently sampling from the true marginals described by the reviewer. Thus Eq. (5) vanishes if and only if the co-decoded tokens are independent.

W1(b): Invariance factor

We thank the reviewer for this thoughtful example. We agree with the computation, but respectfully note that it does not contradict the role of invariance in parallel decoding, the setting our analysis targets. We clarify this in three points.

(1) The example lies in the sequential, exact regime. One token is decoded per step from its exact conditional. In this regime the chain rule guarantees zero error under any order, so no factor (dependency or invariance) can cause error. DLMs instead decode many tokens per step, using a factorized denoiser that predicts all masked tokens from the same context xt\mathbf{x}_t. In this regime, invariance governs the error as follows.

(2) Invariance controls the error of each parallel step. Let MtM_t be the masked positions at step tt, and let Iti:=Iπ(xi;xMt∖i∣xt+1)=Eπ[log⁡π(xi∣x0−xi)π(xi∣xt+1)]\mathcal{I}_t^i:=I_\pi(\mathbf{x}^i;\mathbf{x}^{M_t\setminus i}\mid\mathbf{x}_{t+1})=\mathbb{E}_\pi\Big[\log\frac{\pi(\mathbf{x}^i\mid\mathbf{x}_0-\mathbf{x}^i)}{\pi(\mathbf{x}^i\mid\mathbf{x}_{t+1})}\Big], the token-level invariance term of Eq. (6).

Proposition A. For any set Zt⊆MtZ_t\subseteq M_t decoded in parallel, TCπ(xZt∣xt+1)≤∑i∈ZtIti\mathrm{TC}_\pi(\mathbf{x}^{Z_t}\mid\mathbf{x}_{t+1})\le\sum_{i\in Z_t}\mathcal{I}_t^i.

Proof. By the chain rule, H(xZt∣⋅)≥∑iH(xi∣xZt∖i,⋅)H(\mathbf{x}^{Z_t}\mid\cdot)\ge\sum_i H(\mathbf{x}^i\mid\mathbf{x}^{Z_t\setminus i},\cdot). Hence TC≤∑iI(xi;xZt∖i∣⋅)≤∑iIti\mathrm{TC}\le\sum_i I(\mathbf{x}^i;\mathbf{x}^{Z_t\setminus i}\mid\cdot)\le\sum_i\mathcal{I}_t^i. □\square

Invariant tokens can therefore be co-decoded with any other tokens without factorization error. In the reviewer's example, decoding both bits in parallel incurs TC=log⁡2\mathrm{TC}=\log 2, which the bound captures. Unlike TC, which requires the intractable joint over the adaptively chosen ZtZ_t, Iti\mathcal{I}_t^i is per-token. It equals I(xi;xVi−∣xt+1)I(\mathbf{x}^i;\mathbf{x}^{\mathcal{V}_i^-}\mid\mathbf{x}_{t+1}), which is what Inv-E estimates.

(3) Non-invariant decoding induces error at later steps. Let π(010)=π(111)=14\pi(010)=\pi(111)=\tfrac14 and π(100)=π(101)=π(000)=π(001)=18\pi(100)=\pi(101)=\pi(000)=\pi(001)=\tfrac18. Here x2x^2 and x3x^3 are independent, while x1x^1 is non-invariant. Even with exact marginals:

  • Committing to x1x^1 first makes x2,x3x^2,x^3 dependent given x1=1x^1{=}1. Parallel decoding then outputs the invalid 110110, with KL=0.216\mathrm{KL}=0.216 nats.
  • Decoding x2,x3x^2,x^3 first gives KL=0\mathrm{KL}=0.

The model and the marginals are identical in both cases; only the invariance of the committed token differs.

[1] Lavenant, H., & Zanella, G. (2025). Error bounds and optimal schedules for masked diffusions with factorized approximations. arXiv preprint arXiv:2510.25544.

AuthorsOpenReview ↗

Response to Reviewer tcSu

Response to Q1/W2

We thank the reviewer for this important suggestion. With the chain-rule form of Eq. (3), we validate the sampling-error decomposition as follows:

Closed form of the corrected distributions. Substituting Eq. (11) and Eq. (13) into Eq. (15), and using Eq. (14), gives:

Ind-E: pϕ,θ(x0∣xt)=pAR(x0∣xt),Uni-E: pϕ,θ(x0∣xt,xs)=pAR(x0∣xs).\text{Ind-E: } p_{\phi,\theta}(\mathbf{x}_0\mid\mathbf{x}_t)=p_{AR}(\mathbf{x}_0\mid\mathbf{x}_t),\qquad\text{Uni-E: } p_{\phi,\theta}(\mathbf{x}_0\mid\mathbf{x}_t,\mathbf{x}_s)=p_{AR}(\mathbf{x}_0\mid\mathbf{x}_s).

Without the energy, the DLM predicts the tokens still masked in xs\mathbf{x}_s (the set MsM_s) from xt\mathbf{x}_t only, as ∏i∈Mspθ(xi∣xt)\prod_{i\in M_s}p_\theta(\mathbf{x}^i\mid\mathbf{x}_t). Let Δ\Delta denote the positions revealed between tt and ss.

Proposition B (exact identity). In expectation over x0∼π\mathbf{x}_0\sim\pi and the forward process,

E KL(π(⋅∣xs)∥∏i∈Mspθ(xi∣xt))=E TCπ(xMs∣xs)⏟dependency+∑i∈Ms[Iπ(xi;xΔ∣xt)⏟invariance+E KL(π(xi∣xt)∥pθ(xi∣xt))⏟capacity],\mathbb{E}\thinspace \mathrm{KL}\Big(\pi(\cdot\mid\mathbf{x}_s)\Big\Vert\prod_{i\in M_s}p_\theta(\mathbf{x}^i\mid\mathbf{x}_t)\Big)=\underbrace{\mathbb{E}\thinspace \mathrm{TC}_\pi(\mathbf{x}^{M_s}\mid\mathbf{x}_s)}_{\text{dependency}}+\sum_{i\in M_s}\Big[\underbrace{I_\pi(\mathbf{x}^i;\mathbf{x}^{\Delta}\mid\mathbf{x}_t)}_{\text{invariance}}+\underbrace{\mathbb{E}\thinspace \mathrm{KL}\big(\pi(\mathbf{x}^i\mid\mathbf{x}_t)\Vert p_\theta(\mathbf{x}^i\mid\mathbf{x}_t)\big)}_{\text{capacity}}\Big],

whereas for Uni-EDLM, E KL(π(⋅∣xs)∥pϕ,θ)=E KL(π(⋅∣xs)∥pAR(⋅∣xs))=:εAR\mathbb{E}\thinspace \mathrm{KL}\big(\pi(\cdot\mid\mathbf{x}_s)\Vert p_{\phi,\theta}\big)=\mathbb{E}\thinspace \mathrm{KL}\big(\pi(\cdot\mid\mathbf{x}_s)\Vert p_{AR}(\cdot\mid\mathbf{x}_s)\big)=:\varepsilon_{AR}.

Proof sketch. For fixed xs\mathbf{x}_s, the KL to a product distribution equals TC plus the sum of the marginal KLs. Each marginal log-ratio splits as log⁡π(xi∣xs)π(xi∣xt)+log⁡π(xi∣xt)pθ(xi∣xt)\log\frac{\pi(\mathbf{x}^i\mid\mathbf{x}_s)}{\pi(\mathbf{x}^i\mid\mathbf{x}_t)}+\log\frac{\pi(\mathbf{x}^i\mid\mathbf{x}_t)}{p_\theta(\mathbf{x}^i\mid\mathbf{x}_t)}. Since masking is independent of content, the first part has expectation I(xi;xΔ∣xt)I(\mathbf{x}^i;\mathbf{x}^{\Delta}\mid\mathbf{x}_t). □\square.

Consequences.

  • Each energy targets a specific factor. All three factors appear as exact, additive terms, and the energy correction replaces their sum by a single proxy error εAR\varepsilon_{AR}. Ind-E (conditioning on xt\mathbf{x}_t) addresses the dependency and capacity terms. Uni-E (conditioning on the lookahead xs\mathbf{x}_s) additionally removes the invariance term. Inv-E's term log⁡pAR(x0∣xt)pAR(x0∣xs)\log\frac{p_{AR}(\mathbf{x}_0\mid\mathbf{x}_t)}{p_{AR}(\mathbf{x}_0\mid\mathbf{x}_s)} is the sample-level counterpart of how much the newly decoded tokens xΔ\mathbf{x}^{\Delta} change the prediction of the remaining tokens, which is exactly what drives the invariance term. This matches our ablations (Fig. 2, Table 3): Uni-E > Ind-E / Inv-E alone.
  • Refined Lemma 2. Uni-EDLM has strictly smaller error than the DLM if and only if εAR\varepsilon_{AR} is smaller than the right-hand side above. In particular, with an accurate proxy (εAR≤γ\varepsilon_{AR}\le\gamma, Lemma 1), the improvement is strict in every non-independent (TC>γ\mathrm{TC}>\gamma) or non-invariant (∑iI>γ\sum_i I>\gamma) scenario.
  • The condition holds in practice. An AR proxy has no factorization error by construction. In Table 1, the proxy AR reaches PPL 17.56 versus 23.83 for MDLM on OWT.

We will revise the proof in A.5 with Prop. B and refine Lemma 2.

AuthorsOpenReview ↗

Response to Reviewer tcSu

Response to Q2/W3

We thank the reviewer for these suggestions. Due to the time limit, we report a representative subset; complete results will be included in the revision.

(1) Confidence-based decoding on the same MDLM backbone. Same checkpoint and protocol as Table 2 (Llama-2 Gen PPL; 128/256/512/1024 steps).

MethodGen PPL ↓Entropy
MDLM48.5 / 37.2 / 30.6 / 27.68.1 / 7.9 / 7.8 / 7.6
MDLM (Top-k confidence)44.3 / 36.9 / 24.5 / 14.27.9 / 7.4 / 6.8 / 5.9
MDLM (Entropy)46.1 / 35.5 / 25.4 / 16.77.8 / 7.4 / 7.1 / 6.5
MDLM (Threshold)46.6 / 36.9 / 25.1 / 15.47.9 / 7.7 / 7.0 / 5.8
Uni-EDLM-AR40.4 / 32.4 / 23.2 / 13.48.1 / 7.9 / 7.5 / 7.2
Uni-EDLM-NCE40.6 / 32.2 / 22.8 / 13.18.1 / 7.9 / 7.5 / 7.2
  • Uni-EDLM achieves the lowest Gen PPL at every step count.
  • Its entropy stays close to the backbone (7.2 at 1024 steps), whereas confidence-based strategies lower Gen PPL partly by sacrificing diversity (entropy drops to 5.8–6.5).

Confidence-based methods only change the planner μϕ\mu_\phi (which tokens to decode) and do not correct the dependency or invariance errors of the decoded tokens. Uni-E corrects these errors directly, improving quality without collapsing diversity.

(2) Uncertainty estimates for Table 3. Mean / std over multiple random seeds.

ModelMATH500GSM8KMBPPLCB V2LiveBenchMMLUHellaSwag
Qwen2.5-0.5B33.6/1.241.7/0.538.8/0.710.6/0.611.5/0.746.9/0.953.6/1.3
LLaDA-8B35.4/0.877.6/0.633.2/1.06.3/0.83.9/0.657.5/0.468.7/0.9
+ APD36.5/1.178.6/1.333.1/0.57.6/1.25.7/0.962.5/1.470.7/1.2
+ CORE36.1/0.878.6/0.534.2/0.75.6/1.74.1/1.461.5/1.069.1/0.9
+ Uni-E39.6/0.981.6/1.434.9/0.611.3/1.19.3/1.065.9/0.772.5/1.6
  • The gains over LLaDA-8B (+4.2, +4.0, +1.7, +5.0, +5.4, +8.4, +3.8) exceed the standard deviations on all benchmarks.
  • The gains over the strongest baseline exceed the std on 6/7 benchmarks; on MBPP, Uni-E is comparable to CORE.
  • LLaDA + Uni-E outperforms its own AR proxy on 5/7 benchmarks, so Uni-E does not simply copy the proxy but combines the strengths of both models.

Q2: SNIS vs. arg-min

We thank the reviewer for pointing this out. With SNIS weights wj∝exp⁡(−E(j))w_j\propto\exp(-E^{(j)}) over kk candidates from pθp_\theta, Algorithms 2–3 select the largest-weight candidate, i.e., the greedy (zero-temperature) variant of SNIS (Min-E), instead of resampling (IS). We choose Min-E deliberately, for three reasons:

  1. Better quality. Min-E selects the mode of the corrected distribution pϕ,θp_{\phi,\theta}.
  2. More stable. With few candidates, normalized weights are noisy and resampling adds variance; Min-E depends only on the energy ranking.
  3. Suited to task-oriented decoding. Math/code benchmarks reward the most plausible answer, as in greedy decoding of LLMs.

The experiments below support these reasons.

MDLM (Gen PPL under Llama-2 / Entropy; 128/256/512/1024 steps)

MethodGen PPL ↓Entropy
EDLM43.2 / 34.7 / 26.8 / 19.08.0 / 7.9 / 7.6 / 7.2
Uni-EDLM-AR (Min-E)40.4 / 32.4 / 23.2 / 13.48.1 / 7.9 / 7.5 / 7.2
Uni-EDLM-AR (IS)41.1 / 33.2 / 23.6 / 13.88.2 / 8.0 / 7.6 / 7.5
Uni-EDLM-NCE (Min-E)40.6 / 32.2 / 22.8 / 13.18.1 / 7.9 / 7.5 / 7.2
Uni-EDLM-NCE (IS)41.6 / 32.8 / 23.1 / 13.68.2 / 7.9 / 7.6 / 7.4

LLaDA-8B (mean / std; kk = number of candidates)

VariantMATH500GSM8KMBPPLCB V2LiveBenchMMLUHellaSwag
Min-E, k=3k{=}339.6/0.981.6/1.434.9/0.611.3/1.19.3/1.065.9/0.772.5/1.6
IS, k=3k{=}339.4/1.182.1/1.535.2/0.410.2/1.310.1/0.866.4/0.971.2/1.4
Min-E, k=6k{=}640.0/0.482.1/0.735.4/0.910.6/1.310.1/0.766.6/1.173.2/1.0
IS, k=6k{=}640.2/1.282.6/1.234.3/1.212.3/1.210.9/1.365.6/0.972.8/0.9

Evidence.

  • Quality (reason 1). Min-E has lower Gen PPL than IS at every step count for both variants, with minor entropy cost (7.2 vs. 7.5 at 1024 steps), still far above confidence-based decoding (5.8–6.5, Part 4).
  • Stability (reason 2). Min-E's average std is comparable at k=3k{=}3 (1.04 vs. 1.06) and lower at k=6k{=}6 (0.87 vs. 1.13).
  • Task accuracy (reason 3). On LLaDA, Min-E matches IS within one std on most benchmarks.
  • Both variants are effective. Both clearly outperform EDLM and the backbones, so the gains come from the energy design. Larger kk brings further gains (consistent with Table 7); we use k=3k{=}3 for efficiency.

We will revise the statement to "greedy variant of SNIS", state this rationale, and report both variants.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer tcSu,

We hope this message finds you well. As the discussion period is nearing its end, we would like to kindly check whether our responses have addressed your concerns. Your feedback is very important to us, and we are happy to clarify anything further.

Thank you again for your insightful comments, which have greatly improved our paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.