- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Unified Energy (Uni-E) to address the degradation in generation quality caused by insufficient modeling of token relationships during parallel decoding in diffusion language models. Through a KL divergence analysis, the authors identify three factors contributing to distribution shifts: model capacity, token dependency, and invariance across decoding steps. Based on this analysis, they introduce an Invariant Energy (Inv-E) that uses an AR proxy model to assess the influence of future context, and combine it with the existing Independent Energy (Ind-E) to improve parallel decoding through energy-based candidate reweighting. The authors also derive an analytical expression for the partition function. Experiments on both conventional and large-scale diffusion language models demonstrate improvements in generation quality, reduced performance degradation at higher levels of parallelism, and competitive inference efficiency.
Critical strengths
I really appreciate the paper's clear and insightful analysis of parallel decoding limitation with corresponding solutions of two factors. this has inspired me a lot, And I find the following two aspects particularly compelling.
- The paper decomposes the distribution shift in DLM parallel decoding into three factors through a KL divergence analysis: model capacity, token dependency, and invariance across decoding steps. This provides a valuable perspective for understanding generation quality degradation during parallel decoding. Building on this analysis, the authors introduce Inv-E and combine it with the existing Ind-E to construct a unified energy-based correction framework. The method is well motivated, and the ablation studies demonstrate the complementary benefits of the two energy components in improving generation quality.
- The paper provides comprehensive experimental evaluations, demonstrating improvements in language modeling and generation quality on conventional DLMs and extending Uni-E to large-scale models such as LLaDA-8B and Dream-7B. The evaluation covers diverse tasks, including mathematical reasoning, code generation, and knowledge-based benchmarks. Furthermore, component ablations, hyperparameter analyses, and accuracy–throughput comparisons provide substantial empirical evidence of the method's effectiveness across different model scales and decoding configurations.
Critical weaknesses
There are still weakness in this paper, and I hope this can be addressed by the authors.
- Although the paper discusses and compares against existing methods such as EDLM, it lacks sufficient discussion of closely related work, particularly Diffusion Language Model Parallel Decoding via Product-of-Experts Bridge (PoE-Bridge). These methods similarly leverage probabilistic information from AR models to correct the parallel generation distribution of DLMs. The authors should more clearly distinguish Uni-E from existing distribution correction approaches, particularly regarding the unique contributions of the proposed invariance modeling. This omission makes the novelty and positioning of the paper less convincing.
- While the proposed KL divergence decomposition provides an insightful perspective, several key assumptions in the subsequent theoretical analysis are not sufficiently justified. Specifically, the sampling estimation in Lemma 1 relies on the additivity of joint probability contributions, which does not necessarily hold when complex token dependencies exist. Furthermore, Lemma 2 claims that Uni-E strictly reduces the KL divergence from the ground-truth distribution, but the assumed correct energy ranking is insufficient to guarantee such a reduction. The paper also does not clearly establish how these theoretical guarantees extend to the finite-candidate, energy-based selection strategy used in practice. Consequently, the current proofs do not fully support the general theoretical guarantees claimed by the authors.
Questions for authors
After reading this paper, I have following questions:
- Could the authors clarify the key differences between Uni-E and closely related AR-guided distribution correction methods, particularly PoE-Bridge? What specific advantages does the proposed invariance modeling offer beyond existing approaches?
- Could the authors clarify the assumptions underlying Lemmas 1 and 2? Specifically, how is the additivity assumption in the invariance sampling estimation justified when token dependencies exist? Furthermore, why does correct energy ranking necessarily guarantee a strict reduction in KL divergence?
- Could the authors provide more direct experimental evidence that Inv-E specifically mitigates invariance violations across decoding steps? For instance, analyzing inconsistencies between early-decoded tokens and subsequently revealed context would help substantiate the proposed mechanism beyond the overall improvements reported in the ablation studies.
- The theoretical analysis considers an energy-reweighted probability distribution, while the actual inference algorithm selects the lowest-energy candidate from a finite set. Could the authors clarify whether the theoretical guarantees remain valid under this practical selection strategy and, if so, under what conditions? Helpful explanations will help me to update my rate.
5 comments
Response to Reviewer 77yy
Response to Q2/W2: Assumptions of Lemmas 1 and 2
We thank the reviewer for these precise questions. We give rigorous versions of both lemmas, which make the assumptions explicit. These revision will be further updated in the paper.
Lemma 1: additivity is not required. Let , let be the not-yet-decoded invariant part of token , and let be a uniformly random subset of size of the future positions. Stated in expectation,
where and is a random ordering of the future positions.
Proof.
- The left-hand side is .
- Taking as the first elements of , the chain rule of mutual information gives .
- Tokens outside contribute zero by the definition of the invariant part.
This holds for arbitrary token dependencies. Dependencies only change the value of ; is always non-decreasing in , with . Additivity is needed only for the specific value . Hence the estimator remains a valid, monotone proxy of the invariance term in general.
Lemma 2: an exact identity replaces the ranking argument. We agree that a correct energy ranking alone does not control probability magnitudes. We therefore replace the ranking argument with an exact identity. The Uni-E corrected kernel has the closed form . Let be the positions still masked in and those revealed between and . Then
while Uni-EDLM incurs only . Here TC denotes total correlation.
Proof. The KL to a product distribution equals TC plus the sum of the marginal KLs. Splitting each marginal log-ratio through gives the mutual-information term.
Revised Lemma 2. The KL is strictly reduced if and only if is smaller than the right-hand side. In particular, the reduction is strict in non-independent or non-invariant scenarios whenever the proxy is accurate. The required assumption, an accurate proxy, is now explicit.
Response to Reviewer 77yy
Response to Q3/Q4: Direct evidence of invariance, and finite-candidate selection
Q3: Direct evidence. Following the reviewer's suggestion, we directly measured invariance violations across decoding steps. After generation, we checked each early-decoded token against the fully revealed context. The flip rate is the fraction of these tokens that an oracle AR model would change given the complete sequence. A lower flip rate means that committed tokens remain consistent with the context revealed later, i.e., fewer invariance violations.
| Method | Flip rate ↓ |
|---|---|
| MDLM | 0.40 |
| MDLM + Ind-E | 0.32 |
| MDLM + Inv-E | 0.28 |
| MDLM + Uni-E | 0.24 |
- Inv-E alone reduces the flip rate more than Ind-E (0.28 vs. 0.32), although Ind-E corrects the joint distribution within each step. This directly confirms that Inv-E targets cross-step invariance rather than only within-step dependency.
- The two energies are complementary: Uni-E reduces the flip rate by 40% relative to MDLM (0.40 → 0.24). This mirrors their complementary effect on Gen PPL in our ablation (Fig. 2).
We will add this analysis to the revision, together with the qualitative cases in Appendix A.12 (Figs. 5 and 7).
Q4: Finite-candidate selection. The guarantee in Lemma 2 concerns the energy-reweighted distribution , and the practical algorithm approximates it as follows.
- Resampling variant (SNIS). Drawing one of candidates with weights is sampling-importance-resampling. Its output distribution converges to as , so the guarantee holds in the limit and is approached as grows.
- Arg-min variant (our default). This is the zero-temperature limit, which targets the mode of . It satisfies a simple exact property: the expected energy of the selected candidate, , is non-increasing in .
- Empirical support. We added a comparison of the two variants. Both clearly outperform EDLM and the backbone, and they differ mostly within one standard deviation on LLaDA. For example, on MATH500 at , arg-min gives 39.6 ± 0.9 and resampling gives 39.4 ± 1.1. Increasing from 3 to 6 further improves performance (40.0 and 40.2, respectively), consistent with the approximation tightening as grows.
Response to Reviewer 77yy
Response to Q1/W1: Relation to PoE-Bridge and other AR-guided corrections
We thank the reviewer for pointing to PoE-Bridge (Shi et al., ICML 2026), a recent work that we will discuss in the revision. Both methods use an AR model and multiple DLM drafts, but they differ fundamentally in goal, in the error they model, and in mechanism.
1. Target distribution. PoE-Bridge drafts with the DLM and corrects the drafts toward the AR target. Its quality is therefore bounded by the AR model, and it reports reaching at least 95% of the AR model's performance. In Uni-E, the DLM remains the target model, and the AR model is only a lightweight proxy (Qwen2.5-0.5B for LLaDA-8B). As a result, Uni-E can exceed its own proxy: over multiple seeds, LLaDA-8B + Uni-E reaches 39.6 vs. 33.6 on MATH500, 81.6 vs. 41.7 on GSM8K, and 65.9 vs. 46.9 on MMLU (Uni-E vs. Qwen2.5-0.5B). This shows that Uni-E combines the strengths of both models rather than reproducing the AR model.
2. Invariance modeling (our unique contribution). Existing AR-guided corrections, including EDLM and PoE-Bridge, correct the joint distribution of the tokens decoded in the current step, i.e., the dependency factor. To our knowledge, they do not model whether a committed token will remain valid once the future context is revealed. Inv-E addresses exactly this question. It scores each candidate together with its lookahead xs and penalizes commitments whose plausibility changes once more context is revealed. Part 3 gives direct evidence that this reduces invariance violations.
3. Mechanism and cost. PoE-Bridge relies on rejection sampling with token-level verification. Uni-E needs one batched energy evaluation over k candidates per step. Its partition function is available in closed form, so no MCMC or sampling-based estimation is required.