acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Continuous Diffusion Language Models

Abstract

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mmoUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hierarchical Continuous Diffusion Language Models (HC-DLM), which combine a continuous latent trajectory with a discrete token scaffold. Tokens are repeatedly read out, re-noised, and fed back to guide continuous denoising. The paper presents a hierarchical generative model and a variational lower bound, with experiments on Sudoku, Countdown, and LM1B.

Critical strengths

Using a discrete token scaffold to guide continuous denoising is a promising design. The structured-task results are convincing: at a similar sampling-time parameter count, HC-DLM achieves 37.52% accuracy on CD5, compared with 25.35% for the reproduced CCDD baseline. The Sudoku ablation also shows a substantial drop when the scaffold is removed. The analyses of noise kernels, sampling steps, and runtime are useful.

Critical weaknesses
  1. The connection between the hierarchical model and the sampler is unclear. Eq. (3) conditions kt−1k_{t-1} only on xt−1x_{t-1}, while Algorithm 2 generates it from x^0\hat{x}_0 computed using (xt,kt)(x_t,k_t). It is not clear why this update preserves the continuous latent as the only persistent state, which is central to the paper's distinction from other hybrid models.

  2. Evidence for the token scaffold's contribution to language generation is less direct. The component ablation in Table 4 covers only Sudoku. The LM1B comparison includes LangFlow with self-conditioning, but the models and training setups differ, making the benefit over continuous self-conditioning difficult to isolate.

Questions for authors
  1. In Algorithm 2, the next token state is generated from x^0\hat{x}_0, which depends on (xt,kt)(x_t,k_t). How does this correspond to pθ(kt−1∣xt−1)p_\theta(k_{t-1}\mid x_{t-1}) in Eq. (3)? Is this shorthand notation, or an approximation used in sampling?

  2. Could the authors compare no feedback, continuous self-conditioning, and full HC-DLM on LM1B using the same backbone, tokenizer, training budget, and number of sampling steps? Using w=1w=1 for HC-DLM and reporting Gen. PPL, entropy, and sampling time would help isolate the token scaffold's contribution.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mmoU (1/2)

We thank the reviewer for their time and feedback. We appreciate the reviewer noting that "using a discrete token scaffold to guide continuous denoising is a promising design", that "the structured-task results are convincing", and that "the analyses of noise kernels, sampling steps, and runtime are useful". We answer questions next.

Recap. HC-DLM denoises a continuous latent xtx_t. At every step it decodes all tokens, re-noises them, and feeds the result ktk_t back as a condition, called the token scaffold. Classifier-free guidance (CFG) controls the strength of this condition, with w=1w=1 meaning no CFG. Generative perplexity (Gen. PPL) measures sample quality under GPT-2 Large (lower is better). We report unigram entropy alongside it because reducing sample diversity can also lower Gen. PPL.

W1 / Q1: Eq. (3) vs. Algorithm 2

This is a conditioning-notation omission: Eq. (3) should use pθ(kt−1∣xt−1,xt)p_\theta(k_{t-1}\mid x_{t-1},x_t). The precise readout depends on two adjacent latents, while still requiring no separate token transition chain. Algorithm 2 computes x^0=dϕ(xt,kt,t/T)\hat{x}_0=d_\phi(x_t,k_t,t/T), takes an Euler step, and decodes and re-noises tokens from x^0\hat{x}_0. For its deterministic Euler update,

xt−1=xt−1t(xt−x^0)⟹x^0=xt−t(xt−xt−1).x_{t-1}=x_t-\frac{1}{t}(x_t-\hat{x}_0) \quad\Longrightarrow\quad \hat{x}_0=x_t-t(x_t-x_{t-1}).

The omitted conditioning argument is therefore xtx_t, and the next scaffold can be written as a readout of the adjacent continuous states:

pθ(kt−1∣xt−1,xt)=∑k0q(kt−1∣k0),pθ(k0∣xt−t,(xt−xt−1)).p_\theta(k_{t-1}\mid x_{t-1},x_t)=\sum_{k_0} q(k_{t-1}\mid k_0)\\,p_\theta\bigl(k_0\mid x_t-t\\,(x_t-x_{t-1})\bigr).

The current scaffold influences x^0\hat{x}_0 and the latent update. Once the adjacent latents are fixed, however, the next scaffold has no additional dependence on the earlier token states. Every token is decoded anew, so there is no independent chain of token commitments. The precise claim is that the continuous trajectory carries cross-step information, not that xt−1x_{t-1} alone is a sufficient state for this readout.

This correspondence is specific to the deterministic Euler sampler of Algorithm 2. The consistent model notation uses pθ(kt∣xt,xt+1)p_\theta(k_t\mid x_t,x_{t+1}) for intermediate readouts and retains the terminal initialization kT∼πk_T\sim\pi. Writing pθ,ϕadjp^{\mathrm{adj}}_{\theta,\phi} for that normalized generative model, Jensen's inequality gives

log⁡pθ,ϕadj(k0)≥Eqψ![log⁡pθ,ϕadj(k0:T,x0:T)qψ(x0:T,k1:T∣k0)].\log p^{\mathrm{adj}}_{\theta,\phi}(k_0) \geq \mathbb{E}_{q_\psi}\\!\left[ \log\frac{p^{\mathrm{adj}}_{\theta,\phi}(k_{0:T},x_{0:T})} {q_\psi(x_{0:T},k_{1:T}\mid k_0)} \right].

The token-prediction terms and their expectations retain the adjacent-latent argument.

We have made this correction throughout the revised paper: Eq. (3) and the description of the token predictor in Section 3.1, the bound in Proposition 1 and its proof in Appendix A, the token predictor in Appendix B.1, and the conditional model in Appendix B.2. Section 3.4 (Eq. (7)) now states the correspondence with Algorithm 2.

AuthorsOpenReview ↗

Response to Reviewer mmoU (2/2)

W2 / Q2: Feedback ablation on LM1B

We ran the requested comparison on LM1B: same backbone, tokenizer, frozen encoder and token predictor, 1M training steps, 128 sampling steps and w=1w=1. At the same entropy, token feedback reduces Gen. PPL from 163.8 (continuous self-conditioning) and 251.7 (no feedback) to 87.9.

The three models differ in what the denoiser receives at each step: nothing, its previous clean-latent estimate (continuous self-conditioning), or the decoded and re-noised tokens (HC-DLM). LangFlow and CCDD are included at the same sampling steps and entropy.

LM1B, w=1w=1Gen. PPL ↓ / entropySampling time (s)
No feedback251.7 / 4.341.11 ± 0.01
Continuous self-conditioning163.8 / 4.311.14 ± 0.0003
LangFlow92.3 / 4.313.48 ± 0.002
CCDD-MDiT314.6 / 4.3119.4 ± 0.0004
HC-DLM (token scaffold)87.9 / 4.312.84 ± 0.0004

Sampling time is the median of 5 runs for a batch of 64 on one RTX PRO 6000 GPU with compilation.

Quality. At matched entropy, token feedback lowers Gen. PPL from 163.8 to 87.9 (46%) relative to continuous self-conditioning, and from 251.7 (65%) relative to no feedback. HC-DLM also outperforms LangFlow and CCDD at the same sampling steps.

Sampling time. HC-DLM samples faster than LangFlow and nearly 7× faster than CCDD (19.4 s), while achieving better Gen. PPL than both. Relative to the two latent baselines (1.11 s and 1.14 s), the token scaffold brings a much larger quality gain than its sampling cost: Gen. PPL drops from 163.8 and 251.7 to 87.9.

The feedback comparison and the sampling times are now in Appendix E.6 of the revised paper, and the comparison with LangFlow and CCDD at matched sampling steps is in Appendix E.3.

Dear Reviewer mmoU,

We thank the reviewer for the helpful suggestions. In our response, we corrected the notation in Eq. (3) and ran the requested LM1B comparison of no feedback, continuous self-conditioning and the token scaffold (now Appendix E.6). We would be grateful to know whether these address the reviewer's concerns, and we look forward to the reviewer's further input during the discussion period.

Reviewer_jMu9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a hierarchical diffusion model that maintains a continuous latent trajectory and repeatedly decodes it into tokens. The decoded tokens are corrupted and fed back into subsequent latent denoising steps. The paper derives a variational training objective and evaluates the approach on Sudoku, Countdown, and LM1B, with ablations of the continuous latent, discrete conditioning, and corruption kernel.

Critical strengths
  1. The method provides a concrete coupling between continuous denoising and discrete token feedback. Reading tokens from the latent at every step is a meaningful structural distinction from hybrids that update continuous and discrete states through separate prediction heads.

  2. The experiments show useful gains on structured tasks, particularly Countdown CD5, and the component ablations support the importance of both latent dynamics and token feedback. Evaluation on LM1B provides evidence beyond the structured-task setting.

Critical weaknesses
  1. The contribution relative to CCDD needs a more precise empirical interpretation. Appendix B.4 provides a useful adapted baseline, but it replaces the original backbone and frozen per-token embedding setup with the authors’ backbone and learned latent. This tests a particular architectural adaptation rather than the original method as a whole. The paper should distinguish conclusions about the coupling design from conclusions about superiority over CCDD.

  2. The corruption kernel has a large effect: on hard Sudoku, the best absorbing configuration reaches 48.25%, compared with 72.41% for uniform corruption. This is much larger than the reported gap to the adapted CCDD baseline. The component ablations support the overall design, but the evidence does not yet cleanly separate the benefit of hierarchical coupling from the corruption and token-update choices.

Questions for authors
  1. Which discrete corruption kernel and token-update rule are used in the adapted CCDD baseline? Does the advantage remain when these choices and the sampling budget are matched?

  2. How sensitive is the CCDD comparison to the discrete-loss weight and the replacement of its original embedding representation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer jMu9 (1/2)

We thank the reviewer for their time and feedback. We appreciate the reviewer noting that "reading tokens from the latent at every step is a meaningful structural distinction from hybrids that update continuous and discrete states through separate prediction heads", and that the experiments "show useful gains on structured tasks, particularly Countdown CD5". We answer questions next.

Recap. HC-DLM denoises a continuous latent, decodes all tokens at every step, and feeds the re-noised tokens back as a condition, called the token scaffold. The corruption kernel determines how tokens are re-noised. CCDD instead predicts tokens and continuous embeddings in parallel, with a separate head for each. Classifier-free guidance (CFG) controls the strength of a conditioning signal, with w=1w=1 meaning no CFG. Generative perplexity (Gen. PPL) measures sample quality under GPT-2 Large (lower is better). We compare it at matched unigram entropy because reducing sample diversity can also lower Gen. PPL.

W1: Original vs. adapted CCDD

We have now separated the two comparisons: original CCDD on LM1B, and adapted CCDD on Sudoku and Countdown. HC-DLM improves over original CCDD on LM1B both without and with CFG. The LM1B comparison retains CCDD's original backbone and frozen per-token embeddings from a pretrained text encoder. For Sudoku and Countdown, whose implementations are absent from CCDD's official code, we use our backbone and learned latent (Appendix B.4). The revised paper states this distinction in Section 4.1 and Appendix B.4.

We reproduced both CCDD variants, MDiT and MMDiT, using the official code. Both match or improve on the reported validation perplexity (28.6 vs. 29.22 for MDiT and 23.15 vs. 25.76 for MMDiT). We then evaluate generation at matched entropy, without and with CFG:

LM1B, Gen. PPL ↓ at entropy4.04.14.24.3
Without CFG
CCDD-MMDiT174.6 ± 2.9315.0 ± 14.6571.3 ± 9.7770.0 ± 7.9
CCDD-MDiT71.6 ± 0.6106.6 ± 5.0176.7 ± 1.6296.6 ± 2.1
Continuous self-conditioning–––139.1 ± 1.4
HC-DLM–69.1 ± 0.773.7 ± 0.586.6 ± 1.6
With CFG
CCDD-MMDiT429.6 ± 11.8450.0 ± 11.8610.2 ± 13.81107.9 ± 3.6
CCDD-MDiT75.4 ± 0.4113.8 ± 0.3216.3 ± 1.4400.5 ± 3.6
Continuous self-conditioning75.6 ± 0.180.8 ± 0.491.6 ± 0.2127.6 ± 1.6
HC-DLM46.2 ± 0.451.1 ± 1.059.9 ± 1.375.7 ± 1.8

Without CFG, up to 1,024 sampling steps. With CFG, 128 steps. A dash means the entropy is not reached.

HC-DLM achieves substantially better Gen. PPL than both original CCDD variants under either sampling mode at shared entropies. The improvement on LM1B also holds with CCDD's original representation (Appendix E.1).

Q1: Kernel, update rule and sampling budget of the adapted CCDD

The adapted CCDD uses uniform corruption and a direct re-noising token update, matching HC-DLM and the 50-step sampling budget of Table 1. The advantage on Sudoku Hard remains under these matched choices.

Kernel. Original CCDD uses absorbing corruption. We tested both kernels in the adapted baseline and report its stronger uniform-kernel version in Table 1.

Token-update rule. Both models obtain the next token state by re-noising a clean-token prediction with the forward kernel. The adapted CCDD uses its token head, which reads both tokens and latent. HC-DLM decodes from the denoised latent alone.

Other settings. Both models use the same 6M-parameter DiT backbone and a single learned latent token (M=1M=1). The adapted CCDD adds a discrete loss with weight 0.2. These settings are now stated in Appendix B.4 and Appendix E.5.

To test the sampling budget, we evaluate the Sudoku Hard checkpoints at the same number of sampling steps (Appendix E.3):

Sudoku Hard accuracy (%) ↑ at sampling steps4060100200
CCDD (adapted), absorbing kernel56.2761.2767.0574.59
CCDD (adapted), uniform kernel69.5271.5273.8074.32
HC-DLM69.2872.9877.1081.90

With uniform corruption in both models, HC-DLM performs better from 60 steps onward and improves more as the sampling budget increases.

AuthorsOpenReview ↗

Response to Reviewer jMu9 (2/2)

W2: Coupling vs. corruption kernel

We separate the coupling and kernel contributions by comparing coupling designs at a fixed kernel and token-update rule, and comparing kernels within HC-DLM. The coupling benefit remains when the corruption kernel and token-update rule are held fixed.

On LM1B, we compare no feedback, continuous self-conditioning and token feedback on our backbone. All models are trained for 280k steps and sampled with 128 steps. Continuous self-conditioning feeds back the denoiser's previous clean-latent estimate, as in LangFlow. The first group uses no CFG. The second compares guided feedback at matched entropy (Appendix E.6).

To test whether the benefit of token feedback extends beyond uniform and absorbing corruption to kernels that use token similarity, we also evaluate kkNN, Gaussian-neighbour and graph-walk kernels. These kernels draw replacement tokens from neighbours in the frozen encoder's representation space. We compare them with uniform HC-DLM on the same backbone at matched entropy.

LM1B without CFGGen. PPL ↓ / entropy
No feedback274.5 ± 4.3 / 4.33
Continuous self-conditioning176.7 ± 1.1 / 4.31
HC-DLM, uniform kernel132.7 ± 0.8 / 4.35
LM1B with CFG, Gen. PPL ↓ at entropy4.04.14.24.3
Continuous self-conditioning97.4 ± 0.4101.4 ± 0.6105.1 ± 0.9144.1 ± 4.4
HC-DLM, absorbing kernel64.6 ± 1.270.9 ± 0.776.0 ± 0.895.5 ± 0.6
HC-DLM, kkNN kernel62.4 ± 0.862.5 ± 0.766.2 ± 1.2123.2 ± 1.6
HC-DLM, Gaussian-neighbour kernel57.6 ± 0.561.0 ± 1.367.7 ± 0.5134.6 ± 3.0
HC-DLM, graph-walk kernel65.8 ± 1.467.0 ± 0.669.4 ± 0.7100.3 ± 1.1
HC-DLM, uniform kernel64.0 ± 0.968.1 ± 0.671.9 ± 1.183.6 ± 2.1

Best in bold and second best in italics.

Token feedback achieves better Gen. PPL than both continuous self-conditioning and no feedback without reducing entropy. With CFG, all five scaffold kernels outperform guided continuous self-conditioning across the tested entropies. Their relative performance varies with entropy, with uniform corruption performing best near real-text entropy. Thus the benefit of token feedback on LM1B is present beyond uniform corruption alone.

Coupling at a fixed kernel. On Sudoku, HC-DLM and adapted CCDD use the same uniform kernel and direct re-noising update, and HC-DLM performs better on Hard at moderate and larger sampling budgets (Appendix E.3).

Kernel within HC-DLM. Uniform corruption improves Sudoku Hard accuracy from 48.25% to 72.41% (Table 6). Both kernels decode every token anew at each step. To test how they handle incorrect feedback, we inject three wrong scaffold digits at t=0.9t=0.9 into models trained to convergence, using 50 sampling steps without CFG and 2,000 puzzles per split. The absorbing variant copies 30–42% of the injected errors and loses 17–23 accuracy points, while the uniform variant copies ≤ 5% and loses at most 1.5 points. The accuracy changes are measured against each model's sampling without injection. This directly shows better robustness to scaffold errors under uniform corruption (Appendix E.6).

Q2: Discrete-loss weight and embedding representation

The adapted CCDD is sensitive to its discrete-loss weight, and the selected weight of 0.2 performs best among the three tested values. HC-DLM achieves substantially better Gen. PPL on LM1B with CCDD's original embedding representation retained.

Discrete-loss weight. We trained the adapted CCDD (uniform kernel) with three weights on its discrete loss (Appendix E.5):

Discrete-loss weightSudoku Easy (%) ↑Sudoku Hard (%) ↑
0.0283.2858.52
0.2 (Table 1)94.6570.73
26.194.64

Weight 0.2 performs best on both splits. Reducing it to 0.02 lowers accuracy by 11 points on Easy and 12 points on Hard, and increasing it to 2 leaves fewer than 7% of the boards solved. This supports the selected weight, while also showing that the adapted baseline is sensitive to the relative loss weighting.

Embedding representation. The LM1B experiment uses CCDD's official code and retains its frozen per-token embeddings from a pretrained text encoder. HC-DLM achieves substantially better Gen. PPL at matched entropy in that setting as well (Appendix E.1). The Sudoku comparison uses a shared learned latent and evaluates the adapted architecture.

Dear Reviewer jMu9,

We thank the reviewer for the helpful suggestions. In our response, we separated the comparisons with the original and adapted CCDD, matched the kernel, update rule and sampling budget of the adapted baseline, and added the kernel robustness and loss-weight analyses. We would be grateful to know whether these address the reviewer's concerns, and we look forward to the reviewer's further input during the discussion period.

Reviewer_CpBEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces HC-DLM, a generative diffusion language model which couples discrete token generation with a continuous state in a single denoising process, in which each successive discrete state conditions the respective continuous denoising step. The authors provide an ELBO and practical training loss for this generative process. Experiments on Sudoku, Countdown, and LM1B show competitive performance in respective accuracy/perplexity metrics.

Critical strengths
  • The paper proposes an intuitive and theoretically-grounded approach to joint continuous-discrete denoising process that shows promise given synthetic and small-scale language modeling experiments. The approach clearly improves sample-efficiency compared to MDMs whose denoising processes do not have a notion of continuous state.
Critical weaknesses
  • The LM1B perplexity of 75.5 reported in Table 3 appears to use a CFG scale w=2.75w=2.75 rather than 1, meaning it makes substantial use of CFG and relies on the discrete token scaffold — without this weight Perplexity would be at 130 based on Figure 5, well above the baselines. Could the authors discuss the relationship between HC-DLM's empirical reliance on CFG and the better performance with uniform kernel? Do any of the baselines use CFG, and if not, is there a conflation of comparing between methods and comparing between guided vs unguided approaches?

  • The experiments do not exceed LM1B sentence-scale (e.g. OWT), and the results are not impressively better than all the baselines that are provided. For example, HC-DLM improves only a little bit on Sudoku Hard over CCDD (no error bars are reported in the table), and decreases on Sudoku Easy compared to CCDD. The Gen. PPL on LM1B is quite close to Plaid.

  • The "Information Emergence" and "Inference Efficiency" ablations are not very informative or surprising. A denoising process which uses token conditioning signal for scaffolding will intuitively achieve faster convergence than one which does not. There is also no analysis of how exactly the evolving continuous state improves the token dependence / per-position marginals problem that was the motivating problem in the introduction. Moreover, in Figure 3b, most continuously-augmented approaches will sample in fewer steps than an MDM — the fair comparison is therefore against other continuously-augmented approaches.

  • Latent-Augmented Discrete Diffusion Models (https://arxiv.org/pdf/2510.18114) should be discussed in more detail (see question).

Questions for authors
  • This paper is quite closely related to the joint setting of Latent-Augmented Discrete Diffusion Models (https://arxiv.org/pdf/2510.18114), which "introduces a learnable auxiliary latent channel and performs diffusion over the joint (token, latent) space", albeit with a dedicated discrete denoising transition kernel. However, the authors mention LADD only in passing in the related works section. LADD also derives an ELBO for both their joint and sequential instantiations and trains on LM1B. Could the authors discuss LADD in more depth and include LADD-joint and LADD-seq in Table 10?
4 comments
AuthorsOpenReview ↗

Response to Reviewer CpBE (1/4)

We thank the reviewer for their time and feedback. We appreciate the reviewer noting that the paper proposes "an intuitive and theoretically-grounded approach" that "clearly improves sample-efficiency compared to MDMs". We answer questions next.

Recap. HC-DLM denoises a continuous latent. At every step it decodes all tokens, re-noises them, and feeds them back to the denoiser as a condition, called the token scaffold. Classifier-free guidance (CFG) controls the strength of this condition, with w=1w=1 meaning no CFG. Generative perplexity (Gen. PPL) measures sample quality under GPT-2 Large (lower is better). Since reducing diversity can also lower Gen. PPL, we report unigram entropy and compare methods at matched entropy.

W1 (part 1): Does the LM1B comparison mix guided and unguided models?

HC-DLM improves over LangFlow without CFG and over CCDD both without and with CFG, at matched entropy.

Table 3 uses the baselines' reported sampling designs, without added CFG. We now distinguish these results from comparisons in which both methods use CFG. In the revised paper, Section 4.4 states which models use CFG, and the comparison below is Appendix E.1.

Without CFG (w=1w=1). At entropy 4.3, HC-DLM achieves Gen. PPL 86.6 ± 1.6, compared with 296.6 ± 2.1 for CCDD, 139.1 ± 1.4 for continuous self-conditioning, and 92.3 reported by LangFlow at entropy 4.31. The HC-DLM, continuous self-conditioning and CCDD-MDiT checkpoints are trained for 1M steps. Continuous self-conditioning uses our latent-diffusion backbone and feeds back the denoiser's previous clean-latent estimate instead of tokens.

With CFG. For CCDD, we use the CFG from its official implementation. For continuous self-conditioning on our backbone, we combine predictions with and without the previous latent estimate, just as HC-DLM combines predictions with and without the scaffold. This tests whether the advantage persists when continuous feedback is also guided. The table includes both sampling modes at matched entropy. Real LM1B text has entropy about 4.3.

LM1B, Gen. PPL ↓ at entropy4.04.14.24.3
Without CFG
CCDD-MMDiT174.6 ± 2.9315.0 ± 14.6571.3 ± 9.7770.0 ± 7.9
CCDD-MDiT71.6 ± 0.6106.6 ± 5.0176.7 ± 1.6296.6 ± 2.1
Continuous self-conditioning–––139.1 ± 1.4
HC-DLM–69.1 ± 0.773.7 ± 0.586.6 ± 1.6
With CFG
CCDD-MMDiT429.6 ± 11.8450.0 ± 11.8610.2 ± 13.81107.9 ± 3.6
CCDD-MDiT75.4 ± 0.4113.8 ± 0.3216.3 ± 1.4400.5 ± 3.6
Continuous self-conditioning75.6 ± 0.180.8 ± 0.491.6 ± 0.2127.6 ± 1.6
HC-DLM46.2 ± 0.451.1 ± 1.059.9 ± 1.375.7 ± 1.8

Without CFG, up to 1,024 sampling steps. With CFG, 128 steps. A dash means the entropy is not reached.

HC-DLM achieves substantially better Gen. PPL than continuous self-conditioning and both CCDD variants under either sampling mode. The advantage over CCDD is largest near the entropy of real text, so it persists at realistic sample diversity. Our reproduction of CCDD uses its official code and matches its reported validation perplexity (28.6 vs. 29.22).

These results separate HC-DLM without and with CFG and report sample entropy explicitly. Figure 7 of the revised paper plots Gen. PPL against entropy for all measured points.

AuthorsOpenReview ↗

Response to Reviewer CpBE (2/4)

W1 (part 2): CFG and the uniform kernel

Uniform corruption improves robustness to errors in the scaffold, while CFG controls how strongly the denoiser follows the scaffold. The uniform kernel's error-correction benefit is present without CFG, so it does not require guidance. The two kernels are defined in Section 2 and CFG in Appendix B.7.

Without CFG. On Sudoku Hard, the uniform kernel is substantially more accurate than the absorbing kernel at w=1w=1 (Table 6). On LM1B, the feedback comparison also favors token feedback over continuous self-conditioning and latent diffusion without feedback, at comparable or higher entropy.

With CFG. We compare both scaffold variants with continuous self-conditioning on our backbone, applying CFG to each model's feedback condition. This tests whether strengthening continuous feedback alone accounts for the improvement. All models are trained for 280k steps and sampled with 128 steps (Appendix E.6).

LM1B without CFGGen. PPL ↓ / entropy
No feedback274.5 ± 4.3 / 4.33
Continuous self-conditioning176.7 ± 1.1 / 4.31
HC-DLM, uniform kernel132.7 ± 0.8 / 4.35
LM1B with CFG, Gen. PPL ↓ at entropy4.04.14.24.3
Continuous self-conditioning97.4 ± 0.4101.4 ± 0.6105.1 ± 0.9144.1 ± 4.4
HC-DLM, absorbing kernel64.6 ± 1.270.9 ± 0.776.0 ± 0.895.5 ± 0.6
HC-DLM, uniform kernel64.0 ± 0.968.1 ± 0.671.9 ± 1.183.6 ± 2.1

Both scaffold variants achieve better Gen. PPL than guided continuous self-conditioning. The uniform variant performs best near real-text entropy, while the two variants are comparable at entropy 4.0.

Robustness to scaffold errors. We test both kernel variants. At t=0.9t=0.9, we overwrite three scaffold digits with wrong digits and continue sampling without CFG, using 50 steps and 2,000 puzzles per split. The absorbing variant copies 30–42% of the injected errors into its next readout and loses 17–23 accuracy points relative to sampling without injection. The uniform variant copies ≤ 5% and loses at most 1.5 points. This shows that uniform corruption improves error correction even without CFG (Appendix E.6).

AuthorsOpenReview ↗

Response to Reviewer CpBE (3/4)

W2 (part 1): Experiments beyond LM1B

The evaluation now extends to long-sequence generation on OpenWebText at length 1,024. Near real-data entropy, HC-DLM achieves the best Gen. PPL of all compared methods, including the substantially larger ELF-M. This extends the evaluation beyond the length-128 sequences of LM1B. The table compares methods around the entropy of real OpenWebText text (5.44). MDLM and Duo are discrete diffusion baselines. LangFlow and ELF are continuous diffusion models. HC-DLM uses scaffold CFG, and ELF uses CFG on continuous self-conditioning, making ELF a guided continuous baseline. We report deterministic (ODE) and stochastic (SDE) samplers separately.

OpenWebText, Gen. PPL ↓ at entropyParamsSteps5.205.255.305.355.405.455.50
Discrete diffusion
MDLM130M64–102433.537.141.247.254.462.070.4
Duo130M64–102432.936.840.645.651.559.968.6
ODE sampler
LangFlow130M64–102434.036.540.647.354.463.777.1
ELF-B105M6431.437.043.453.065.380.899.9
ELF-M342M64–29.835.242.952.965.784.2
HC-DLM125M25628.932.536.541.446.954.364.1
SDE sampler
LangFlow130M102435.439.744.449.7–––
ELF-B105M6426.931.737.6––––
ELF-M342M6423.127.433.539.2–––
HC-DLM125M25623.725.628.931.535.239.847.8

Interpolated at each target entropy. A dash means the entropy is not reached. A step range indicates a sweep. Best in bold and second best in italics within each sampler.

Due to resource and time constraints during the rebuttal, we trained HC-DLM for only 300k denoiser steps, about 30% of LangFlow's denoiser-training budget at the same batch size. Gen. PPL is a meaningful measure of quality only near the entropy of real text (5.44), since lowering the entropy below it reduces Gen. PPL at the cost of diversity. In this range (entropy 5.35 to 5.50), HC-DLM achieves the best Gen. PPL of all methods with both samplers, including ELF-M with nearly three times as many parameters (125M vs. 342M). With the SDE sampler, ELF-M does not reach entropy 5.40. HC-DLM also achieves better Gen. PPL than MDLM, Duo, LangFlow and the comparably sized ELF-B throughout their shared entropy range in the table. This experiment is now Appendix E.2 of the revised paper, and Section 4.4 and Table 4 summarize it. Figure 8 of the revised paper plots Gen. PPL against entropy for both samplers.

W2 (part 2): Size of the gains and error bars

The improvements over CCDD are substantial on Countdown and LM1B. HC-DLM also improves on Sudoku Hard and remains comparable on Sudoku Easy. The table below reports sampling standard deviations alongside the structured-task accuracies.

ModelSudoku Easy (%) ↑Sudoku Hard (%) ↑Countdown CD4 (%) ↑Countdown CD5 (%) ↑
CCDD (adapted), uniform kernel94.65 ± 0.0970.73 ± 0.1581.18 ± 0.5325.35 ± 0.59
HC-DLM94.21 ± 0.0972.41 ± 0.2084.41 ± 0.9137.52 ± 0.46

Sudoku and Countdown. The reported results are comparable with CCDD on Sudoku Easy and better on Sudoku Hard and both Countdown settings, with the largest improvement on the harder five-number task (CD5). The measured sampling standard deviations are small across all four settings. For these tasks, CCDD is adapted to our backbone and learned latent (Appendix B.4), and we use the stronger uniform-kernel version. The standard deviations are now reported in Appendix E.5.

LM1B. At matched entropy, HC-DLM achieves substantially better Gen. PPL than the original CCDD, both without and with CFG, and than guided continuous self-conditioning on our backbone. For Plaid, we use the Gen. PPL of 77.3 reported by LangFlow, without a corresponding mean sample entropy. The LM1B comparison at matched entropy is Appendix E.1.

AuthorsOpenReview ↗

Response to Reviewer CpBE (4/4)

W3 (part 1): How the continuous state improves token dependence

The shared continuous state improves consistency among tokens decoded in parallel, and token feedback improves consistency further during denoising. In masked diffusion (MDM), tokens sampled in the same step are conditionally independent given the current token context. In HC-DLM, every cell is decoded from the same continuous representation of the board. Although the token predictor factorizes given that latent, integrating over the shared latent can induce dependence among cells. The latent denoiser also processes the whole scaffold, so an update at one cell can affect subsequent predictions at other cells.

Concretely, we take each model's prediction for all blank cells at the first sampling step, which is the fully parallel case, and count the rows, columns and boxes that contain a repeated digit. We call them conflicting units, and a board has 27 units in total. We count them again in the final output.

ModelConflicting units, all cells at the first step ↓ (Easy / Hard)Conflicting units, final output ↓ (Easy / Hard)
MDM19.3 / 20.80.07 / 1.30
Latent DM9.4 / 13.71.95 / 2.82
CCDD (adapted), absorbing kernel8.4 / 13.30.26 / 1.16
CCDD (adapted), uniform kernel12.5 / 15.60.38 / 0.94
HC-DLM6.7 / 11.90.11 / 0.62

HC-DLM has the fewest first-step conflicts on both splits, including fewer than latent diffusion without token feedback and both adapted CCDD variants. On Sudoku Hard, it also has the fewest final conflicts. The first-step comparison measures consistency before iterative token commitments, while the final comparison measures consistency after refinement (Appendix E.4).

W3 (part 2): Comparison with continuously-augmented approaches

HC-DLM's advantage also holds against other models that denoise a continuous state. At matched sampling steps and entropy, it improves over LangFlow and CCDD even without CFG. Both baselines denoise a continuous state. We compare them on LM1B at matched sampling steps and matched entropy.

LM1B, Gen. PPL ↓ at sampling steps (entropy)16 (4.35)32 (4.33)64 (4.32)128 (4.31)
CCDD-MDiT383.0323.5302.1314.6
Continuous self-cond.319.8238.4191.1163.8
LangFlow179.6127.3104.892.3
HC-DLM154.7116.396.887.9

Target entropies at each step count. All models use w=1w=1.

The improvement over LangFlow is largest at the smallest sampling budget. This comparison shows that the advantage extends to continuous and hybrid baselines, beyond the comparison with MDM (Appendix E.3). The separate CFG comparison also favors HC-DLM over CCDD and guided continuous self-conditioning at matched entropy.

W4 / Q: Relation to LADD

The expanded comparison below includes both LADD-seq and LADD-joint. The structural distinction is the token transition: LADD retains a masked token chain, while HC-DLM regenerates every token from the denoised latent at each step. LADD-seq generates a latent first and then denoises tokens conditioned on it. LADD-joint denoises both channels together. The comparison below uses ktk_t for tokens and xtx_t for continuous latents. We have added both LADD variants to Table 11 (Table 10 in the submission), the discussion below to Appendix C.4, and LADD to the related work in Section 5.

MethodContinuous variableUpdate of the continuous variableInput of the token predictorDecoded token can change later
LADD-seqencoder latentlearned denoiser, run before the tokenskt,,x0k_t,\\,x_0no
LADD-jointencoder latentlearned denoiser, conditioned on ktk_tkt,,xtk_t,\\,x_tno
HC-DLMglobal latentlearned denoiser, conditioned on ktk_tx^0\hat{x}_0 onlyyes

LADD-joint is closest on the continuous side, since its learned latent denoiser also conditions on the tokens. The difference is the token transition: LADD conditions on the current tokens and retains unmasked tokens, whereas HC-DLM reads only the denoised latent and can revise every token until the final step. LADD also identifies CCDD as a special case of joint Co-LADD, so our CCDD experiments compare with one member of that family.

Comments on the paper

1 comment

Dear reviewers,

The author-reviewer discussion period is ending soon. If you have not done so yet, please read the authors’ rebuttal and reply to it. Tell the authors whether it addresses your concerns, and update your score if your assessment has changed.

The ICLR 2027 Reviewer Guidelines list « discussing with the authors to clarify and improve the paper » as one of the core duties you accepted as a reviewer. With the high number of submissions this year, your input matters more than ever. Recall that reviewers that do not fulfill their duties may also face penalties on their own submissions.

Thank you for your time and help.

Best regards,


Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.