- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Hierarchical Continuous Diffusion Language Models (HC-DLM), which combine a continuous latent trajectory with a discrete token scaffold. Tokens are repeatedly read out, re-noised, and fed back to guide continuous denoising. The paper presents a hierarchical generative model and a variational lower bound, with experiments on Sudoku, Countdown, and LM1B.
Critical strengths
Using a discrete token scaffold to guide continuous denoising is a promising design. The structured-task results are convincing: at a similar sampling-time parameter count, HC-DLM achieves 37.52% accuracy on CD5, compared with 25.35% for the reproduced CCDD baseline. The Sudoku ablation also shows a substantial drop when the scaffold is removed. The analyses of noise kernels, sampling steps, and runtime are useful.
Critical weaknesses
-
The connection between the hierarchical model and the sampler is unclear. Eq. (3) conditions only on , while Algorithm 2 generates it from computed using . It is not clear why this update preserves the continuous latent as the only persistent state, which is central to the paper's distinction from other hybrid models.
-
Evidence for the token scaffold's contribution to language generation is less direct. The component ablation in Table 4 covers only Sudoku. The LM1B comparison includes LangFlow with self-conditioning, but the models and training setups differ, making the benefit over continuous self-conditioning difficult to isolate.
Questions for authors
-
In Algorithm 2, the next token state is generated from , which depends on . How does this correspond to in Eq. (3)? Is this shorthand notation, or an approximation used in sampling?
-
Could the authors compare no feedback, continuous self-conditioning, and full HC-DLM on LM1B using the same backbone, tokenizer, training budget, and number of sampling steps? Using for HC-DLM and reporting Gen. PPL, entropy, and sampling time would help isolate the token scaffold's contribution.
5 comments
Response to Reviewer mmoU (2/2)
W2 / Q2: Feedback ablation on LM1B
We ran the requested comparison on LM1B: same backbone, tokenizer, frozen encoder and token predictor, 1M training steps, 128 sampling steps and . At the same entropy, token feedback reduces Gen. PPL from 163.8 (continuous self-conditioning) and 251.7 (no feedback) to 87.9.
The three models differ in what the denoiser receives at each step: nothing, its previous clean-latent estimate (continuous self-conditioning), or the decoded and re-noised tokens (HC-DLM). LangFlow and CCDD are included at the same sampling steps and entropy.
| LM1B, | Gen. PPL ↓ / entropy | Sampling time (s) |
|---|---|---|
| No feedback | 251.7 / 4.34 | 1.11 ± 0.01 |
| Continuous self-conditioning | 163.8 / 4.31 | 1.14 ± 0.0003 |
| LangFlow | 92.3 / 4.31 | 3.48 ± 0.002 |
| CCDD-MDiT | 314.6 / 4.31 | 19.4 ± 0.0004 |
| HC-DLM (token scaffold) | 87.9 / 4.31 | 2.84 ± 0.0004 |
Sampling time is the median of 5 runs for a batch of 64 on one RTX PRO 6000 GPU with compilation.
Quality. At matched entropy, token feedback lowers Gen. PPL from 163.8 to 87.9 (46%) relative to continuous self-conditioning, and from 251.7 (65%) relative to no feedback. HC-DLM also outperforms LangFlow and CCDD at the same sampling steps.
Sampling time. HC-DLM samples faster than LangFlow and nearly 7× faster than CCDD (19.4 s), while achieving better Gen. PPL than both. Relative to the two latent baselines (1.11 s and 1.14 s), the token scaffold brings a much larger quality gain than its sampling cost: Gen. PPL drops from 163.8 and 251.7 to 87.9.
The feedback comparison and the sampling times are now in Appendix E.6 of the revised paper, and the comparison with LangFlow and CCDD at matched sampling steps is in Appendix E.3.
Dear Reviewer mmoU,
We thank the reviewer for the helpful suggestions. In our response, we corrected the notation in Eq. (3) and ran the requested LM1B comparison of no feedback, continuous self-conditioning and the token scaffold (now Appendix E.6). We would be grateful to know whether these address the reviewer's concerns, and we look forward to the reviewer's further input during the discussion period.
Response to Reviewer mmoU (1/2)
We thank the reviewer for their time and feedback. We appreciate the reviewer noting that "using a discrete token scaffold to guide continuous denoising is a promising design", that "the structured-task results are convincing", and that "the analyses of noise kernels, sampling steps, and runtime are useful". We answer questions next.
Recap. HC-DLM denoises a continuous latent xt. At every step it decodes all tokens, re-noises them, and feeds the result kt back as a condition, called the token scaffold. Classifier-free guidance (CFG) controls the strength of this condition, with w=1 meaning no CFG. Generative perplexity (Gen. PPL) measures sample quality under GPT-2 Large (lower is better). We report unigram entropy alongside it because reducing sample diversity can also lower Gen. PPL.
W1 / Q1: Eq. (3) vs. Algorithm 2
This is a conditioning-notation omission: Eq. (3) should use pθ(kt−1∣xt−1,xt). The precise readout depends on two adjacent latents, while still requiring no separate token transition chain. Algorithm 2 computes x^0=dϕ(xt,kt,t/T), takes an Euler step, and decodes and re-noises tokens from x^0. For its deterministic Euler update,
xt−1=xt−t1(xt−x^0)⟹x^0=xt−t(xt−xt−1).
The omitted conditioning argument is therefore xt, and the next scaffold can be written as a readout of the adjacent continuous states:
pθ(kt−1∣xt−1,xt)=∑k0q(kt−1∣k0),pθ(k0∣xt−t,(xt−xt−1)).
The current scaffold influences x^0 and the latent update. Once the adjacent latents are fixed, however, the next scaffold has no additional dependence on the earlier token states. Every token is decoded anew, so there is no independent chain of token commitments. The precise claim is that the continuous trajectory carries cross-step information, not that xt−1 alone is a sufficient state for this readout.
This correspondence is specific to the deterministic Euler sampler of Algorithm 2. The consistent model notation uses pθ(kt∣xt,xt+1) for intermediate readouts and retains the terminal initialization kT∼π. Writing pθ,ϕadj for that normalized generative model, Jensen's inequality gives
logpθ,ϕadj(k0)≥Eqψ![logqψ(x0:T,k1:T∣k0)pθ,ϕadj(k0:T,x0:T)].
The token-prediction terms and their expectations retain the adjacent-latent argument.
We have made this correction throughout the revised paper: Eq. (3) and the description of the token predictor in Section 3.1, the bound in Proposition 1 and its proof in Appendix A, the token predictor in Appendix B.1, and the conditional model in Appendix B.2. Section 3.4 (Eq. (7)) now states the correspondence with Algorithm 2.