- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Hierarchical Continuous Diffusion Language Models (H-CDLMs), a framework that jointly diffuses token representations and coarse-grained semantic clusters derived from pretrained token embeddings. The author combines the token cluster in HDLM to the bit stream diffusion from CoBit.
Critical strengths
1, The presentation of the proposed method is easy to follow.
Critical weaknesses
1, Experiment setups and results are not persuasive enough. For example, for the measurement of Gen PPL, all methods are supposed to use the same sampling steps, or it would be unfair to methods with fewer sampling steps. However, there are 5 different steps used in Table 1 (32, 128, 256, 512, 1024), and the proposed method uses much more steps than some of the baselines (LangFlow, FMLM), making the advantage of the proposed method suspicious. Also, Gen PPL is a hackable metric. Therefore, reporting Gen PPL only is not enough to prove the soundness of the method. PPL or performance on reasoning tasks like Sudoku should be included.
2, The soundness and novelty of the method is also questionable. The method seems to be a combination of HDLM [1] and CoBit [2], where the semantic cluster comes from HDLM and the continuous diffusion language modeling from CoBit. I do not see the necessity and the intuition of combining these two methods.
References:
[1] Next Semantic Scale Prediction via Hierarchical Diffusion Language Models.
[2] CoBit: Language Modeling with Bitstream Diffusion
Questions for authors
1, What is the intuition of combining HDLM and CoBit? Why do the clusters need to be joint diffused?
2, Can you report the PPL results of H-CoBit over LM1B and OpenWebText.
6 comments
Author response to 34Fr 2/2
B. GenPPL's soundness: GenPPL is hackable by decreasing sample entropy. We systematically compare models' GenPPL at matched dataset entropy to avoid this caveat. We note that GenPPL/entropy frontiers remain a main evaluation metric in recent diffusion language model works [3,4,5].
C. GenPPL is not our only metric: in the initial pdf, we also evaluated MAUVE (Appendix E1, table 11) and GSM8K accuracy (Section 5.2, Table 2). Sudoku makes little sense in our case, since we construct semantic token clusters (sudoku tokens are just numbers, we cannot regroup them semantically).
D. PPL evaluation: neither CoBit nor FLM report an NLL bound or PPL results, so reporting our own H-CoBit or H-FLM bounds would leave no published baseline to compare against. Moreover, apart from LangFlow [6], recent continuous diffusion language model work does not report PPL [3, 4, 5, 7]. We further discuss evaluation of diffusion language models in Appendix B.8.
This practice is principled. The variational bound of continuous diffusion includes a score-error component (its divergence-free part) that does not affect the generated samples [9, 8]. This component is absent for conservative score parametrizations, but generically nonzero for unconstrained networks, and can be substantial for Transformers, which continuous diffusion language models use [9]. Two models with equally good samples can therefore have different bounds.
For H-CDLMs, the bound also includes score error on the cluster channel, so a bound comparison with the base model would not isolate model quality. We therefore evaluate samples directly with GenPPL at matched data entropy, MAUVE (Table 11) and GSM8K accuracy (Table 2); H-CoBit improves on CoBit on all three.
References:
[1] Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. Boosting generative image modeling via joint image-feature synthesis. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025
[2] Alan Baade, Eric Ryan Chan, Kyle Sargent, Changan Chen, Justin Johnson, Ehsan Adeli, and Li Fei-Fei. Latent forcing: Reordering the diffusion trajectory for pixel-space image generation. In Forty-third International Conference on Machine Learning, 2026
[3] Justin Deschenaux and Caglar Gulcehre. Language modeling with hyperspherical flows, Transactions on Machine Learning Research, 2026
[4] Georgios Batzolis, Mark Girolami, and Luca Ambrogioni. Cobit: Language modeling with bitstream diffusion, 2026.
[5] Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, Kaiming He. ELF: Embedded Language Flows, 2026.
[6] Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, and Ge Liu. Langflow: Continuous diffusion rivals discrete in language modeling, 2026.
[7] Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Se-unghoon Hong, Nicholas M. Boffi, and Jinwoo Kim. Flow map language models: One-step language modeling via continuous denoising, 2026.
[8] Song, Durkan, Murray & Ermon. Maximum likelihood training of score-based diffusion models. NeurIPS, 2021.
[9] Naïl Khelifa, Richard Turner and Ramji. Venkataramanan. Diffusion models observe only gradients: A geometric perspective on score matching errors, 2026.
[10] Antonio Franca and Alexander Tong. Hacking generative perplexity: Why unconditional text evaluation needs distributional metrics , 2026.
[11] Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. Generative frontiers: Why evaluation matters for diffusion language models, 2026.
[12] Huangjie Zheng, Shansan Gong, Ruixiang Zhang, Tianrong Chen, Jiatao Gu, Mingyuan Zhou, Navdeep Jaitly, and Yizhe Zhang. Continuously augmented discrete diffusion model for categorical generative modeling, 2025.
[13] Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. CANDI: Hybrid discrete-continuous diffusion models, 2026.
Thank you for your response. I can see the potential of the method, where the token classes are useful to be generated together with the text tokens themselves. However, the current version of the paper seems to focus only on one variant of the method: its application to CoBit, while a comprehensive validation of application to other DLM frameworks is missing. This could be misleading and I suggest the author reorganize the paper.
Also, I still hold some technical concerns on the proposed method:
-
the sampling of H-CoBit uses stochastic sampling, while both FLM and LangFlow use deterministic sampling. According to experience in image diffusion, different samplers could have great impacts on the results. I think a unified sampling method is then necessary for a fair comparison.
-
The evaluation is still not persuasive enough to me. I agree that entropy and gen ppl could affect each other. However, the standard way of benchmarking gen ppl is reporting the pareto frontier instead of tuning the entropy to data's, the latter of which is adopted by the paper.
-
Since the proposed method needs to diffuse 2x tokens then normal, I think the scalability could be a problem.
Based on above views, I keep my score while still encouraging the author to re-submit the paper to the next cycle.
Author response to 34 Fr 1/2
Dear reviewer,
Thank you for your time and comments of our work. Below we address each of your questions. All new results are in the updated PDF.
1. Novelty/soundness:
Our motivation behind parallel diffusion of tokens and clusters is that the two levels can correct each other without an arbitrary ordering enforced. We provide further insights on the intuition and inner workings of H-CoBit (and H-CDLM in general) in our answer 2 to reviewer 7JTd and 4 to reviewer AtJr. This follows succesful innovations in the image diffusion literature, including ReDi [1] that diffuses DinoV2 and pixels in parallel, and Latent forcing [2] that diffuses these modalities via an ordered schedule.
Moreover, we show that:
A. Joint diffusion is empirically better than HDLM-style ordering. We trained a model allowing explicit orderings and compared schedules on LM1B (100k steps, GenPPL at data entropy):
The best schedules keep both levels evolving together, so the levels can correct each other. Full results are in Appendix E.4.4 and our response 2 to Reviewer 7JTd.
B. The framework is not specific to CoBit. Applied to FLM [7], it improves GenPPL by 8–14% at matched NFE and entropy on LM1B (Section 5.4).
Our contributions are: the first hierarchical modality for continuous text diffusion; per-modality information-uniform schedules, learned rather than hand-crafted; per-modality samplers; and a recipe shown on two host models and 3 datasets.
2. Evaluation:
A. GenPPL comparability: We have updated Langflow's GenPPL/Entropy numbers to its 256/512 NFE variant. This was not done in the initial submission because Langflow does not natively reach dataset entropy above NFE 128. This required adding temperature in place of Langflow's argmax sampling so as to reach dataset entropy. LangFlow reaches GenPPL 93.7/62.6 at dataset entropy on LM1B/OWT at NFE 512/256, which is above H-CoBit+PMS in all cases at matched NFE. We also remove FMLM to avoid confusion: it is a flow map model, distilled from FLM, so it is not directly comparable to our model (and is not meant to be sampled at high NFE), and its reported results are at low entropies (5.22 on OWT).
For the discrete and hybrid baselines (MDLM, Duo, CANDI, CADD), we report the results published in the original papers or in [10, 11]. On OpenWebText, NFE 128 is the only budget at which these works report results at data entropy, and no public OpenWebText checkpoints are available for CANDI and CADD, so we cannot re-evaluate them at other budgets. Comparing these baselines at NFE 128 does not disadvantage them: CANDI's authors report that increasing NFE beyond 128 does not significantly expand the GenPPL–entropy frontier of MDLM, Duo or CANDI ([13], Figure 14). As stated in the caption of Table 1, we report GenPPL at matched data entropy whenever the model reaches it, and we have made the NFE of every entry explicit in Table 1 to avoid any ambiguity. We will add our own results for MDLM, SEDD and Duo's dataset entropy GenPPL at NFE 256 before the end of the rebuttal.
At comparable or lower NFE, H-CoBit already performs well: