acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Continuous Diffusion Language Models

Abstract

Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation. Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations and diffusion/flow spaces. In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead. Drawing on the discrete DLM and continuous image diffusion literature on joint diffusion, we diffuse multiple modalities in parallel. These modalities represent tokens at different semantic granularities: in our instantiation, the tokens themselves and coarser clusters obtained by clustering pretrained token embeddings. We propose a general setup that allows per-modality samplers and schedules to enhance the interplay between modalities, and apply it to CoBit, yielding our H-CoBit model, which delivers large empirical gains. At dataset entropy, H-CoBit improves MAUVE and reaches a generative perplexity (GenPPL) of 49.4 on LM1B and 50.4 on OWT, improving on the baseline by 35.8 and 20.7 points and surpassing even discrete DLMs of comparable size. On GSM8K, it reaches 27.4% accuracy, outperforming prior diffusion and flow-based models.

Primary area:
generative models
Keywords:
Diffusion, Language Modeling, Multimodal

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_34FrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Continuous Diffusion Language Models (H-CDLMs), a framework that jointly diffuses token representations and coarse-grained semantic clusters derived from pretrained token embeddings. The author combines the token cluster in HDLM to the bit stream diffusion from CoBit.

Critical strengths

1, The presentation of the proposed method is easy to follow.

Critical weaknesses

1, Experiment setups and results are not persuasive enough. For example, for the measurement of Gen PPL, all methods are supposed to use the same sampling steps, or it would be unfair to methods with fewer sampling steps. However, there are 5 different steps used in Table 1 (32, 128, 256, 512, 1024), and the proposed method uses much more steps than some of the baselines (LangFlow, FMLM), making the advantage of the proposed method suspicious. Also, Gen PPL is a hackable metric. Therefore, reporting Gen PPL only is not enough to prove the soundness of the method. PPL or performance on reasoning tasks like Sudoku should be included.

2, The soundness and novelty of the method is also questionable. The method seems to be a combination of HDLM [1] and CoBit [2], where the semantic cluster comes from HDLM and the continuous diffusion language modeling from CoBit. I do not see the necessity and the intuition of combining these two methods.

References:

[1] Next Semantic Scale Prediction via Hierarchical Diffusion Language Models.

[2] CoBit: Language Modeling with Bitstream Diffusion

Questions for authors

1, What is the intuition of combining HDLM and CoBit? Why do the clusters need to be joint diffused?

2, Can you report the PPL results of H-CoBit over LM1B and OpenWebText.

6 comments
AuthorsOpenReview ↗

Author response to 34 Fr 1/2

Dear reviewer,

Thank you for your time and comments of our work. Below we address each of your questions. All new results are in the updated PDF.

1. Novelty/soundness:

Our motivation behind parallel diffusion of tokens and clusters is that the two levels can correct each other without an arbitrary ordering enforced. We provide further insights on the intuition and inner workings of H-CoBit (and H-CDLM in general) in our answer 2 to reviewer 7JTd and 4 to reviewer AtJr. This follows succesful innovations in the image diffusion literature, including ReDi [1] that diffuses DinoV2 and pixels in parallel, and Latent forcing [2] that diffuses these modalities via an ordered schedule.

Moreover, we show that:

A. Joint diffusion is empirically better than HDLM-style ordering. We trained a model allowing explicit orderings and compared schedules on LM1B (100k steps, GenPPL at data entropy):

  • Strict coarse-first, clusters before tokens: +77% to +90% worse than parallel diffusion, or unable to reach data entropy.
  • Clusters slightly ahead of tokens, but diffused jointly: 4% to 9% better.
  • Clusters behind tokens: 6% to 14% worse.

The best schedules keep both levels evolving together, so the levels can correct each other. Full results are in Appendix E.4.4 and our response 2 to Reviewer 7JTd.

B. The framework is not specific to CoBit. Applied to FLM [7], it improves GenPPL by 8–14% at matched NFE and entropy on LM1B (Section 5.4).

Our contributions are: the first hierarchical modality for continuous text diffusion; per-modality information-uniform schedules, learned rather than hand-crafted; per-modality samplers; and a recipe shown on two host models and 3 datasets.

2. Evaluation:

A. GenPPL comparability: We have updated Langflow's GenPPL/Entropy numbers to its 256/512 NFE variant. This was not done in the initial submission because Langflow does not natively reach dataset entropy above NFE 128. This required adding temperature in place of Langflow's argmax sampling so as to reach dataset entropy. LangFlow reaches GenPPL 93.7/62.6 at dataset entropy on LM1B/OWT at NFE 512/256, which is above H-CoBit+PMS in all cases at matched NFE. We also remove FMLM to avoid confusion: it is a flow map model, distilled from FLM, so it is not directly comparable to our model (and is not meant to be sampled at high NFE), and its reported results are at low entropies (5.22 on OWT).

For the discrete and hybrid baselines (MDLM, Duo, CANDI, CADD), we report the results published in the original papers or in [10, 11]. On OpenWebText, NFE 128 is the only budget at which these works report results at data entropy, and no public OpenWebText checkpoints are available for CANDI and CADD, so we cannot re-evaluate them at other budgets. Comparing these baselines at NFE 128 does not disadvantage them: CANDI's authors report that increasing NFE beyond 128 does not significantly expand the GenPPL–entropy frontier of MDLM, Duo or CANDI ([13], Figure 14). As stated in the caption of Table 1, we report GenPPL at matched data entropy whenever the model reaches it, and we have made the NFE of every entry explicit in Table 1 to avoid any ambiguity. We will add our own results for MDLM, SEDD and Duo's dataset entropy GenPPL at NFE 256 before the end of the rebuttal.

At comparable or lower NFE, H-CoBit already performs well:

  • LM1B: at NFE 256, H-CoBit (57.3) outperforms all discrete and hybrid baselines (98.6–119.9), including MDLM at NFE 1024. It does so at data entropy (4.34), while the baselines are reported at lower entropies (4.28–4.32), which favours them since GenPPL rises with entropy.
  • OpenWebText: at matched NFE 128, H-CoBit (67.8) is close to MDLM (66.2) and Duo (63.3), and better than CANDI (71.0); CADD (55.3, at H = 5.48) remains ahead. With the per-modality sampler at NFE 256, H-CoBit reaches 50.4 at data entropy (5.46), below all discrete baselines. At the same NFE 256, CADD's own paper [12] reports a GenPPL of 49.6 at entropy 5.27, far below data entropy: H-CoBit + PMS matches this GenPPL at much higher entropy, indicating a better GenPPL–entropy trade-off at matched NFE.
AuthorsOpenReview ↗

Author response to 34Fr 2/2

B. GenPPL's soundness: GenPPL is hackable by decreasing sample entropy. We systematically compare models' GenPPL at matched dataset entropy to avoid this caveat. We note that GenPPL/entropy frontiers remain a main evaluation metric in recent diffusion language model works [3,4,5].

C. GenPPL is not our only metric: in the initial pdf, we also evaluated MAUVE (Appendix E1, table 11) and GSM8K accuracy (Section 5.2, Table 2). Sudoku makes little sense in our case, since we construct semantic token clusters (sudoku tokens are just numbers, we cannot regroup them semantically).

D. PPL evaluation: neither CoBit nor FLM report an NLL bound or PPL results, so reporting our own H-CoBit or H-FLM bounds would leave no published baseline to compare against. Moreover, apart from LangFlow [6], recent continuous diffusion language model work does not report PPL [3, 4, 5, 7]. We further discuss evaluation of diffusion language models in Appendix B.8.

This practice is principled. The variational bound of continuous diffusion includes a score-error component (its divergence-free part) that does not affect the generated samples [9, 8]. This component is absent for conservative score parametrizations, but generically nonzero for unconstrained networks, and can be substantial for Transformers, which continuous diffusion language models use [9]. Two models with equally good samples can therefore have different bounds.

For H-CDLMs, the bound also includes score error on the cluster channel, so a bound comparison with the base model would not isolate model quality. We therefore evaluate samples directly with GenPPL at matched data entropy, MAUVE (Table 11) and GSM8K accuracy (Table 2); H-CoBit improves on CoBit on all three.

References:

[1] Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. Boosting generative image modeling via joint image-feature synthesis. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

[2] Alan Baade, Eric Ryan Chan, Kyle Sargent, Changan Chen, Justin Johnson, Ehsan Adeli, and Li Fei-Fei. Latent forcing: Reordering the diffusion trajectory for pixel-space image generation. In Forty-third International Conference on Machine Learning, 2026

[3] Justin Deschenaux and Caglar Gulcehre. Language modeling with hyperspherical flows, Transactions on Machine Learning Research, 2026

[4] Georgios Batzolis, Mark Girolami, and Luca Ambrogioni. Cobit: Language modeling with bitstream diffusion, 2026.

[5] Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, Kaiming He. ELF: Embedded Language Flows, 2026.

[6] Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, and Ge Liu. Langflow: Continuous diffusion rivals discrete in language modeling, 2026.

[7] Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Se-unghoon Hong, Nicholas M. Boffi, and Jinwoo Kim. Flow map language models: One-step language modeling via continuous denoising, 2026.

[8] Song, Durkan, Murray & Ermon. Maximum likelihood training of score-based diffusion models. NeurIPS, 2021.

[9] Naïl Khelifa, Richard Turner and Ramji. Venkataramanan. Diffusion models observe only gradients: A geometric perspective on score matching errors, 2026.

[10] Antonio Franca and Alexander Tong. Hacking generative perplexity: Why unconditional text evaluation needs distributional metrics , 2026.

[11] Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. Generative frontiers: Why evaluation matters for diffusion language models, 2026.

[12] Huangjie Zheng, Shansan Gong, Ruixiang Zhang, Tianrong Chen, Jiatao Gu, Mingyuan Zhou, Navdeep Jaitly, and Yizhe Zhang. Continuously augmented discrete diffusion model for categorical generative modeling, 2025.

[13] Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. CANDI: Hybrid discrete-continuous diffusion models, 2026.

Thank you for your response. I can see the potential of the method, where the token classes are useful to be generated together with the text tokens themselves. However, the current version of the paper seems to focus only on one variant of the method: its application to CoBit, while a comprehensive validation of application to other DLM frameworks is missing. This could be misleading and I suggest the author reorganize the paper.

Also, I still hold some technical concerns on the proposed method:

  1. the sampling of H-CoBit uses stochastic sampling, while both FLM and LangFlow use deterministic sampling. According to experience in image diffusion, different samplers could have great impacts on the results. I think a unified sampling method is then necessary for a fair comparison.

  2. The evaluation is still not persuasive enough to me. I agree that entropy and gen ppl could affect each other. However, the standard way of benchmarking gen ppl is reporting the pareto frontier instead of tuning the entropy to data's, the latter of which is adopted by the paper.

  3. Since the proposed method needs to diffuse 2x tokens then normal, I think the scalability could be a problem.

Based on above views, I keep my score while still encouraging the author to re-submit the paper to the next cycle.

Reviewer_AtJrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Continuous Diffusion Language Models, which jointly denoise token representations and auxiliary cluster identities derived from pretrained token embeddings. The implementation, H-CoBit, appends cluster bits to CoBit’s token representation and introduces per-modality noise schedules and stochastic sampling. Experiments report improvements on LM1B, OpenWebText, and GSM8K with small parameter and per-step computational overhead.

Critical strengths
  1. The method achieves substantial improvements over CoBit on LM1B and OpenWebText, with little additional parameter and per-step computational cost.

  2. The paper includes a good range of ablations, covering cluster count, auxiliary loss weight, random cluster assignments, noise schedules, and sampling choices.

Critical weaknesses
  1. Evaluation is limited to a single model family. Despite presenting a general framework for hierarchical continuous diffusion language modeling, the authors evaluate their method exclusively on CoBit. This does not establish whether the improvements generalize to other continuous diffusion models or depend on CoBit’s particular representation and training procedure. Evaluation on additional model families is necessary to support the broader claims. Results across multiple model scales, including medium and large variants, would also clarify whether the gains persist as capacity increases.

  2. The strongest generation results depend on the proposed sampler. The largest reported improvements on LM1B and OpenWebText are obtained using per-modality sampling. Although improvements are also reported without this sampler, the headline results combine changes to the representation and the sampling procedure. The paper should more clearly establish how much each component contributes through controlled comparisons with comparably tuned CoBit baselines and matched inference cost.

  3. The GSM8K improvement is modest and its robustness is unclear. H-CoBit achieves 27.4% accuracy, compared with 25.4% for the strongest reported CoBit configuration, which uses twice the NFE, and 23.4% at matched NFE. The efficiency improvement is encouraging, but the two-percentage-point gain over the strongest baseline provides limited evidence of a robust reasoning advantage. Repeated sampling, uncertainty estimates, and a clear validation protocol for selecting the reported “best churn” are needed.

  4. The method’s intuition is insufficiently developed and substantiated. The high-level idea is that coarse cluster information should help recover exact tokens, but the paper does not convincingly explain why these particular embedding-derived clusters improve denoising. It remains unclear whether the gains arise from semantic organization, auxiliary supervision, or other properties of the augmented representation. The random-cluster ablation is useful, but it does not isolate semantics because it does not preserve cluster sizes and corpus-frequency statistics. A clearer mechanistic explanation and stronger controls would make the contribution more convincing.

  5. Several reporting inconsistencies and ambiguities should be resolved.

    • Table 10 gives the LM1B cluster loss weight as 0.23, whereas the main results specify (4/15).
    • Appendix C.3 describes coupled inverse-CDF sampling, while Appendix D specifies log-normal training noise. These descriptions can be compatible, but the exact distributions and coupling used in the implementation should be stated consistently.
    • Table 12 reports an LM1B shared-churn result of 59.2, compared with 57.3 in Table 1. The configuration differences responsible for these values should be explained.
    • Table 9 contains unresolved references to “Section ??”.
    • Figure 1’s claim of a 2.6× GSM8K compute saving needs its exact operating point specified. The 1024-to-512-NFE comparison in Tables 2 and 13 implies approximately 1.94× after accounting for per-step overhead.
Questions for authors

Addressed in weaknesses

6 comments
AuthorsOpenReview ↗

Author response to Atjr 1/3

Dear reviewer,

Thank you for the detailed and constructive review. We ran new experiments for each point and fixed every reporting issue. All additions are in the updated PDF.

1. Cross model transferability

We applied the framework to FLM [1], a continuous flow model with one-hot token representations, quite different from CoBit's bits. We trained FLM and H-FLM on LM1B for 1M steps (Section 5.4). We show a 8-14% improvement of H-FLM on FLM across NFEs at matched entropy, when considering optimal samplers for both models.

The FLM baseline is strong. With stochastic sampling, our FLM reaches 94.6 at data entropy and NFE 1024, against 119.1 at H=4.28 reported in the original paper [1].

Critically, H-FLM's optimal sampler is here again stochastic sampling with late churn on the clusters, just as for H-CoBit. This suggests that stochastic sampling with late cluster churn could be a framework wide sampling knob.

FLM vs H-FLM at matched (NFE, H), one sampler per model, Gen-PPL.

NFEH=4.2 FLMH-FLMΔH=4.3 FLMH-FLMΔH_data FLMH-FLMΔ
12864.655.3−14.3%89.080.9−9.1%107.398.7−8.0%
51256.148.2−14.1%81.571.7−12.1%98.688.6−10.2%
102454.246.6−14.0%78.269.4−11.2%94.6↓ 4.318--

The same sampler is used in every row, only the temperature moves (FLM: γ=0.1 + s_noise=1.05; H-FLM: γ1γ_1=0.3 + cluster γ2γ_2=0.3 late churn). We report sampler sweep results on FLM/H-FLM in Appendix E.5.2 of the updated PDF.

We also ran the random cluster ablation at 100k steps on H-FLM (table below, Figure 12 of the updated PDF), which yields similar results to its counterpart on H-CoBit:

Random-cluster control, LM1B 100k (k=16), Gen-PPL at matched entropy; (Δ) vs FLM.

NFEH=4.3 FLMk-meansrandomH_data FLMk-meansrandom
128115.898.8 (−15%)130.9 (+13%)143.7116.3 (−19%)168.4 (+17%)
512106.389.1 (−16%)118.3 (+11%)135.0108.0 (−20%)188.9 (+40%)
1024103.484.6 (−18%)128.7 (+24%)131.9↓ 4.323169.9 (+29%)

One sampler per model at every NFE: FLM γ=0.03; both H-FLM arms γ1γ_1=0.3 + cluster γ2γ_2=0.3 late churn. ↓/↑: the budget's max/min entropy when it misses the target.

Scale. We agree that larger models are a key open question. Our compute budget limits us to about 135M parameters, as in most recent continuous DLM work. The transfer to FLM shows the gain is not specific to one model; we state the scale limitation explicitly in Section F.1.

2. Sampler tuning

(same as our point 1. with reviewer 7JTd)

We provide below a table of the sampler sweep run on CoBit on OpenWebText for 43 configurations (the full sampler list is in Appendix E.5, Tables 13/14 in the updated PDF) - against only 9 configurations for H-CoBit. It shows that s_noise=1.05 is among the optimal samplers. This is why we included it in the first place, but we did not mention that it was optimal, thank you for pointing this out. Critically, late window churn, whether on all or a subset of bits, does not help improve CoBit as much as H-CoBit.

CoBit-S with its best sampler vs H-CoBit + PMS, OWT 750k, Gen-PPL at matched (NFE, entropy H).

NFEHCoBit-S, best of 43 samplersCoBit-S, s_noise=1.05 (vs best)H-CoBit + PMS (vs CoBit-S best)
645.60254.7 (a)260.6 (+2.3%)212.4 (-16.6%)
1285.54124.8 (b)130.0 (+4.1%)109.6 (-12.2%)
2565.4671.1 (c)71.1 (+0.0%)50.4 (-29.1%)
5125.30530.9 (d)33.3 (+7.9%)20.9 (-32.4%)

Best CoBit-S sampler per row (over 43 configurations, the full list and results are in Appendix E.5): (a) global s_noise=1.04; (b) 4 leading bits: late γ₂=0.3 + global s_noise=1.05; (c) global s_noise=1.05; (d) 4 trailing bits: late γ₂=0.3 + global s_noise=1.05. CoBit-S has no cluster bits, so "4 trailing/leading bits" applies H-CoBit's cluster-bit churn to 4 of its own 16 bits. Every value is a measured generation within 0.008 nats of the row's H; for each NFE, H is the entropy closest to H_data at which all three samplers have a measured point. We do not report only results at H_data, since not all models reach H_data at all NFEs.

Therefore:

  • H-CoBit + PMS beats CoBit-S's best sampler at every NFE; at H_data (NFE 256), 50.4 against 71.1 (−29.1%).
  • s_noise = 1.05 is among the optimal samplers for CoBit-S: within +0% to +7.9% of its best across NFEs, no sampler beats it consistently across all NFEs. It reaches CoBit's minimal GenPPL at dataset entropy (71.1) for NFE 256.
  • Late window churn on CoBit's bits does not beat s_noise = 1.05 at dataset entropy (NFE 256): H-CoBit's own recipe applied to 4 of CoBit's bits reaches 71.3–72.6, against 71.1.

(see followup in the next comment)

AuthorsOpenReview ↗

Author response to Atjr 2/3

3. GSM8K robustness:

Our original evaluation followed the protocol of prior work on this benchmark [2, 3], which reports a single run per model. To assess robustness, we re-evaluated both models over 6 sampling seeds. To remove selection effects, both models use the same fixed churn γ = 0.414 (following [3]), the maximum of the range, chosen in advance rather than tuned per model, per NFE or on the test set.

H-CoBit improves on CoBit consistently across NFEs, and H-CoBit at NFE 384 closely matches CoBit at NFE 1024, showing efficiency gains. The accuracy gains are above 2 standard errors at every NFE.

GSM8K accuracy (%) and standard error at maximal churn for CoBit and H-CoBit (so no γ is chosen per model or per NFE) across 6 evaluation seeds.

ModelNFE 256NFE 384NFE 512NFE 1024
CoBit-S22.5 ± 0.423.6 ± 0.424.1 ± 0.425.3 ± 0.3
H-CoBit (ours)23.5 ± 0.225.1 ± 0.326.2 ± 0.226.4 ± 0.3
Δ (± SE)+1.0 ± 0.4+1.6 ± 0.5+2.0 ± 0.5+1.1 ± 0.4

We observe:

  • H-CoBit leads at every NFE, by 2.2 to 4.2 standard errors, with both models at the same fixed churn.
  • It surpasses CoBit's best result at half the NFE. Including per-call overhead, this is 1024 × 1174 / (512 × 1213) ≈ 1.9× less compute.
  • It matches CoBit's best at about 2.6× less compute. At NFE 384, H-CoBit (25.1) is within one standard error of CoBit at NFE 1024, for 2.6× less compute.
  • Relation to the single-run results. H-CoBit's single-run 27.4% at NFE 512 corresponds to 26.18% averaged over seeds. Table 2 now reports the 6-seed results for CoBit and H-CoBit.

4. Method's intuition:

We provide elements on this matter in point 2 of our response to reviewer 7JTd. Regarding uncovering where the gains come from, we provide additional ablations, which show that the gain needs semantic clusters, and it needs them as both input and target:

A. Random cluster ablation: We reproduce this experiment with random clusters matched to the k-means clusters in both cluster size (number of tokens) and corpus-frequency mass (share of corpus tokens), so that only the semantic grouping differs. Results in the table below (and figure 10, Appendix E.4.3 of the updated pdf) show that balanced random clusters do not match H-CoBit's improvement, and stay close to CoBit (balanced random at NFE 128 improves by 9% on CoBit but is at lower entropy, 4.326 vs 4.334).

Beyond the random cluster control, LM1B 100k, per NFE. Each cell is Gen-PPL (entropy); Δ is relative to CoBit in the same row. Data entropy is 4.34.

NFECoBitH-CoBit, k-means (Δ)Balanced random (Δ)
12895.2 (4.334)76.6 (4.343) (−20%)86.2 (4.326) (−9%)
25673.3 (4.33)62.0 (4.33) (−15%)75.2 (4.316) (+3%)
51266.9 (4.32)59.5 (4.32) (−11%)66.4 (4.279) (−1%)

Points: CoBit and H-CoBit are read at the highest entropy both reach. Balanced random is read at its own highest entropy, always lower, which favours it since Gen-PPL rises with entropy, yet it stays above H-CoBit at every NFE. Values differ from 4B because the 2 tables read models at different entropies.

B. Mechanistic explanation: we train H-CoBit by using (i) only its hierarchical input (no cluster decoder) and (ii) only its cluster target. Experiments show that each variant is worse than H-CoBit, and hence necessary (see table below, and appendix E.4.2).

Cluster input only vs cluster output only, LM1B 100k. Each cell is Gen-PPL (entropy); Δ is relative to H-CoBit at the data entropy (4.34) in the same row.

NFEH-CoBitCoBit (Δ)Cluster output only (Δ)Cluster input only (Δ)
12876.6 (4.34)95.2 (4.334) (+24%)101.3 (4.337) (+32%)93.9 (4.34) (+23%)
25664.9 (4.34)75.3 (4.331) (+16%)83.3 (4.34) (+28%)80.5 (4.34) (+24%)
51265.2 (4.34)68.2 (4.323) (+5%)70.6 (4.331) (+8%)74.2 (4.321) (+14%)

Cluster output only: cluster bits are a prediction target only, not an input. Cluster input only: cluster bits are an input, with no cluster loss and H-CoBit's cluster noise schedule kept fixed. One sampler per model at every NFE, as for H-CoBit. When an arm does not reach the data entropy, its cell is its highest-entropy point. This favours the arm, since Gen-PPL rises with entropy, yet every variant stays above H-CoBit at every NFE.

C. The cluster channel carries information the token head does not express. Across the sampling path, the cluster head predicts the true cluster better than the cluster implied by the token head, by 5% at high noise up to 41% near the end (Figure 8; table in our response 2.C to Reviewer 7JTd).

D. Ordering. An explicit ordering sweep shows that clusters leading slightly helps, clusters lagging hurts, and strict coarse-first is much worse (+77–90%). Details are in our response to Reviewer 7JTd, point 2A.

AuthorsOpenReview ↗

Author response to Atjr 3/3

5. Reporting inconsistencies

Thank you for noticing these reporting mistakes. We addressed all of them:

  1. Table 10: we corrected all loss weights to 4/15 (0.27), this was a typo.
  2. Appendix D now specifies log normal training noise during the warmup phase (so as to train the entropy gated schedule). We use the coupled shared quantile approach, based on CoBit's entropy gated schedule.
  3. The 59.2 and 57.3 are two points corresponding to different churn (and hence entropy) levels, that is churn/entropy 0.175/4.344 and 0.225/4.3422 respectively. This was fixed in the updated pdf.
  4. This was a typo, it is now fixed in the updated pdf.
  5. We have updated this in light of the multi seed GSM8K results reported above. H-CoBit closely matches CoBit's best at NFE 384 (2.6 times less compute) and surpasses it at NFE 512 (1.9 times less compute)

References: [1] Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Seunghoon Hong, Nicholas M. Boffi, and Jinwoo Kim. Flow map language models: One-step language modeling via continuous denoising, 2026. [2] Justin Deschenaux and Caglar Gulcehre. Language modeling with hyperspherical flows, Transactions on Machine Learning Research, 2026 [3] Georgios Batzolis, Mark Girolami, and Luca Ambrogioni. Cobit: Language modeling with bitstream diffusion, 2026.

Reviewer_7JTdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes hierarchical continuous diffusion language models (H-CDLMs), which augment a continuous diffusion language model with an auxiliary coarse modality that is diffused jointly with the tokens. In the instantiation studied, the coarse modality is a cluster ID per token, obtained by K-means on pretrained token embeddings (n = 16 clusters), encoded with the base model's own token encoder and concatenated to the token representation at each position. Each modality has its own noise level, its own information-uniform noise schedule, and optionally its own stochastic sampler settings; the cluster channel is discarded at the end of sampling, which is justified because the clusters are a deterministic function of the tokens (Prop. 1). Applied to CoBit, the resulting H-CoBit improves GenPPL at data entropy on LM1B (85.2 to 57.3, at 600k vs. 1M training steps) and OpenWebText (83.2 to 67.8 at matched steps), with further gains from a per-modality sampler (49.4 and 50.4), and improves GSM8K accuracy from 25.4% to 27.4%. Ablations on LM1B show that n = 16 is the best cluster count and that random cluster assignments give no gain.

Critical strengths

The idea is nice and, I think, points in the right direction. Generating text jointly across several levels of granularity is a natural way to give a continuous diffusion model a coarse signal to anchor on at high noise, and the implementation is simple, adds negligible parameters and compute (Table 13), and can in principle be applied to any continuous DLM. The random-cluster control (Fig. 5a) is a good experiment: it shows that the gain comes from meaningful cluster structure rather than from the added channels alone. Together with consistent improvements on LM1B, OpenWebText, and GSM8K, this makes for a convincing case that hierarchical structure helps continuous text diffusion.

Critical weaknesses
  1. It is not clear how much of the headline gain on OpenWebText is due to the hierarchy rather than sampler tuning. The largest improvements come from the per-modality sampler, and the baseline does not receive a comparable search. In Fig. 4b, CoBit with s_noise = 1.05 is essentially matched with H-CoBit, and Table 1 shows the same thing at data entropy (71.1 vs. 67.8). The per-modality sampler setting that gives 50.4 was selected from a large sweep run on H-CoBit only (Table 12), and its components are strikingly non-additive: s_noise = 1.05 alone makes H-CoBit worse (80.0), late cluster churn alone gives 69.2, and the combination gives 50.4. I'd like to see the baseline given an equivalent sampler search (e.g., a late-window churn on CoBit's own token bits) before attributing these gains to modality separation.
  2. The paper does not explain why diffusing all levels of the hierarchy in parallel should help. The intuitive reason to introduce a coarse modality is to resolve it first and then generate tokens conditioned on it, as in HDLM or latent forcing. Here, the default coupled per-modality schedule deliberately puts both modalities at the same information quantile and so removes any ordering (App. C.2), training noise is coupled through a single shared quantile (App. C.3), and the denoiser is conditioned only on the token noise level (App. D). The shared schedule, which yields an implicit coarse-first ordering, is not clearly better or worse (Table 3), and no explicit coarse-to-fine variant is tested. This leaves open whether the benefit comes from the hierarchy as such or from something closer to an auxiliary prediction target.
Questions for authors
  1. (Experiments) Can the authors run CoBit with a sampler search comparable to the one in Table 12, for example applying the late-window churn to a subset of CoBit's bits? If H-CoBit + PMS still clearly outperforms, this would substantially strengthen the main OpenWebText claim.
  2. (Clarification / experiments) Why diffuse all modalities in parallel rather than coarse first, then tokens conditioned on the clusters? Have the authors tried training with independent per-modality noise levels and sampling with an explicit coarse-to-fine schedule? Understanding why parallel diffusion helps at all would clarify the mechanism.
  3. (Future work) How should the coarse structure be chosen in general? K-means on a fixed set of pretrained embeddings, with n = 16 and silhouette scores near 0.05 (Table 5), seems unlikely to scale to larger vocabularies, larger models, or deeper hierarchies. Do the authors see a path toward learned or more principled coarse representations?
3 comments
AuthorsOpenReview ↗

Author response to 7JTd 1/2

Dear reviewer,

Thank you for your time and appreciation of our work. Below we address each of your questions.

We have also updated the pdf, and will point to its relevant additions throughout our answers.

1. Sampler tuning of CoBit:

We provide below a table of the sampler sweep run on CoBit on OpenWebText for 43 configurations (the full sampler list is in Appendix E.5, Tables 13/14 in the updated PDF) - against only 9 configurations for H-CoBit. It shows that s_noise=1.05 is among the optimal samplers. This is why we included it in the first place, but we did not mention that it was optimal, thank you for pointing this out. Critically, late window churn, whether on all or a subset of bits, does not help improve CoBit as much as H-CoBit.

CoBit-S with its best sampler vs H-CoBit + PMS, OWT 750k, Gen-PPL at matched NFE, entropy H.

NFEHCoBit-S, best of 43 samplersCoBit-S, s_noise=1.05 (vs best)H-CoBit + PMS (vs CoBit-S best)
645.60254.7 (a)260.6 (+2.3%)212.4 (-16.6%)
1285.54124.8 (b)130.0 (+4.1%)109.6 (-12.2%)
2565.4671.1 (c)71.1 (+0.0%)50.4 (-29.1%)
5125.30530.9 (d)33.3 (+7.9%)20.9 (-32.4%)

Best CoBit-S sampler per row (over 43 configurations, the full list and results are in Appendix E.5): (a) global s_noise=1.04; (b) 4 leading bits: late γ₂=0.3 + global s_noise=1.05; (c) global s_noise=1.05; (d) 4 trailing bits: late γ₂=0.3 + global s_noise=1.05. CoBit-S has no cluster bits, so "4 trailing/leading bits" applies H-CoBit's cluster-bit churn to 4 of its own 16 bits. Every value is a measured generation within 0.008 nats of the row's H; for each NFE, H is the entropy closest to H_data at which all three samplers have a measured point. We do not report only results at H_data, since not all models reach H_data at all NFEs.

Therefore:

  • H-CoBit + PMS beats CoBit-S's best sampler at every NFE; at H_data (NFE 256), 50.4 against 71.1 (−29.1%).
  • s_noise = 1.05 is among the optimal samplers for CoBit-S: within +0% to +7.9% of its best across NFEs, no sampler beats it consistently across all NFEs. It reaches CoBit's minimal GenPPL at dataset entropy (71.1) for NFE 256.
  • Late window churn on CoBit's bits does not beat s_noise = 1.05 at dataset entropy (NFE 256): H-CoBit's own recipe applied to 4 of CoBit's bits reaches 71.3–72.6, against 71.1.

2.Why diffuse the levels in parallel rather than coarse-first?:

We emphasize parallel sampling in the paper, as in ReDi for image diffusion [1], which is a prior work to latent forcing's ordered schedules [2]. Our motivation behind parallel rather than ordered sampling is that coarse-first diffusion risks letting tokens inherit errors (which is a known weakness of HDLM [3]). In parallel diffusion, the two levels can correct each other without an arbitrary ordering enforced. We provide below (and in appendix E.4) a more in depth investigation:

A. Explicit ordering sweep. Following your suggestion, we trained H-CoBit with a mix of parallel and independent noise schedules (see Appendix E.4.4 for details) on LM1B for 100k steps. We then sampled with clusters leading or lagging tokens by a warp of strength s which moves the cluster's noise quantile to u′=u/(s−(s−1)u)u′ = u/(s − (s − 1)u) , and with strict coarse-first schedules :

LM1B, 100k steps, GenPPL at data entropy (percentages in the "parallel" row are relative to H-CoBit with coupled training (first row); all other percentages are relative to the "parallel" row):

ScheduleNFE 128NFE 256NFE 512
H-CoBit (coupled training)76.664.965.2
parallel79.6 (+4%)64.1 (-1%)61.8 (-5%)
clusters lead, s = 373.8 (-7%)60.9 (-5%)59.2 (-4%)
clusters lead, s = 1072.1 (-9%)58.4 (-9%)61.4 (-1%)
clusters lag, s = 384.3 (+6%)67.9 (+6%)66.7 (+8%)
clusters lag, s = 1089.9 (+13%)73.1 (+14%)↓4.332
coarse first, 25% of steps↑4.359↑4.356↑4.349
coarse first, 50% of steps141.1 (+77%)122.0 (+90%)↓4.338
parallel + late cluster churn74.1 (-7%)62.1 (-3%)57.3 (-7%)
lead s = 3 + late cluster churn73.9 (-7%)62.4 (-3%)56.0 (-9%)
lead s = 10 + late cluster churn75.3 (-5%)↓4.337↓4.330

↓/↑: max/min entropy when it misses the target.

We observe that:

  • A soft cluster lead helps (−4% to −9%).
  • A cluster lag hurts, more as the lag grows (+6% at s = 3, up to +14% at s = 10).
  • Strict coarse-first is far worse (+77% to +90%), or cannot reach data entropy at all.

The best schedules are therefore joint, with clusters slightly ahead, not sequential. Late cluster churn gives a similar gain to a soft lead, and the two combine slightly (56.0 at NFE 512). One reading is that late churn keeps clusters revisable, consistent with the levels correcting each other.

We agree that exploring ordered schedules is an interesting way forward, and leave it to future work.

(see the rest in the next comment)

AuthorsOpenReview ↗

Author response to 7JTd 2/2

B. Auxiliary target or hierarchy? We trained input-only and target-only variants (we call it over encoding/decoding in Appendix E.4.2). Both are worse than full H-CoBit at every NFE, so the gain is not explained by auxiliary supervision alone (full table in our response 4B to Reviewer AtJr).

C. The cluster channel carries a sharper coarse estimate than the token head. We compare the cluster head's prediction with the cluster implied by the token head, on the same noisy input (token-cluster head agreement is shown in Figure 8, Appendix E.4.1).

LM1B, H-CoBit n = 16, cross-entropy of the true cluster in nats per token (lower is better), 2,048 test rows:

Path position u (1 = noise)σ_infoToken-implied clusterCluster headGain (nats)Gain (%)
0.8751.812.5602.4350.1255%
0.751.242.4132.1940.2199%
0.6250.982.1811.8830.29714%
0.50.811.8541.5120.34218%
0.4375 (peak)0.741.6561.3030.35321%
0.3750.681.4341.0880.34624%
0.250.570.9550.6540.30132%
0.1250.450.4510.2660.18541%

The token-implied cluster sums the token head's distribution (the product of its per-bit marginals over valid token ids, which is what the sampler reads) over each cluster's tokens. Part of the gap may reflect the limits of this factorized token head. Either way, the cluster channel gives the sampler a coarse estimate the token head alone does not express.

3.Future work:

A clear follow-up of this work is exploring more complex high level modalities. We emphasize n=16 token clusters because this very simple recipe yields consistent experimental results. Multi-token or sentence level descriptors could for instance be interesting high level modalities. Regarding scalability of clustering, this should not be too much of an issue here: clustering embeddings for a 50 000 large vocabulary takes a few minutes on a CPU.

References: [1] Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. Boosting generative image modeling via joint image-feature synthesis. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

[2] Alan Baade, Eric Ryan Chan, Kyle Sargent, Changan Chen, Justin Johnson, Ehsan Adeli, and Li Fei-Fei. Latent forcing: Reordering the diffusion trajectory for pixel-space image generation. In Forty-third International Conference on Machine Learning, 2026

[3] Cai Zhou, Chenyu Wang, Dinghuai Zhang, Shangyuan Tong, Yifei Wang, Stephen Bates, and Tommi Jaakkola. Next semantic scale prediction via hierarchical diffusion language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.

AuthorsOpenReview ↗

End of rebuttal questions

Dear Reviewer,

Thank you again for your review.

Your feedback on Table 1 has strengthened our paper. As the discussion period ends today, we wanted to check whether our rebuttal has addressed your concerns.

If any questions remain, we would be glad to clarify them in the time left.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

We thank the reviewers for their time spent on our work. Their questions led to several new experiments, all included in the updated PDF:

  • A second host model. H-FLM, our framework applied to FLM, improves GenPPL by 8–14% over FLM on LM1B at matched NFE and entropy. A random-cluster control again gives no gain (Section 5.4, Appendix E.5.2/E.6).
  • An equal-or-larger sampler search for CoBit. 43 configurations, including H-CoBit's late-window churn applied to subsets of CoBit's own bits. The best CoBit sampler closely matches snoise=1.05s_{noise}=1.05 and stays well behind H-CoBit with its per-modality sampler (Appendix E.5), ensuring fair comparison of CoBit and H-CoBit+PMS.
  • An explicit ordering sweep. Clusters leading tokens slightly improves GenPPL, clusters lagging hurts, and strict coarse-first generation is much worse (+77–90%). This explains why we diffuse levels jointly (Appendix E.4.4).
  • Mechanism ablations. Random clusters matched to the k-means clusters in size and frequency, input-only and target-only variants, and a comparison of the cluster head with the token head's implied cluster (Appendix E.4).
  • Robust GSM8K evaluation. We reevaluate CoBit and H-CoBit on 6 seeds, one fixed churn for both models (no per-model selection). H-CoBit's gain is 2.2–4.2 standard errors from zero at every NFE, and it surpasses CoBit's best result at half the NFE (Table 2).
  • Fairer comparisons. LangFlow re-evaluated at higher NFE and matched entropy, and FMLM removed due to unfair comparison (Table 1)
  • All reporting inconsistencies fixed (see point 5 of our answer to reviewer AtJr).
  • Teaser Figure changed to take into account the fair optimal sampler comparison (right panel for OpenWebtext) and the multi seed GSM8k results (both panels).

Modifications in the updated PDF's main text are in violet to ease reviewing - it will be back to normal formatting for the camera ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.