acceptodds
Under review as a conference paper at ICLR 2027

Denoising Blocks, Not Tokens: Efficient Compressed Continuous Diffusion with Branching Token Realization

Abstract

Diffusion language models (DLMs) generate text through iterative parallel refinement, offering the potential for higher throughput than autoregressive (AR) decoding. However, most DLMs still maintain one generative state per token, so every denoising step processes a state sequence as long as the output sequence, limiting the throughput gains from parallel generation. Continuous DLMs provide an additional degree of freedom: a single continuous state can represent multiple tokens, allowing diffusion to operate on a much shorter latent sequence. We introduce Branching Latent Diffusion (BLD), which exploits this flexibility by compressing a 1024-token sequence into only 64 block latents, a reduction. BLD combines latent compression with branching token realization, where each latent is decoded by a local AR branch and all branches run in parallel. Because strong compression makes joint latent generation difficult, BLD generates the latents in groups, conditioning each group on previously generated latents. In end-to-end evaluation on the same GPU, BLD reduces generation FLOPs by more than and increases throughput by more than relative to the similarly sized ELF-L baseline. Compared with the AR baseline, BLD achieves more than higher throughput and more than lower latency. Despite the compression, BLD maintains competitive local fluency and diversity, although long-range coherence remains challenging. Overall, BLD shows that moving diffusion from token-level states to compressed latent sequences can substantially improve the efficiency of long-sequence generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continuous diffusion languge models, Branching token realization, Efficient language generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_v27AOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Branching Latent Diffusion, a framework for continuous diffusion language model that compresses block-level latents instead of token-level latents. BLD uses 16x token-to-latent compression ratio and leverages a branching token realization decoder. BLD further introduces a grouped latent flow that divides the K latents into groups with size G. The groups are generated autoregressively while the latents inside each group is denoised by flow model.

Experiments on unconditional text generation show that BLD reduces generation compute and much higher throughput compared with the baselines. Ablation study also investigates the effect of the group size, the design of the decoder, and the effect between reconstruction and diffusion generation.

Critical strengths
  1. The paper is well motivated to compress the representation of the long sequence into groups to achieve speedup for the diffusion. The experiment also verifies the efficacy of such a design in terms of remarkably reducing the inference latency.

  2. The ablation study is comprehensive and the in-depth analyses, e.g., the PCA experiments and interpolation probes in appendix, offer much more valuable insights.

Critical weaknesses
  1. The evaluation scope is limited. The current evaluation is performed on unconditional generation using OpenWebText. It would make the paper stronger if experiments on conditional tasks, e.g. summarization or translation, could be evaluated, like in ELF. This might be critical since each branch decodes tokens independently without observing the tokens realized in adjacent branches, so the behavior on unconditional task may be different from conditional tasks.

  2. It is not clearly discussed in the paper how the grouped BLD can leverage KV cache across groups, which would be important given that previous works like Block Diffusion have already implemented KV cache, which is quite standard to control the inference complexity when the model operate on long sequences.

Questions for authors
  1. Could the authors provide more justifications on conditional tasks, or why they are not evaluated?

  2. Could the authors make more discussions about how KV cache could be integrated into the framework, which is quite important when scaling the model to long sequence generation?

  3. Why was G=64 (G=K) excluded from Table 3? How does it perform compared with the reference appeared in Table 1?

I am willing to raise the score if the concerns are resolved sufficiently.

6 comments

Thank you for the positive assessment of our motivation, architecture, and ablation analyses. We have added conditional generation experiments, clarified KV caching across groups, and extended the controlled group-size comparison.

W1 / Q1: Conditional generation. Our primary focus has been on unconditional generation, as we aim to develop a general framework for language generation under strong latent compression. Unconditional generation involves greater uncertainty over content and global structure, whereas conditional tasks such as summarization provide source information that reduces this uncertainty but introduces the challenge of satisfying conditioning constraints.

To evaluate BLD in this setting, we added XSum summarization experiments using ELF's setup and evaluation script on the full test set of 11,334 articles (General Response R4; Appendix D.4). We evaluate both fine-tuning from OpenWebText and training from scratch, reporting ROUGE-1/2/L scores (higher is better), as shown below.

ModelROUGE-1ROUGE-2ROUGE-L
BLD, fine-tuned from OpenWebText28.19.723.6
BLD, trained on XSum from scratch33.911.727.8
ELF-B (ELF, Table 1)36.012.227.8
MDLM (ELF, Table 1)33.411.625.8
AR (ELF, Table 1)30.510.224.4

BLD trained on XSum achieves ROUGE-1/2/L of 33.9/11.7/27.8 with approximately 105M parameters used at generation time, comparable in size to ELF-B (105M), while matching its ROUGE-L and outperforming the reported MDLM and AR baselines across all three metrics. These results demonstrate that BLD can support conditional generation despite decoding token branches independently, as the branches share the complete latent context.

W2 / Q2: KV caching across groups. KV caching can be applied to BLD in a similar way to Block Diffusion. Since generation is causal across groups and previously generated latents remain fixed, their keys and values can be cached and reused by subsequent groups. We have implemented this optimization and evaluated its effect on computation and runtime.

  • Correctness and computation. With and without caching, the conditioning states agree within 6×10−66\times10^{-6} (compared with a standard deviation of 0.9), and generation quality remains unchanged. Caching reduces computation from 1.10 to 0.95 TFLOPs at G=8G=8 (14%) and from 1.32 to 0.97 TFLOPs at G=4G=4 (27%).
  • Single-document latency. Caching has little effect at batch size 1 (0.475 vs. 0.476 s for G=8G=8), as runtime is dominated by the 400 denoising steps of the small flow head, which reuses the backbone conditioning states.
  • Batched throughput. At G=8G=8, caching improves throughput by 8%, 10%, and 11% at batch sizes 128, 256, and 512, respectively, reaching 112.2 documents/s at batch size 512 (from 101.4). At G=4G=4, the corresponding improvements are 14%, 19%, and 22%.

The benefit increases with the number of groups, as caching avoids repeatedly processing previously generated latents, suggesting that it may be particularly useful for longer sequences. We have added the implementation details and measurements to Appendix C.5.

Q3: Fully parallel generation (G=64G=64). Due to limited time and computational resources, we had not completed the controlled G=64G=64 experiment at the time of the original submission. We have now extended Table 4 (original Table 3) to include G=64G=64, trained with the same architecture, encoder, decoder, training budget, and sampler as G=8G=8 (General Response R3).

The complete controlled group-size comparison is shown below (Table 4 in the revised manuscript).

ModelGen-PPLEntropyMAUVELatency (s)TFLOPs
G=131.5 ± 0.45.22 ± 0.010.89 ± 0.024.932.23
G=424.4 ± 0.25.22 ± 0.020.86 ± 0.013.611.32
G=8 (main model)26.5 ± 0.35.16 ± 0.010.82 ± 0.011.821.10
G=1627.3 ± 0.25.07 ± 0.040.79 ± 0.011.211.00
G=3223.9 ± 0.15.01 ± 0.010.75 ± 0.020.530.95
G=64 (fully parallel)25.1 ± 0.15.06 ± 0.020.75 ± 0.010.300.93
Real text15.35.320.95——

The original G=KG=K reference in Table 1 was separately developed and tuned to provide a stronger fully parallel baseline. It achieves Gen-PPL 24.6, entropy 5.08, and MAUVE 0.77 at 4.8 TFLOPs. The new controlled G=64G=64 model achieves slightly lower generation quality (25.1/5.06/0.75), but requires only 0.93 TFLOPs. Compared with G=8G=8, it is approximately 6× faster (0.30 vs. 1.82 s), but has lower MAUVE (0.75 vs. 0.82). These results clarify the trade-off between fully parallel and grouped latent generation.

Dear Reviewer,

With the discussion period coming to a close, we hope that our responses and new experimental results have helped address your concerns. We would be glad to discuss any remaining questions if time permits.

Dear Reviewer v27A,

Apologies for the additional follow-up. As the discussion period is coming to a close, we wanted to kindly ask whether our new experiments on conditional generation, KV caching, and fully parallel generation have sufficiently addressed your concerns.

We would be happy to clarify any remaining questions. If you feel that your concerns have been resolved, we would sincerely appreciate your reconsideration of our work in your final evaluation.

Thank you again for your time and consideration.

Reviewer_39rsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the generation cost of diffusion language models, where every denoising step processes a state sequence as long as the output, and proposes Branching Latent Diffusion (BLD), which performs diffusion over a compressed sequence of block latents. An encoder compresses 1024 tokens into 64 block latents through a bidirectional Transformer, dynamic chunking, and attention pooling. A prior generates the latents in groups of eight, autoregressively across groups and jointly within each group with rectified flow. After all latents are generated, 64 local autoregressive branches decode them in parallel into token spans of about 16 tokens. The authors train the model on OpenWebText, compare it with ELF, MDLM, BD3-LM, and an AR baseline on unconditional generation in terms of Gen-PPL, entropy, MAUVE, FLOPs, and end-to-end runtime, and ablate the group size and decoder adaptation.

Critical strengths
  1. The design is coherent. Compression shortens the denoising sequence, branching decoding prevents the sequential bottleneck from shifting to token realization, and grouped generation addresses the difficulty of jointly generating strongly compressed latents, so each component corresponds to a clear problem.

  2. The paper reports end-to-end wall-clock measurements on the same GPU and documents the precision, attention kernels, framework versions, and timing protocol in the appendix, which makes the runtime results easy to check.

  3. The appendix diagnostics are informative. In Table 5, with two tokens per latent, clean reconstruction stays at 0.999, while midpoint recovery drops from 0.925 at one token per latent to 0.791, showing that reconstructability and generatability can diverge. The authors also explicitly acknowledge the weak long-range coherence and the uncontrolled comparisons.

Critical weaknesses
  1. The experimental results do not support the efficiency claim. The more than 80×80\times FLOPs reduction and more than 6×6\times throughput gain in the abstract are measured against ELF-L, whose quality is not comparable (MAUVE 0.82 vs. 0.91). Against ELF-B, whose quality is comparable (Gen-PPL 23.8 vs. 26.6, MAUVE 0.80 vs. 0.82), BLD uses about 6.5×6.5\times fewer FLOPs but is worse in both latency (2.67 s vs. 0.46 s) and throughput (9.4 vs. 18.2 doc/s). The FLOPs reduction does not translate into a measured speed advantage.

  2. The runtime measurements do not reflect the computational cost of the algorithm. Using the specifications of the NVIDIA H100 80GB used in the paper (about 3.35 TB/s memory bandwidth and 67 TFLOPS fp32 peak), the difference between the 50-step and 32-step latencies implies about 5.8 ms per flow-head denoising step for BLD, roughly 40 times the time needed to read its 117M weights. AR takes about 12 ms per token, roughly 14 times the time needed to read its weights once. Both also become slower with bf16 (Table 10). This indicates that the measured time is dominated by call overhead rather than computation. In addition, BLD and ELF use different precisions (fp32 vs. bf16), and throughput is measured only in eager mode at batch size 32. Under these conditions, neither the 6.3×6.3\times speedup over AR nor the disadvantage relative to ELF-B reflects the efficiency of the algorithm itself.

  3. The benefit of grouped generation over fully parallel generation lacks a controlled comparison. Table 3 varies GG within the same architecture, but only up to G=16G = 16; the trend is non-monotonic (G=1G = 1 is the worst), MAUVE is not reported, and no variance across seeds is given. At the fully parallel end, G=KG = K is replaced by a separately developed 684M bidirectional denoiser whose training budget, training recipe, and sampler all differ from those of the grouped prior, and the authors acknowledge in Appendix C.1 that this comparison does not isolate the effect of grouping. Under these differences, G=KG = K is faster in both latency (1.16 s vs. 2.67 s) and throughput (13.5 vs. 9.4 doc/s), while quality is mixed. It is therefore unclear whether introducing sequential groups is worthwhile in a design aimed at efficiency.

  4. The paper lacks the compression-ratio and sequence-length experiments needed to support its core claim. The paper argues that stronger compression makes generation harder but long-sequence generation more efficient. However, all main experiments are fixed at L=1024L = 1024, r=16r = 16, and a single model scale, and no sequence longer than 1024 tokens is generated. The compression sweep in Table 5 is conducted in a separate simplified setting and reports results only up to four tokens per latent. It is therefore unclear whether 16×16\times compression is a suitable trade-off, and the claim about long-sequence efficiency lacks support.

Questions for authors
  1. Can the authors provide profiling results for BLD, AR, and ELF that break down the runtime into computation, memory access, and call overhead?

  2. Under matched precision, larger batch sizes, and reduced call overhead (e.g., torch.compile or CUDA graphs), can the authors report quality–throughput Pareto curves? At matched MAUVE, is BLD still faster than ELF and AR?

  3. Can the authors compare G=8G = 8 and G=KG = K using the same Transformer with only the attention mask changed (causal across groups, bidirectional within groups) under the same training budget, or at least report results for G=32G = 32?

  4. How do quality and efficiency change for compression ratios rr of 4, 8, and 32, and does the method extend to generation beyond 1024 tokens?

4 comments

Thank you for the detailed review and for the careful analysis of our runtime measurements and experimental comparisons. We have added optimized runtime benchmarks, detailed profiling, and controlled group-size experiments to address these concerns.

W1 / Q2: Architectural efficiency and quality–throughput trade-off. We would like to emphasize that BLD's efficiency advantage comes primarily from its architecture, rather than implementation optimizations. Unlike token-level DLMs that process 1024 states at each denoising step, BLD operates on only 64 compressed latent states (16× compression). Its branching decoder further avoids shifting the sequential bottleneck to token realization, while grouped latent generation makes strong compression practical. This design substantially reduces generation computation, independently of the inference implementation.

To verify that these computational savings translate into actual runtime improvements, we re-measured BLD, ELF, and AR on the same H200 using bf16 weights, CUDA graphs, and a unified optimized inference setup (see General Response R2). Compared with ELF-B (32 steps), BLD requires 6.6× fewer FLOPs (1.10 vs. 7.22 TFLOPs) and achieves 2.3× higher peak throughput (100.7 vs. 43.9 documents/s). At batch size 32, their throughput is comparable (45.3 vs. 42.4 documents/s), while ELF-B remains 7.3× faster for single-document generation.

Importantly, at similar MAUVE (0.83 vs. 0.82), BLD achieves 4.4× higher peak throughput than ELF-B (64 steps) (100.7 vs. 22.9 documents/s). BLD also achieves higher MAUVE than ELF-B (32 steps) (0.83 vs. 0.78), although ELF-B has lower Gen-PPL. Compared with AR, BLD achieves 11× higher peak throughput, while AR retains higher MAUVE (0.87 vs. 0.83). We therefore do not claim a strictly matched-quality speedup against AR or the larger ELF models. We have added quality–throughput Pareto plots for both MAUVE and Gen-PPL at batch size 1 and peak throughput (see Appendix C.3).

These results demonstrate that BLD's structural reduction in computation translates into substantial end-to-end throughput improvements, particularly at larger batch sizes, while maintaining competitive generation quality.

W2 / Q1: Runtime profiling and execution overhead. We added detailed profiling of BLD, AR, and ELF under both eager and optimized inference (see General Response R2; Appendix C.4). The results confirm that BLD's single-document latency is substantially affected by execution overhead. GPU idle periods account for 73% of its eager fp32 latency and 59% under the optimized stack. Even after optimization, BLD launches approximately 57k short kernels per document.

We further examine computation and memory access. Matrix multiplication and attention account for at least 0.13 s of BLD's 0.47 s optimized latency. Dedicated data-movement kernels account for about 1% of eager latency, although memory access within compute kernels is not separately measured. We also report estimated cumulative weight traffic of 221 GB for BLD in fp32, compared with 2.99 TB for AR. These measurements explain why BLD's FLOP reduction does not translate proportionally into single-document latency. Together with the batch-scaling results in R2, they show how BLD's lower computation per document translates into substantially higher throughput at larger batches.

W3 / Q3: Controlled comparison of grouped and fully parallel generation. We extended Table 4 (original Table 3) to include G=32G=32 and G=64G=64, using the same grouped-prior architecture, frozen encoder and decoder, training recipe, budget, sampler, and training seed as G=8G=8 (R3). Only the group size changes, unlike the separately tuned fully parallel reference in Table 1.

The controlled comparison shows that G=64G=64 is approximately 6× faster than G=8G=8 (0.30 vs. 1.82 s), with slightly fewer FLOPs (0.93 vs. 1.10 TFLOPs), but achieves lower MAUVE (0.75 vs. 0.82). Across all six group sizes, increasing GG consistently reduces latency, while smaller groups generally achieve higher MAUVE. We also report the mean and standard deviation over five evaluations for each group size. These results clarify the benefit of grouping: it improves distributional quality by trading some latent-level parallelism for more clean conditioning context. We select G=8G=8 as a balance between generation quality and efficiency, rather than claiming it is optimal on every metric.

W4 / Q4: Compression ratio and sequence length. We trained the complete BLD pipeline at compression ratios of r=4,8,16,32r=4,8,16,32, including a new autoencoder, branching decoder, and grouped latent prior for each ratio. All models use the same prior size (511M), group size (G=8G=8), and training budget, with one training seed per setting. We evaluate generation quality and optimized runtime using the settings described in R2.

Compression ratio (latents)Reconstruction NLL ↓Gen-PPL ↓Entropy ↑MAUVE ↑Latency (s) ↓Peak docs/s (batch) ↑TFLOPs ↓
4× (256)0.01423.15.240.881.9118.4 (128)6.55
8× (128)0.02025.25.230.830.9052.1 (256)2.50
16× (64)0.03026.65.170.820.47100.7 (512)1.10
32× (32)0.3926.44.940.3070.36125.6 (512)0.57

The r=16r=16 row uses the original single evaluation of the main model, while R2 reports the mean over five evaluations of the same model.

Compression and generation efficiency. Increasing compression substantially reduces generation cost. From 4× to 16× compression, FLOPs decrease from 6.55 to 1.10 TFLOPs, while peak throughput increases from 18.4 to 100.7 documents/s. In particular, moving from 8× to 16× nearly doubles peak throughput (52.1 to 100.7 documents/s) while maintaining similar MAUVE (0.83 vs. 0.82). This demonstrates that stronger compression can improve practical efficiency without substantially degrading generation quality.

Quality–efficiency trade-off. Compression beyond 16× becomes considerably more difficult. At 32×, reconstruction NLL increases from 0.030 to 0.39, while MAUVE drops from 0.82 to 0.307 and entropy decreases to 4.94. Despite the lower FLOPs, peak throughput improves by only 25% relative to 16×. These results support our choice of 16× compression as a balance between generation quality and efficiency, rather than claiming it is universally optimal.

Longer-sequence generation. The BLD architecture can in principle extend beyond 1024 tokens, as the grouped prior generates latents autoregressively across groups and uses rotary embeddings rather than a fixed learned positional embedding table. However, all models are trained only on 1024-token sequences, and the main r=16r=16 model has seen only 64 block positions during training, so generation quality at longer lengths cannot be guaranteed. Evaluating longer sequences would require testing generation beyond the training length, potentially with additional training on longer documents. Given our limited computational resources, we leave systematic evaluation beyond 1024 tokens for future work.

Dear Reviewer,

With the discussion period coming to a close, we hope that our responses and new experimental results have helped address your concerns. We would be glad to discuss any remaining questions if time permits.

Dear Reviewer 39rs,

Apologies for following up again. As the discussion period is nearing its end, we hope our new experiments, particularly the optimized runtime comparisons, detailed profiling, and controlled compression and group-size analyses, have helped address your main concerns.

We would welcome any remaining questions or feedback. If you find that these results have sufficiently addressed your concerns, we would greatly appreciate your consideration of them in your final evaluation.

Thank you again for your time and consideration.

Reviewer_QXhsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BLD compresses 1024 tokens into 64 block latents (bidirectional encoder, dynamic chunking, attention pooling). It generates the latents in groups of G=8 with rectified flow, each group conditioned on the clean previous groups. Each latent is then decoded by a short local AR branch, and all 64 branches run in parallel. On OpenWebText, BLD uses 1.1 TFLOPs vs 96.4 for ELF-L (88× fewer) and gets 9.4 vs 1.4 doc/s throughput (6.7×). Against AR, throughput is 6.3× higher and latency 4.7× lower. Quality is Gen-PPL 26.6, entropy 5.17, MAUVE 0.82. The appendix analyses why strongly compressed latents are hard to generate.

Critical strengths
  • A clear design that targets both bottlenecks. Compressing to 64 states cuts denoiser cost, and branch decoding avoids moving the sequential cost into token generation. The measured gain is large: 88× fewer FLOPs than ELF-L and about 24× fewer than Cosmos.
  • A simple grouped prior with a useful, honest analysis. Grouping trains in parallel with teacher forcing. G cleanly trades quality against latency (Table 3). Target-specific conditioning cuts the hard-loop rate from 14.5% to 4.3%. The appendix findings (reconstruction ≠ generatability, PCA and interpolation probes) are informative beyond this method, and the limitations are stated openly.
  • Large batch-throughput gain over the AR baseline at similar quality. Gen-PPL matches AR at 26.6 vs 26.5, and throughput is 6.3× higher in Table 2.
Critical weaknesses
  • The efficiency claim depends on the choice of baseline. The 6× headline is against ELF-L. The smaller ELF-B has equal or better quality on Gen-PPL and entropy (Gen-PPL 23.8 vs 26.6, entropy 5.15 vs 5.17) and a similar MAUVE (0.80 vs 0.82). It also has higher throughput (18.2 vs 9.4 doc/s) and much lower latency (0.46 vs 2.67 s). So BLD does not beat the best small token-level model. The runtime comparison is also uneven: BLD and AR run in fp32, ELF runs in bf16 with a different PyTorch version, and the AR model uses random weights in eager mode. The FLOP gain over AR is small (1.1 vs 1.6).
  • Quality and coherence are weak and under-measured. MAUVE (0.82) is below ELF-M, ELF-L and AR (0.91–0.93), and entropy is below AR. The D.2 samples are incoherent, and the authors admit long-range coherence is unsolved. The evaluation is unconditional only, with no coherence metric, no human evaluation, and one scale, length and dataset. Gen-PPL alone is a weak quality signal.
  • The ablations are confounded and have no variance. The G=K reference uses a different recipe (about 2B vs 10.5B tokens, per-block noise, rollout training), so Table 1 does not isolate the effect of grouping. Table 3 uses an H200 and Table 2 an H100. There is a single run per setting with no seeds. G=4 beats the default G=8 on Gen-PPL (24.7 vs 26.6). The appendix still says a token budget is “to be confirmed”, and the Cosmos perplexity mismatch is left unresolved.
Questions for authors
  • Clarification: how does BLD compare to ELF-B under the same precision and software stack? Can you show a quality–throughput Pareto plot including ELF-B?
  • Clarification: is the AR baseline measured with trained weights, batched KV-cache decoding, and bf16 or compiled execution? Does the 6× throughput gain hold then?
  • Can you report seeds or variance for Table 1 and Table 3, and a G=K model trained with the same recipe and budget as G=8?
  • Can you add a long-range coherence measure (e.g., topic or entity consistency across 1024 tokens) or a human evaluation?
  • Suggestion for future work: conditional generation and larger scale.
6 comments

Thank you for the detailed review and for highlighting the important comparisons and experimental concerns.

W1 / Q1: Baseline comparison, matched inference, and Pareto analysis. Please see General Response R2. We re-measured all models on the same H200 using bf16 weights, CUDA graphs, and a unified optimized inference setup. Compared with ELF-B (32 steps), BLD requires 6.6× fewer FLOPs and achieves 2.3× higher peak throughput (100.7 vs. 43.9 documents/s). At batch size 32, their throughput is comparable (45.3 vs. 42.4 documents/s), while ELF-B remains 7.3× faster for single-document generation. In terms of quality, BLD achieves higher MAUVE (0.83 vs. 0.78) with comparable entropy (5.17 vs. 5.16), although ELF-B has lower Gen-PPL (23.2 vs. 26.4). We also added quality–throughput Pareto plots for both MAUVE and Gen-PPL, comparing batch size 1 and each model's peak throughput (Appendix C.3). These results show that BLD's computational advantage translates into higher throughput at larger batch sizes, while making the quality–efficiency trade-offs explicit.

W2 / Q4: Generation quality and long-range coherence.

  • Generation quality. We acknowledge the remaining quality gap to larger baselines. In Table 1, BLD achieves MAUVE 0.82, comparable to ELF-B (0.80) but below ELF-M, ELF-L, and AR (0.91–0.93). Its entropy is also lower than AR (5.17 vs. 5.37), while ELF-B achieves lower Gen-PPL (23.8 vs. 26.6). However, we believe generation quality and efficiency should be considered together, rather than comparing efficiency only against smaller models and quality only against larger ones. Under the optimized inference setting (R2), BLD requires 6.6× fewer FLOPs and achieves 2.3× higher peak throughput than ELF-B while maintaining comparable generation quality. Compared with the similarly sized ELF-L, BLD requires 88× fewer FLOPs and achieves approximately 27× higher peak throughput. The new quality–throughput Pareto plots further illustrate these trade-offs. Importantly, BLD also represents substantial progress over previous compressed latent diffusion models. The closest work, Cosmos, achieves 8× compression for 128-token generation, but even 2× compression on 512-token OpenWebText generation substantially reduces MAUVE (0.702 to 0.263). In contrast, BLD achieves 16× compression over 1024 tokens while maintaining competitive generation quality. Although the evaluation settings differ and the quality scores are not directly comparable, this represents an important advance in making strong latent compression practical for language generation. We believe that achieving this level of compression with substantial efficiency gains is a valuable contribution, while further improving generation quality remains an important direction.
  • Generation quality metrics. We agree that Gen-PPL alone is insufficient to assess generation quality. The paper reports Gen-PPL alongside entropy and MAUVE, which provide complementary measures of diversity and distributional quality. Our expanded group-size experiments (R3) further illustrate their importance: G=32G=32 achieves lower Gen-PPL than G=1G=1 (23.9 vs. 31.5), but also substantially lower entropy (5.01 vs. 5.22) and MAUVE (0.75 vs. 0.89). We therefore consider these metrics together rather than relying on Gen-PPL alone.
  • Long-range coherence evaluation (new Appendix A.7). We added an LLM-as-a-judge evaluation following the six evaluation dimensions in ELF's Appendix D.2. Claude Opus 4.8 evaluates each generated document independently, without knowing which model produced it, on six criteria scored from 1 to 5. The judge reads the complete 1024-token document, and the coherence criterion assesses whether adjacent sentences connect naturally and whether the topic remains consistent throughout the document. Since ELF does not provide its full judge prompt, we evaluate all models, including ELF-B, using the same scoring instructions. We also include real OpenWebText documents as a reference.
ModelFluencyNon-repetitionCleanlinessCoherenceSensibilityOverall
Real text (OpenWebText)4.694.674.684.364.664.29
AR3.893.504.212.151.751.83
ELF-B, 32 steps3.432.303.961.961.301.36
BLD, G=8G=83.612.934.072.021.471.55

BLD scores slightly higher than ELF-B across all six dimensions, including coherence (2.02 vs. 1.96) and overall quality (1.55 vs. 1.36), despite operating on a 16× compressed latent sequence. The largest improvement is in non-repetition (2.93 vs. 2.30). However, both models remain below AR and real text, particularly in coherence and sensibility. These results provide an additional assessment of whole-document quality and show that long-range coherence remains challenging.

W2 / Q4: Generation quality and long-range coherence (continued).

Longer-sequence generation. BLD is not inherently limited to 1024 tokens. The grouped prior generates latents autoregressively across groups, allowing longer sequences to be generated by extending the latent sequence with additional groups. Moreover, block positions use rotary embeddings rather than a fixed learned positional embedding table. However, the current model is trained and evaluated only on 1024-token sequences (64 blocks), so positions beyond 64 have not been seen during training, and generation quality at longer lengths cannot be guaranteed. Evaluating this capability would require testing generation beyond the training length, potentially with additional training on longer documents. Given the limited rebuttal period, we leave these experiments for future work and make no empirical claims beyond 1024 tokens.

W3 / Q3: Controlled ablations and variance.

  • Controlled group-size comparison. We extended Table 4 (original Table 3) to include G=32G=32 and G=64G=64, using the same architecture, frozen encoder and decoder, training recipe, budget, sampler, and training seed as G=8G=8; only the group size changes (R3). The original G=KG=K reference remains in Table 1 as a separately tuned model. In the controlled comparison, G=8G=8 achieves higher MAUVE than G=64G=64 (0.82 vs. 0.75), while G=64G=64 is approximately 6× faster (0.30 vs. 1.82 s).
  • Group-size selection and variance. We now report Gen-PPL, entropy, and MAUVE for all six group sizes, with mean and standard deviation over five evaluations of 1000 documents per checkpoint. G=4G=4 achieves better Gen-PPL (24.4 vs. 26.5) and MAUVE (0.86 vs. 0.82) than G=8G=8, but requires nearly twice the latency (3.61 vs. 1.82 s). We therefore select G=8G=8 as a balance between quality and efficiency.
  • Runtime measurements. All new timings use the same H200 GPU. Table 2 reports optimized inference results, while Table 4 uses eager fp32 measurements, removing the original hardware difference (R2 and R3).
  • Training budget and Cosmos comparison. We confirmed that the original G=KG=K reference was trained for approximately 2.95B tokens (30k initial steps and 15k fine-tuning steps, batch size 64), and updated Appendix C.1. For Cosmos, we checked the released sampling and evaluation code but could not identify the cause of the perplexity difference (98.9 in our evaluation vs. 55.0 reported). We clarify this in Appendix D.3 and avoid directly ranking generation quality against Cosmos because the evaluation settings and output lengths differ. Our comparison focuses on compression and computational cost.

Q2: AR baseline. We re-measured the trained 730M AR model using batched generation with a static KV cache, bf16 weights, and a compiled decoding step on the same H200 as BLD (R2). BLD reaches 100.7 vs. 9.1 documents/s at each model's best batch size (11× higher throughput) and takes 0.467 vs. 3.84 s for a single document (8.2× lower latency). The advantage also holds under eager fp32 execution on the same H200, with 7.7× higher throughput and 4.6× lower latency.

Importantly, BLD requires even fewer generation FLOPs than AR (1.10 vs. 1.57 TFLOPs). This is particularly meaningful for diffusion language models, where iterative denoising typically incurs substantially higher computation than autoregressive generation (e.g., 51.17 and 96.38 TFLOPs for ELF-M and ELF-L). By operating on only 64 compressed latent states and decoding token branches in parallel, BLD substantially reduces both computation and sequential generation depth. We believe this demonstrates an important advantage of strong latent compression: diffusion language generation can achieve not only lower latency and higher throughput than AR, but also lower computational cost, while maintaining competitive generation quality.

Q5: Conditional generation and scale. We added XSum summarization experiments (R4; Appendix D.4). BLD trained on XSum achieves ROUGE-1/2/L of 33.9/11.7/27.8 with 141M parameters, matching ELF-B (105M) on ROUGE-L. Due to limited computational resources, we leave evaluations at larger model scales and longer generation lengths for future work.

Dear Reviewer,

With the discussion period coming to a close, we hope that our responses and new experimental results have helped address your concerns. We would be glad to discuss any remaining questions if time permits.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response (Part 1 / 5)

Dear reviewers,

Thank you for your detailed and constructive feedback. We have carefully considered the reviewers' concerns and revised the manuscript with additional experiments and analyses.

We would like to highlight the significance of strong latent compression in continuous diffusion language modeling. BLD achieves 16× compression over 1024-token sequences while maintaining competitive generation quality, a substantial step beyond the compression settings demonstrated in prior latent diffusion language models. We believe this result is important not only for computational efficiency, but also for showing that strong compression can be made practical through an effective generation architecture. More broadly, it highlights a promising advantage of continuous DLMs: the ability to reduce generation cost by modeling substantially fewer latent states rather than operating at token-level granularity.

The main changes in the revised manuscript are:

  • Architecture contribution and motivation (R1). We clarify the motivation for strong latent compression in continuous DLMs and further distinguish BLD from previous compressed latent diffusion methods, particularly Cosmos. We have added a new discussion of the motivation for compression in Appendix A.1. We also added a compression-ratio sweep (r=4,8,16,32r=4,8,16,32) to examine the trade-off between generation quality and efficiency (Table 3).
  • Computational and runtime efficiency (R2). We re-measured the models under a unified, optimized inference setup on an NVIDIA H200, including bf16 execution, CUDA graphs, and batch sizes up to 512. We also added detailed runtime profiling to analyze computation, memory access, and execution overhead. We further implemented and evaluated cross-group KV caching, reporting its computational and throughput benefits in Appendix C.5. The main runtime results and corresponding appendix have been updated.
  • Group-size analysis (R3). We extended the controlled group-size comparison to include G=32G=32 and G=64G=64, using the same architecture and training settings as the grouped models. We also added MAUVE measurements and repeated evaluations for all group sizes, and updated Table 4 and the corresponding analysis.
  • Conditional generation (R4). We added XSum summarization experiments, including both fine-tuning from OpenWebText and training from scratch. The results show that BLD can support conditional generation with competitive performance, and are reported in Appendix D.4.

The revised manuscript includes these new results and analyses, with changes highlighted in blue. We address the main concerns in more detail below.

R1: Architecture contribution. We would like to emphasize that the main contribution of BLD is an architecture that makes strong latent compression practical for diffusion language generation. The closest prior work, Cosmos, achieves 8× compression for 128-token generation on ROCStories (128 tokens to 16 latents), but struggles to maintain generation quality under compression for longer sequences. On 512-token unconditional OpenWebText generation, even 2× compression reduces MAUVE from 0.702 to 0.263, while its better-performing model uses 512 latents for 512 tokens without sequence-length compression. In contrast, BLD compresses 1024 tokens into only 64 block latents, achieving 16× compression while maintaining competitive generation quality. To make strong compression effective, a grouped latent flow denoises GG latents jointly while conditioning on previously generated groups. A branching decoder then generates tokens through 64 short autoregressive branches running in parallel, so the sequential bottleneck is not simply shifted to token realization.

  • Motivation for compression (new Appendix A.1). We argue that latent compression is an important advantage of continuous DLMs. While DLMs offer the potential for lower generation latency than AR models, iterative denoising often comes at a much higher computational cost. For example, ELF-M (342M) and ELF-L (652M) require 51.2 and 96.4 TFLOPs to generate 1024 tokens, compared with only 1.6 TFLOPs for our AR baseline (730M). This overhead is a major obstacle to practical deployment. Unlike standard discrete DLMs, which maintain one generative state per token, continuous DLMs can represent multiple tokens with a single latent state. Compression therefore offers a distinct opportunity to reduce the computation required at each denoising step, rather than only reducing the number of sampling steps. We now explicitly discuss this motivation in Appendix A.1.
AuthorsOpenReview ↗

General Response (Part 2/5)

  • Efficiency from strong compression. By reducing the generative sequence from 1024 token-level states to only 64 block latents, BLD substantially reduces diffusion computation while retaining efficient parallel token realization. BLD requires only 1.10 TFLOPs per document, compared with 7.2 for ELF-B (6.6×), 96.4 for ELF-L (88×), 26.7 for Cosmos (24×; our measurement on shorter outputs, Appendix D.3), and 1.6 for the AR baseline. We see the ability to translate strong latent compression into end-to-end computational efficiency while maintaining competitive generation quality as a central contribution of BLD. We further validate this trade-off through a compression-ratio sweep (r=4,8,16,32r=4,8,16,32; Table 3). Moving from 8× to 16× compression nearly doubles peak throughput (52.1 to 100.7 documents/s) with little change in MAUVE (0.83 vs. 0.82), while further increasing compression to 32× causes MAUVE to drop sharply to 0.307. The remaining limitations in long-range coherence leave room for further improvements in latent modeling.

R2: Computational and runtime efficiency (Reviewers QXhs, 39rs). We re-measured all models under a unified inference stack using bf16 weights and autocast, CUDA graphs for per-step networks (torch.compile, reduce-overhead), and batch sizes from 1 to 512. The AR baseline uses the trained 730M model with a static key/value cache and a compiled decoding step. Generation quality is reported as the mean over five evaluations of 1000 sampled documents each, using the same configurations as the runtime measurements. In precision-only comparisons under identical sampling and evaluation settings, switching to bf16 changes Gen-PPL by less than 0.1% for both BLD and AR.

ModelParamsLatency, batch 1 (s)Docs/s at batch 1 / 8 / 32Best docs/s (batch)TFLOPs/docGen-PPL / Entropy / MAUVE
BLD G=8, 50 steps594M0.4672.07 / 16.0 / 45.3100.7 (512)1.1026.4 / 5.17 / 0.83
BLD G=8, 32 steps594M0.3622.75 / 19.2 / 55.1109.9 (512)0.8425.5 / 5.14 / 0.76
AR730M3.840.26 / 1.94 / 5.149.1 (256)1.5726.2 / 5.37 / 0.87
ELF-B, 32 steps105M0.06415.6 / 38.8 / 42.443.9 (256)7.2223.2 / 5.16 / 0.78
ELF-B, 64 steps105M0.1188.52 / 20.4 / 22.422.9 (256)14.1919.6 / 5.10 / 0.82
ELF-M, 64 steps342M0.3003.35 / 5.65 / 5.876.0 (128)51.1721.7 / 5.19 / 0.91
ELF-L, 64 steps652M0.5521.81 / 3.50 / 3.693.7 (128)96.3822.9 / 5.28 / 0.91
Real text–––––15.3 / 5.32 / 0.95

ELF and the AR model run out of memory at batch 512. In the paper's eager fp32 setting, re-measured on the same H200 (Table 2 of the paper used an H100), BLD takes 1.82 s at batch 1 and reaches 13.4 documents/s at batch 32, compared with 8.34 s and 1.75 documents/s for AR.

Computational efficiency and throughput gains. BLD's architectural advantage is primarily computational: strong latent compression reduces generation FLOPs, and this reduction translates into substantially higher throughput at larger batch sizes.

  • Compared with ELF-B (32 steps): BLD requires 6.6× fewer FLOPs and achieves 2.3× higher peak throughput (100.7 vs. 43.9 documents/s). At batch 32, the two have comparable throughput (45.3 vs. 42.4 documents/s), while ELF-B remains 7.3× faster for single-document generation.
  • Compared with ELF-M and ELF-L: BLD requires 47× and 88× fewer FLOPs, respectively, while achieving 16.7× and approximately 27× higher peak throughput. For single-document generation, ELF-M is 1.6× faster than BLD, while BLD is 1.2× faster than ELF-L.
  • Compared with AR: BLD achieves 8.2× lower single-document latency and 11× higher peak throughput, while also requiring fewer FLOPs per document.

Batch scaling and GPU utilization. Single-document latency is strongly affected by sequential network calls, whereas large-batch throughput increasingly reflects the computation required per document and GPU utilization. BLD performs 8 backbone passes and 400 flow-head evaluations over groups of 8 latents, followed by short parallel token-decoding branches. ELF-B instead performs 32 denoising steps over all 1024 token positions. At batch 1, BLD's numerous small calls incur substantial overhead, making ELF-B 7.3× faster. However, batching amortizes this overhead across documents: BLD's throughput increases from 2.07 to 100.7 documents/s between batch 1 and 512, whereas ELF-B reaches 42.4 documents/s at batch 32 and peaks at 43.9 documents/s. At larger batch sizes, BLD's 6.6× lower computation per document becomes increasingly important, allowing its computational advantage to translate into higher throughput.

AuthorsOpenReview ↗

General Response (Part 3/5)

Generation quality and limitations. Despite 16× latent compression, BLD maintains competitive generation quality. At 50 steps, BLD achieves Gen-PPL 26.4, closely matching the AR baseline (26.2), and MAUVE 0.83, exceeding ELF-B at both 32 and 64 steps (0.78 and 0.82). Its entropy of 5.17 is also comparable to ELF-B (5.16 at 32 steps) and ELF-M (5.19). While ELF-M and ELF-L achieve better Gen-PPL and MAUVE, they require 47× and 88× more computation, respectively. We therefore emphasize the substantial efficiency gains achieved while maintaining competitive generation quality, without claiming strictly quality-matched speedups. Further improvements in latent modeling may help close the remaining quality gap, particularly in long-range coherence.

Runtime profiling (Reviewer 39rs, Q1). We further profiled single-document generation under both eager and optimized inference settings, separating GPU kernel execution time from GPU idle periods.

Model, settingLatency (s)GPU busy (s)GPU idle (s)Idle shareGPU kernels/document
BLD G=8, eager fp321.820.501.3373%166k
BLD G=8, matched0.470.200.2859%57k
AR, eager fp328.263.414.8459%581k
AR, matched3.842.671.1831%380k
ELF-B, 32 steps, as released0.300.0950.2169%25k
ELF-B, 32 steps, matched0.0650.0430.02233%6.3k
ELF-L, 64 steps, as released1.510.990.5234%125k
ELF-L, 64 steps, matched0.560.510.05410%29k

The profile identifies three aspects of runtime cost:

  • Call overhead. GPU idle periods account for 73% of BLD's latency in eager fp32, decreasing to 59% under the optimized stack. These periods reflect host-side execution and kernel-dispatch overhead, including synchronization. Even after optimization, BLD launches approximately 57k GPU kernels per document, with an average execution time of about 3.4 µs.
  • Computation. Matrix multiplication and attention kernels account for approximately 0.30 s (16%) of eager BLD latency and at least 0.13 s (28%) under the matched stack. The optimized execution substantially reduces overhead, but the large number of small network evaluations continues to limit single-document latency.
  • Memory access. Dedicated data-movement kernels account for approximately 1% of eager BLD latency. Memory access within matrix multiplication and attention kernels cannot be isolated by this measurement. Based on model sizes and invocation counts, we estimate cumulative weight traffic of approximately 221 GB in fp32 for BLD (roughly half in bf16), compared with 2.99 TB for AR in fp32. These are analytical estimates rather than direct measurements of device memory traffic.

Under the matched inference stack, ELF and AR spend a substantially larger fraction of their runtime executing GPU kernels, whereas BLD remains more affected by small-call overhead. This explains why BLD's large computational advantage does not translate proportionally into single-document latency, particularly against the smaller ELF-B model.

Cross-group KV caching (new Appendix C.5). We also implemented KV caching across groups, reducing computation from 1.10 to 0.95 TFLOPs at G=8G=8 and improving peak throughput from 101.4 to 112.2 documents/s. This optimization further reduces computation without changing generation quality.

Overall, these results confirm that BLD's efficiency gains extend beyond FLOP reductions. Strong latent compression substantially reduces generation computation, reaching 100.7 documents/s with the default 50-step sampler and 109.9 documents/s with 32 steps. BLD achieves considerably higher peak throughput than the evaluated baselines under an optimized inference stack. While single-document latency remains affected by sequential calls and kernel overhead, these results demonstrate that strong compression can translate into substantial end-to-end efficiency improvements while maintaining competitive generation quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.