acceptodds
Under review as a conference paper at ICLR 2027

Length from the Latents: Dynamic-Length Compressed Latent Diffusion Models

Abstract

Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive models, offering parallel generation and the ability to revise their output through bidirectional attention. However, the adoption of DLMs is still limited due to running diffusion on high-dimensional token-level representations. Some methods overcome this issue by running diffusion in a compressed fixed-dimensional latent space, where the latent space is realized with a compressor network, and later a decompressor network expanding the denoised latents back to token-level representation followed by decoding them to the output tokens. Yet, current methods run parallel decompression assuming a fixed output sequence length, and thus attend bidirectionally over all output positions, including the ones that are beyond the end-of-sequence token. This results in every request paying for the maximum length, and in compute wasted on padding. As a solution, we propose PACED (Parallel Append-only Cached Extensible Decompression). PACED masks the decompressor so each output position attends to all latents but only to itself and to preceding positions, ensuring prefix-consistent decompression that is efficiently extendable. We realize this with the help of training a length head on the denoised compressed latents, the features that are natively length-aware. Post token decoding, if the end-of-sequence token does not appear, we reuse cached internal states of the decompressor, similar to key-value caching, and decompress only the missing positions, instead of entirely rerunning the decompression. We prove this returns exactly the text a full-length decode would have returned, whatever initial length was chosen. Distilling the pretrained decompressor into the prefix-consistent one on sampled latents recovers much of the generation quality while allowing PACED to run batched 1.2–1.8 faster than full-length decompression. PACED thus makes decompression cost follow the output rather than the maximum length, paving the way for compressed-latent DLMs that support long outputs efficiently.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gxbNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PACED for dynamic-length decompression in compressed-latent diffusion language models. PACED makes the decompressor prefix-consistent by masking attention between output positions, predicts an initial output length from the denoised latent, and uses cached extension when the prediction is too short. The paper proves that recovery reproduces the full-length output of the same prefix-consistent decompressor and evaluates the method on ROCStories, CNN/DailyMail, and ELI5.

Critical strengths
  1. The prefix-consistency and recovery construction is clean. The exact-recovery argument is simple but useful, and the paper validates it carefully, including numerical-precision tests.

  2. The experimental protocol is more careful than usual for this type of systems work. In particular, the paper reports actual decompression timing, calibration behavior, multiple fine-tuning seeds, and the quality/efficiency trade-off introduced by the new attention mask.

Critical weaknesses
  1. The practical end-to-end benefit is not established in a regime with good generation quality. With the default 200-step sampler, decompression is only about 0.6% of total generation time. The reported 1.2--1.8x speedups are for decompression alone. The paper argues that PACED becomes more useful with few-step samplers, but its few-step experiment is obtained by simply reducing the number of steps of the existing model, which severely degrades generation quality. Thus the main regime where PACED would materially affect end-to-end latency remains hypothetical.

  2. Exact recovery is only with respect to the modified prefix-consistent decompressor, not the original full-attention Cosmos decompressor. Obtaining the prefix property requires changing and fine-tuning a large decompressor, and generation quality remains worse than the pretrained model, especially on CNN/DailyMail and ELI5. Therefore the efficiency gain is not output-preserving with respect to the original model. This trade-off weakens the practical significance of the result.

Questions for authors
  1. Can the authors report end-to-end batched generation latency, including diffusion, and demonstrate a meaningful speedup in a setting where generation quality remains comparable to the 200-step model? A result with a quality-preserving few-step sampler would materially change my assessment.

  2. Can the authors demonstrate a setting where the prefix-consistent decompressor matches the original full-attention decompressor more closely in generation quality while retaining meaningful length savings? In particular, results on a larger maximum length or a conditional generation task would help establish that the trade-off remains useful outside the current L=128 unconditional setting.

Reviewer_cEySOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Continuous diffusion is well-established in continuous domains, such as vision. More recently, it has been gaining traction in discrete domains, such as language, too. One approach to continuous diffusion for language is latent diffusion, where a denoising model first iteratively refines a (low-dimensional) latent sequence and a decompressor finally translates the latents to the discrete language domain. Common decompressors, e.g., the one from the Cosmos latent diffusion model, generate a fixed sequence of L tokens in parallel using a Transformer architecture with bidirectional attention. However, part of the length-L sequence will be likely discarded (i.e., all tokens trailing the end-of-sequence (EOS) token), yielding wasted compute.

The goal of this paper is to reduce the wasted compute of the decompression module of Cosmos. The key step is to replace the decompressor's bidirectional attention with causal attention. This allows the decompressor to speculatively only decode a block of length N_0. If the EOS token is contained in the generated block, L-N_0 token computations are saved. Otherwise, the decoding continues, incrementally adding blocks (of reducing size) until the EOS is generated. The initial block size (N_0) is estimated with a learned regressor conditioned on the denoiser's latents. In order to compensate for the reduced token interaction due to causal attention, the paper proposes a distillation technique, which uses a hybrid loss guided by the teacher decompressor (using bidirectional attention).

Experiments are conducted on unconditional language generation, using domain-specific Cosmos checkpoints for ROCStories, CNN/DailyMail, and ELI5. Unfortunately, the proposed distillation recipe cannot retrieve the original generation quality, lagging behind in both generative perplexity as well as MAUVE score. Yet, the number of generated tokens can be reduced thanks to the causal attention and partial block decoding. While the latency reduction is negligible in single sequence generation (concurrency 1) due to the memory-bound nature of decoding, increasing the concurrency reveals the saved compute and gives 1.2-1.8x speedup (only decompression).

Critical strengths

Dynamic length generation is an open problem in diffusion modeling. While limited in their scope, this work aims to improve the efficiency of dynamic length generation of one particular latent diffusion model.

Critical weaknesses

W1: Narrow scope with limited impact. My main reservation concerns the paper's limited scope, optimizing one negligible component of a particular architecture, which has limited impact on the overall performance (i.e., latency).

a) Decompression makes up only a negligible part in the overall generation cost with multi-step denoising. However, multiple denoising steps are required for achieving acceptable generation quality. The negligible impact of the decompression is explicitly shown in Figure 2 (left). Moreover, the reported speedups in concurrent servings (1.2-1.8x) account only for the decompression alone.

b) Decompression is memory-bound in low concurrency settings, hence, spending additional compute for the discarded tokens is negligible in memory-bound regime. Instead, this paper adds additional latency by incrementally decompress blocks. While this saves overall compute, the actual speed up gains can only be demonstrated when going towards compute-bound regime (high concurrency setting). However, diffusion models trail behind autoregressive generation both in terms of per-user latency and system throughput in high concurrency settings (see e.g. [1]); hence, the utility of the end-to-end system compared to other generation approaches would have to be demonstrated.

W2: Low generation quality The proposed causal decompressor can decode blocks incrementally without any loss. However, all the loss-less decompression guarantees rely on a strong causal decompressor (through a transfer from bidirectional to causal), which the paper cannot demonstrate. While a clear comparison between the causal and prefix-consistent decompressor with the proposed distillation is not provided, the ablation in Table 1 between "Reconstruction-adapted" and "Prefix-consistent, earlier recipe" points to the effect (with a different training method). Showing a clear comparison using the distillation as well as overall closing the gap would strengthen the paper.

[1] Yonggan et al., "Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding," https://arxiv.org/abs/2607.05722.

Questions for authors
  • How does the end-to-end speedup look like using the entire generation architecture (denoiser + decompressor)?
  • What is the quality impact of going from bidirectional to causal attention using the same distillation technique?
  • How can the quality gap between bidirectional and causal decompressor be closed?
  • Generative perplexity can be hacked by generating low-diversity samples [2, 3]. Hence, additionally considering the unigram entropy is recommended.

Minor:

  • L194: "PACED [...], instead evaluates only as many positions as the output needs." => that's only true when the estimation N_0 exactly matches the EOS position, which may be rare.
  • Why are the figure captions smaller?
  • Section 5.5.: I would rather use the term "denoiser steps" instead of "sampler steps", particularly as no sampling is in place in latent diffusion.
  • In my understanding, Table 8 uses causal attention for the baseline and the block generation. A clearer comparison would consider bidirectional generation with the proposed method.

[2] Pynadath et al., "Generative Frontiers: Why Evaluation Matters for Diffusion Language Models," http://arxiv.org/abs/2604.02718 [3] France et al., "Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics," http://arxiv.org/abs/2606.08417

Reviewer_EeBVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors consider autoencoders used for latent diffusion for text generation, such as LD4LG and Cosmos. In the paper they describe their approach to the Cosmos architecture. Their goal is to accelerate the decompressor of Cosmos. Specifically, they note that the decompressor produces a sequence of fixed length L (a hyperparameter) in parallel and then trims it at EOS. The authors argue that by using an additional part that predicts output length from the latent, it is possible to accelerate decoding. To do so, the authors propose to use such a length prediction head to decode the predicted number of tokens, and if there is no EOS among them, then to add half of the current token count more until they meet the limit or EOS. To make iterative decoding faster, they use a causal mask between output positions instead of the bidirectional mask used in the original Cosmos, which makes it possible to reuse cached states at the decoding stage. In experiments, the authors use pretrained Cosmos, train an additional length prediction head, and distill the bidirectional decompressor into a causal version.

Critical strengths

None

Critical weaknesses

Limited novelty The authors add a length prediction head to pretrained Cosmos. Then they distill the original Cosmos with a bidirectional mask to work with the Cosmos with a causal mask. They combine decompression with mask together with length prediction into a simple procedure: decode the predicted number of tokens, then keep a cache of hidden states for this part, then, if there is no EOS, decode and extend the sequence by half of its current length until EOS or the length limit is reached. Proposition 1 follows directly from the causal mask. Thus, this method is mainly a small empirical extension to the original Cosmos model. Its practical value therefore needs to be established experimentally.

The practical motivation is not clear. The decompression part is not the bottleneck in the diffusion language models, the bottleneck is the generation part. The authors show by themselves that on the full generation, decompression takes only 0.6% of time for the 200-step solver (used for 128 tokens). Hence, even complete elimination of this part would save only 0.6% of runtime. For a smaller number of steps, the authors show in Table 10 that performance starts to degrade quickly even after transitioning from 200 steps to 50 steps, while the share of the decompression part is only 2.3%.

Questions for authors

Is the first stage with domain adaptation necessary? Did you try to omit this stage?

Why does the implementation cache layer states but recompute their key and value projections (Appendix A.2) instead of classical KV cache?

Comments on the paper

1 comment

We thank the reviewers for their careful reading and feedback. We agree that, with the 200-step sampler, decompression is a small share of the generation cost, and that the gap in quality to the pretrained decompressor remains open while promisingly achievable.

Our main aim, however, was not simply to accelerate decompression, but to make diffusion-based text generation capable of generating variable-length output natively while remaining parallel. Because compressed-latent models fully denoise the content before decoding it, the output length does not need to be fixed in advance. PACED makes the decoded text independent of the number of positions evaluated, so it can be decoded in parallel blocks, streamed, and extended without being rewritten. On the efficiency side, the work is also intended to be forward-looking: as samplers require fewer steps and latents become more compressed, decompression accounts for a larger share of the generation cost, and evaluating only the positions an output requires may become increasingly valuable. We see this as a step toward diffusion models that generate long outputs both in parallel and with lengths determined at generation time, and we will take the reviewers’ suggestions into account in future work on this direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.