acceptodds
Under review as a conference paper at ICLR 2027

One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion

Abstract

Most continuous diffusion language models process one latent position per token at each sampling step, making generation expensive. Two-stage methods lower the cost by reducing the latent length, but they fix the compressed embedding space before training the diffusion model. Embeddings from the fixed space can be dif- ficult to model with diffusion and decode reliably into tokens, which limits gen- eration quality after compression. To address this problem, we introduce JPEG- DLM (Joint-embedding Prediction for Efficient Generation with Diffusion Lan- guage Model), which jointly trains a compressor, a flow matching model and a de- coding module. With joint-embedding prediction, JPEG-DLM learns compressed embeddings that are more structured, easier to model with diffusion and reliably decodable into tokens. JPEG-DLM achieves the lowest mean Gen-PPL and high- est throughput among recent diffusion and flow models on LM1B and OWT. At a compression rate of 0.5 on OWT, it reaches a Gen-PPL of 34.52 and approximately 2.3×ELF’s throughput. These results suggest that jointly learning compressed em- beddings offers a promising path toward efficient diffusion language modeling.We include our implementation code in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Model, Flow Matching, Latent-length Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hUnAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes JPEG-DLM, a continuous diffusion language model that jointly learns compressed latent representations, a flow matching model, and a token decoding module. Its joint-embedding prediction (JEP) objective predicts clean embeddings produced by an EMA target compressor from corrupted features. The JEP predictor shares network parameters with the flow matching model, while feature reconstruction and token-level cross-entropy supervise decoding. Experiments on LM1B and OpenWebText demonstrate improved generation throughput and lower generative perplexity.

Critical strengths
  1. Reducing the latent sequence length lowers the computational cost of repeated model evaluations. The reported throughput improvement over ELF is substantial and supports the practical value of latent compression.

  2. On LM1B, JPEG-DLM achieves substantially lower Gen-PPL than the evaluated baselines while maintaining competitive MAUVE and improving throughput.

  3. The paper examines compression rates, latent widths, network sharing, loss components, corruption strengths, and embedding regularization. The appendix also acknowledges limitations in long-sequence generation and discusses potential evaluation confounds.

Critical weaknesses
  1. Joint learning of latent representations and generative models already exists, as acknowledged in the paper. Table 2 shows a considerably larger performance degradation from removing network sharing than from removing JEP. Since network sharing follows ELF, the experiments do not fully establish what the proposed JEP formulation contributes beyond existing shared-network training and noisy reconstruction objectives.

  2. The comparison does not isolate the benefit of joint training. JPEG-DLM and COSMOS differ in several aspects beyond their joint versus two-stage training schedules. Moreover, the main comparison retains final-layer T5 features for COSMOS despite the appendix demonstrating substantial improvements from earlier-layer features. A controlled two-stage version of JPEG-DLM and comparisons with existing joint-training methods would provide stronger evidence.

  3. The latent-space analyses support narrower conclusions than claimed. Higher correlation between latent positions does not necessarily imply a distribution that is easier to model; it can also reflect redundancy. Similarly, low Gen-PPL along latent interpolations demonstrates fluent decoding but does not establish preservation of the underlying information. These analyses do not sufficiently distinguish robust representations from decoding that favors generic outputs.

Questions for authors
  1. Could the authors compare JPEG-DLM with a two-stage variant using the same architecture, comparable training objectives, and a matched total training budget to isolate the benefit of joint training? How does JPEG-DLM compare with existing joint-training approaches such as LDLM and Cola DLM?

  2. What additional benefit does EMA-target JEP provide over ordinary noisy autoencoding or consistency learning? Could the authors provide controlled ablations that separately remove the EMA target mechanism and feature corruption?

1 comment
Reviewer_hUnAOpenReview ↗

RE: Rebuttal

Thank you for the detailed responses and additional experiments. The new comparisons and ablations substantially address my concerns. Please include the training-budget details for the two-stage comparison and qualify the conclusions drawn from the latent-space analyses in the revised manuscript.

Reviewer_PQSbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes JPEG-DLM, a continuous diffusion language model that generates text in a compressed latent space with fewer latent positions than tokens. Unlike two-stage methods, JPEG-DLM jointly trains a compressor, a flow matching model and a decoding module. The compressed space is shaped by a joint-embedding prediction (JEP) objective, which trains the flow matching network to recover the clean compressed embedding from a noise-corrupted input.

The authors evaluate JPEG-DLM on unconditional generation on LM1B and OWT, where it achieves the lowest Gen-PPL and the highest throughput among diffusion and flow models, including about 2.3× the throughput of ELF on OWT. The paper also compares against the two-stage COSMOS across compression rates, analyzes the structure of the learned latents, and ablates the main design choices.

Critical strengths
  1. The paper is well written and easy to follow. The method is presented clearly, and Figure 2 is especially helpful for understanding how the whole pipeline works. The core motivation of using fewer latent positions than tokens is also well founded, since attention cost grows quadratically with sequence length, and the paper shows this translates into a 2.3× speedup over ELF.

  2. The experiments are thorough. Table 2 ablates most of the design choices, and the paper also studies the effect of latent length and width. Results are averaged over five seeds, and the paper reports MAUVE alongside Gen-PPL, which is useful since Gen-PPL alone can be misleading.

Critical weaknesses
  1. The main motivation of the paper is to learn the compressed representation jointly with the generative model. However, joint training of the latent space has already been explored by LDLM and Cola DLM, which the paper cites but does not compare against. Instead, the experiments in Section 4.3 only compare against the two-stage COSMOS.

  2. While JPEG-DLM achieves the lowest Gen-PPL, it also has the lowest sample entropy among all methods on both LM1B and OWT. Lower entropy is known to artificially reduce Gen-PPL. This is especially clear on OWT, where JPEG-DLM has a similar Gen-PPL to LangFlow (34.52 vs. 36.48) but noticeably lower entropy (5.07 vs. 5.24) and MAUVE (0.80 vs. 0.92).

  3. Most of the reported speedup over the discrete baselines comes from the ELF setup that JPEG-DLM builds on, which already reaches 14.50× on OWT using 32 sampling steps, compared to 128 or 1024 steps for the other baselines. The gain attributable to latent compression is roughly 2.3× over ELF. Moreover, the paper does not include few-step distillation baselines such as FMLM or DFM.

Questions for authors
  1. Can the authors compare against LDLM-style joint training at the same compression rate?

  2. Can the authors provide Gen-PPL versus entropy trade-off curves instead of single-point results, as several recent works have started to do?

Reviewer_jNWQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Joint-embedding Prediction for Efficient Generation with Diffusion Language Model (JPEG-DLM). The proposed method allows for a joint learning of a flow matching model over compressed text representations. Language modeling performance of the method is tested on well established benchmarks of different context lengths. Authors ablate their algorithmical choices and present qualitative analysis of resulting compressed latent space. They also conduct latency throughput analysis proving the method's superior inference speed.

Critical strengths

The paper is clearly written. The text is easy to follow and it is supplied with sufficient technical details. The authors present extensive experiment setup description which is undoubtedly helpful.

The idea to explore joint embedding prediction as the representation shaping signal, to my knowledge, is novel in language diffusion. I find it to be promising, since switching the training signal from token-space to latent-space increases efficiency and seems to be more natural.

Critical weaknesses

Main comparison with the baselines is poor and I find its results to be misinterpreted.

First of all, I can't find a compelling reason to use ODE sampling for ELF, since the original paper brings its best results (with 32 steps) in SDE setting. The urge to equate the setup for fairness sake is understandable, however the claim of "achieving the lowest mean Gen-PPL ... among recent diffusion and flow models" (line 21) must be significantly weakened.

Second, the comparison lacks the main baseline −- LDLM, which also trains its method jointly and which the authors discuss in their work. I find it to be the most natural baseline to compare with.

The authors also seem to ignore very low diversity of the method. While on LM1B the low entropy can be explained as a tradeoff with quality, on OWT it clearly loses to all baselines. Again, this does not hold to the claim of outperforming most recent diffusion. And again, the methods advantages could be presented more honest in the light of equated compute/latency with the notion of significantly lower diversity.

Algorithmical ablations of the method seem rather incomplete too. It lacks one of the most important ablations: two-stage learning of the JPEG-DLM method itself. There is no clear evidence that joint learning is helping. While authors argue that Lflow\mathcal{L}_{flow} indirectly influences the representation space, the gradients from it still do not shape the latent space. It remains unclear for me if the joint training plays a role here, since the co-adaptation of losses seems very limited.

Considering these weaknesses, I rate the current manuscript as weak accept.

Questions for authors
  1. In the experiment with separate nets, did the gradients from Lce\mathcal{L}_{ce} flow into the denoiser?
  2. Why haven't you eplored SDE sampling?

If the authors would supply more thorough baselines comparisons, significantly weaken their claims and provide additional ablation, I would potentially raise my score

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.