acceptodds
Under review as a conference paper at ICLR 2027

AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling

Abstract

Continuous latent-space modeling has proved effective for image, video, and audio generation, while text generation is still dominated by models that directly predict discrete tokens. Existing continuous language models often inherit embedding spaces not designed for both generation and decoding, or compress autoencoded latents to ease diffusion, potentially sacrificing token-level fidelity. Instead of simplifying the representation to accommodate the generative model, we introduce AURORA-LM, a continuous-latent diffusion language model that preserves a high-capacity, decodable text latent and learns its distribution separately. A Query-based Encoder-Decoder organizes text into a prefix-aligned latent sequence. We then freeze the autoencoder and train a Block-causal Diffusion Transformer to model these full-width latents through blockwise flow matching. However, retaining these high-dimensional latents makes diffusion modeling more challenging. We address this difficulty by restricting only the noisy-input pathway through a low-rank bottleneck while retaining the full clean-latent prediction target, and calibrating the noise-level distribution to the latent width. Finally, we introduce self-trajectory consistency to bridge the gap between training on independently sampled noisy states and inference through iterative denoising. AURORA-LM achieves the best Gen-PPL and MAUVE on OpenWebText free generation and the best ROUGE scores on XSum conditional summarization among the evaluated continuous and diffusion-based language models. Scaling its denoiser to 1B parameters, it also outperforms a larger publicly released latent-diffusion language model under matched prompting and scoring.

Primary area:
generative models
Keywords:
Diffusion Models, Text Diffusion, Generative Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rJcMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper explorates latent diffusion for language generation. They follow the usual latent diffusion paradigm of first pre-training the autoencoder then training the diffusion model in the latent space. In this work, the authors train the autoencoder with causal attention masking so the latents are themselves causal. The diffusion model then generates the latents in a blockwise causal manner. The generated latents are mapped to tokens with the frozen decoder that has causal attention and thus supports KV-caching. It adopts a variety of techniques to improve performance, such as self-conditioning, various autoencoder regularizations, consistency regularization, etc.

The model reports results in the usual OWT setting used by discrete diffusion work, reporting generative perplexity, entropy, and MAUVE. For free generation on OWT, the model reports the lowest Gen-PPL and highest MAUVE with some loss of diversity. The authors also report strong XSum summarization results. The authors also train a larger model at the 1B scale which is evaluated on a range of downstream tasks.

Careful ablations are reported regarding the impact of various design decisions (embedding size, noise schedule, bottleneck width, noise allocation, output parameterization/loss, trajectory consistency).

Critical strengths

The paper presents strong generation performance on the OpenWebText and XSum benchmarks compared to recent diffusion language models. Most of the work in this setting has been using discrete language models and it’s nice to benchmark a latent diffusion approach in a comparable setting.

The latent diffusion language architecture is quite natural. Discrete diffusion is increasingly moving towards blockwise autoregressive approaches and this is a nice design to enable that for latent diffusion models without needing to pay the sequential AR token decoding cost.

There are pretty comprehensive ablations presented across various aspects of the pipeline. The quality (MAUVE) is shown to be quite sensitive to various choices here such as the data dimensionality and schedule parameterization.

Critical weaknesses

The motivation for this work isn’t very clear. Is it primarily to one day simplify multimodal generation? That seems to be what is set up in the introduction.

In terms of architecture, the autoencoder setup is pretty close to COSMOS and indeed adopts some, but not all, of the COSMOS regularizations. The biggest difference is the causal attention and doing away with the pretrained encoder. In some sense, this is a blockwise causal version of the COSMOS model without dependence on a pre-trained encoder. Those are meaningful differences, but the relationship with COSMOS could be stated a bit more clearly.

With respect to the comparisons, the OWT should have the GT reference metrics for the actual dataset text. At least for Gen-PPL and entropy. In general you can always trade entropy for Gen-PPL and that collapse does seem to be happening here to an extent.

Almost all of the metrics in Table 2 appear to be at or below random chance on those benchmarks (Squad is an exception), so it’s hard to draw any conclusion. Generative evaluations are challenging at a small scale, but still the results aren’t meaningful. Some prior latent diffusion work adopted a llm-eval style evaluation setup for latent diffusion language models (https://arxiv.org/abs/2602.20528). I believe their evaluation setup should apply here. This should ideally have a comparable AR baseline to contextualize the results.

While there are many ablations, not all aspects of the method are ablated and I’m not sure all of the ablated choices are meaningful. Self-condition guidance is ablated in the appendix, but self-conditioning altogether is never ablated. The bottleneck dimension ablation (Fig. 4c) doesn’t really show any clear trend and looks like it may be noise.

Questions for authors

What is the motivation/benefit for having a latent diffusion language model?

What are the reference metrics for OWT?

Can you apply the llm-eval style scoring from https://arxiv.org/abs/2602.20528 to your model for Table 2? Potentially adopting the benchmarks from the shared prior work if those are more tractable.

What is the impact of removing self-conditioning?

What is the impact of the bottleneck dimension on the other metrics? Any clearer trend there?

3 comments
AuthorsOpenReview ↗

Follow-up on our response to Reviewer rJcM

Dear Reviewer rJcM,

Thank you very much for the time and care you devoted to reviewing our paper. We greatly appreciate the expertise you brought to your review. Your questions about our choice of continuous text representations and how we evaluate them helped us strengthen our evaluation.

Following your suggestion, we scored the answer options directly on the same 6,830 questions and included a 1B autoregressive model as a reference. We also reported generation metrics for real OpenWebText samples, clarified how our approach differs from COSMOS, and tested the effects of self-conditioning and the input bottleneck.

We hope these additional results and clarifications address your concerns. As the discussion period draws to a close, we would welcome any further feedback you may have.

Thank you once again for your thoughtful suggestions and for the time you have given our work.

Best regards,
The Authors

I appreciate the authors response. I think the direct scoring, additional ablations, and the clarified motivation and relationship with Cosmos strengthen the paper.

It does look like some diversity collpase is occuring which is likely strengthening the Gen-ppl and MAUVE results. However, most of my concerns are addressed, so I will raise my score.

AuthorsOpenReview ↗

Thank you for your thoughtful review

Dear Reviewer rJcM,

Thank you very much for your thoughtful reply and for taking the time to read through our responses. We were glad to hear that we had addressed most of your concerns.

Thank you again for the care and attention you’ve given our work throughout the review process.

Sincerely,

The Authors of Submission 10856

Reviewer_8fKwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose AURORA-LM, a continuous latent diffusion LM that keeps a wide, decodable text latent instead of compressing it to make generation easier. They state three main contributions: 1) a query-based autoencoder whose latents are prefix-aligned, so a block-causal denoiser can generate them block by block; 2) design choices for modeling wide latents: a low-rank bottleneck on the noisy input only, and a noise schedule calibrated to the latent width; 3) self-trajectory consistency, which aligns clean-latent predictions before and after one denoising step. The authors report the best Gen-PPL and MAUVE on OpenWebText and the best ROUGE on XSum among the compared models, and the 1B version outperforms Cola-DLM on nine benchmarks.

Critical strengths
  1. The idea is well motivated: instead of simplifying the target latent for the diffusion model, the authors make the diffusion model handle a high-capacity latent. The proposed recipe is reasonable, and most of its components make sense. The ablations of the recipe components are broad and clean.
  2. Strong results, especially in few-step generation with self-trajectory consistency.
Critical weaknesses
  1. Gen-PPL is compared at different entropies (Table 1a). AURORA-LM has noticeably lower entropy than the AR and diffusion baselines. While MAUVE seems a reasonable proxy for overall quality, the Gen-PPL–entropy trade-off remains unclear.
  2. At the 1B scale, most benchmark scores(Table 2) seem close to the random-guess baseline. The comparison then mostly shows how well the models follow the answer format. There is also no AR baseline.
  3. Sampling speed is not reported. Block-by-block generation with many steps per block and guidance needs many denoiser calls (apparently more than both the AR and diffusion baselines), so the speed–quality trade-off is unclear.
Questions for authors
  1. Could you compare the models at matched entropy, for example with Gen-PPL–entropy curves for all models, including the real-text reference?
  2. Could you add an AR baseline to the 1B-scale evaluation?
  3. Could you report sampling speed against AR and the diffusion baselines?
1 comment
AuthorsOpenReview ↗

Follow-up on our response to Reviewer 8fKw

Dear Reviewer 8fKw,

Thank you very much for the time and care you devoted to reviewing our paper. We greatly appreciate the expertise you brought to your review. Your questions about generation quality, sampling efficiency, and the choice of baselines helped us strengthen our evaluation.

In response, we compared generation quality at comparable levels of unigram entropy, measured sampling times across models, and included a 1B autoregressive model as a reference on the multiple-choice tasks, where each model scored the candidate answers directly.

We hope these additional results and clarifications address your concerns. As the discussion period draws to a close, we would welcome any further feedback you may have.

Thank you once again for your thoughtful suggestions and for the time you have given our work.

Best regards,
The Authors

Reviewer_YTk5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AURORA-LM is a two-stage continuous-latent language model. A query-based autoencoder maps text to a full-length, full-width latent sequence in which each position encodes a token prefix, and is then frozen. A block-causal flow-matching Transformer models these latents block by block, with a low-rank bottleneck on the noisy input only, a noise schedule shifted toward high noise, and a self-trajectory consistency loss. A 130M denoiser reports Gen-PPL 23.56 and MAUVE 0.890 on OpenWebText and the best ROUGE on XSum among the compared models.

Critical strengths
  1. The design study is careful and useful beyond this model. The OWT128 ablations change one factor at a time, use five generation seeds and report full tables. Three findings should carry over to other continuous text diffusion work. What matters is how much training mass sits at high noise, whatever the schedule family. Clean-latent prediction works far better than velocity prediction for wide latents. Self-trajectory consistency gives large few-step gains, raising MAUVE from 0.116 to 0.672 at 8 steps. This is the main reason I think the paper has value.

  2. The system is strong for a continuous model and well documented. It shows that an uncompressed, decodable latent can still support competitive generation, and it scales to a 1B denoiser trained on about 300B tokens. Architecture, training and inference settings are given in detail, including the full sampler sweeps, so the results can be reproduced and built on.

Critical weaknesses
  1. The headline generation comparisons are not matched. AURORA's sampler and guidance scale were selected on the same samples that Table 1 reports, by taking the lowest Gen-PPL in the sweep, while the baselines run at their defaults. Without guidance, every configuration has a Gen-PPL of 42.55 or higher, above the AR baseline's 39.40, and every AURORA configuration has lower unigram entropy than AR and the discrete diffusion models. Sampling also uses about twice the non-embedding parameters of the baselines (denoiser plus decoder) and 4,096 sequential denoiser calls against 1,024 for AR. The XSum margin over ELF-B likewise depends on a guidance scale chosen on the evaluation set. Since the main claim is better generation than these baselines, the comparison needs matched entropy and reported cost.

  2. The 1B comparison does not support the abstract's scaling claim. On the eight multiple-choice tasks in Table 2, AURORA-LM-L averages 31.9 against an average chance level of 32.3, and Cola-DLM is below chance on all eight. Answers are generated and then matched to options, so these numbers mostly reflect how often a model produces a parseable answer. SQuAD (38.2 vs. 25.7 EM) is the only clearly informative column. The two models also differ in training data and compute, and there is no autoregressive reference trained on the same data. Fig. 1(c) and the abstract present this as evidence that the approach scales, which it does not yet show.

Questions for authors
  1. Can you compare all models at matched entropy, for example with a temperature or nucleus sweep for AR and the discrete diffusion baselines, and report the number of network calls and the wall-clock time per 1,024-token sample for each system?
  2. For Table 2, how many examples remain per task, and what fraction of each model's outputs map to no option? Would log-likelihood scoring of the options, or an AR model trained on the same data, change the conclusion?
  3. With the clean-latent target, the bottleneck and no bottleneck differ by only 0.008–0.028 MAUVE, about the size of the seed standard deviations. Is the bottleneck needed in the final system?
  4. What clean token-reconstruction accuracy do the S and L autoencoders reach? The paper's premise is a faithfully decodable latent, but Fig. 4(a) gives no number at σ = 0.
1 comment
AuthorsOpenReview ↗

Follow-up on our response to Reviewer YTk5

Dear Reviewer YTk5,

Thank you very much for the time and care you devoted to reviewing our paper. We greatly appreciate the expertise you brought to your review. Your questions about generation quality, sampling cost, and the interpretation of our multiple-choice results helped us strengthen our evaluation.

In response, we compared generation quality at similar unigram entropy, measured sampling times, and examined how answer matching affected multiple-choice accuracy. We also scored the answer options directly, tested the input bottleneck on 1,024-token sequences, and measured token reconstruction accuracy on clean latents.

We hope these additional results and clarifications address your concerns. As the discussion period draws to a close, we would welcome any further feedback you may have.

Thank you once again for your thoughtful suggestions and for the time you have given our work.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Revision Summary

We are grateful to each of the three reviewers for the care they took in reading our paper and for their thoughtful suggestions. All three reviews highlighted the ablations as a strength, and we particularly appreciated YTk5’s observation that the design study was “careful and useful beyond this model.” We were also glad that the reviewers saw value in preserving high-capacity latents (8fKw) and highlighted both the few-step gains from self-trajectory consistency (YTk5, 8fKw) and the “strong generation performance on the OpenWebText and XSum benchmarks compared to recent diffusion language models” (rJcM).

We have uploaded the revised manuscript, with changes highlighted in blue.

  • OpenWebText generation quality (YTk5, 8fKw, rJcM). Table 18 compares generation quality at similar unigram entropy and includes real OpenWebText samples as a reference.

  • Sampling speed (YTk5, 8fKw). Table 19 reports generation time per 1,024-token sample for AURORA at 32 and 64 steps per block, alongside autoregressive and diffusion baselines.

  • 1B evaluation (YTk5, 8fKw, rJcM). In Appendix A.2.9 (Tables 8–9), we report answer-matching statistics and direct candidate-scoring results for the eight multiple-choice tasks. We include Pythia-1B as an autoregressive reference for direct candidate scoring.

  • Relationship to COSMOS (rJcM). In Appendix A.1.1, we clarify how AURORA differs from COSMOS. AURORA prioritizes preserving representation capacity and adapting diffusion modeling to high-dimensional text latents.

Our replies under each review give the details of the additional experiments. We would be happy to discuss any of these results further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.