- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper explorates latent diffusion for language generation. They follow the usual latent diffusion paradigm of first pre-training the autoencoder then training the diffusion model in the latent space. In this work, the authors train the autoencoder with causal attention masking so the latents are themselves causal. The diffusion model then generates the latents in a blockwise causal manner. The generated latents are mapped to tokens with the frozen decoder that has causal attention and thus supports KV-caching. It adopts a variety of techniques to improve performance, such as self-conditioning, various autoencoder regularizations, consistency regularization, etc.
The model reports results in the usual OWT setting used by discrete diffusion work, reporting generative perplexity, entropy, and MAUVE. For free generation on OWT, the model reports the lowest Gen-PPL and highest MAUVE with some loss of diversity. The authors also report strong XSum summarization results. The authors also train a larger model at the 1B scale which is evaluated on a range of downstream tasks.
Careful ablations are reported regarding the impact of various design decisions (embedding size, noise schedule, bottleneck width, noise allocation, output parameterization/loss, trajectory consistency).
Critical strengths
The paper presents strong generation performance on the OpenWebText and XSum benchmarks compared to recent diffusion language models. Most of the work in this setting has been using discrete language models and it’s nice to benchmark a latent diffusion approach in a comparable setting.
The latent diffusion language architecture is quite natural. Discrete diffusion is increasingly moving towards blockwise autoregressive approaches and this is a nice design to enable that for latent diffusion models without needing to pay the sequential AR token decoding cost.
There are pretty comprehensive ablations presented across various aspects of the pipeline. The quality (MAUVE) is shown to be quite sensitive to various choices here such as the data dimensionality and schedule parameterization.
Critical weaknesses
The motivation for this work isn’t very clear. Is it primarily to one day simplify multimodal generation? That seems to be what is set up in the introduction.
In terms of architecture, the autoencoder setup is pretty close to COSMOS and indeed adopts some, but not all, of the COSMOS regularizations. The biggest difference is the causal attention and doing away with the pretrained encoder. In some sense, this is a blockwise causal version of the COSMOS model without dependence on a pre-trained encoder. Those are meaningful differences, but the relationship with COSMOS could be stated a bit more clearly.
With respect to the comparisons, the OWT should have the GT reference metrics for the actual dataset text. At least for Gen-PPL and entropy. In general you can always trade entropy for Gen-PPL and that collapse does seem to be happening here to an extent.
Almost all of the metrics in Table 2 appear to be at or below random chance on those benchmarks (Squad is an exception), so it’s hard to draw any conclusion. Generative evaluations are challenging at a small scale, but still the results aren’t meaningful. Some prior latent diffusion work adopted a llm-eval style evaluation setup for latent diffusion language models (https://arxiv.org/abs/2602.20528). I believe their evaluation setup should apply here. This should ideally have a comparable AR baseline to contextualize the results.
While there are many ablations, not all aspects of the method are ablated and I’m not sure all of the ablated choices are meaningful. Self-condition guidance is ablated in the appendix, but self-conditioning altogether is never ablated. The bottleneck dimension ablation (Fig. 4c) doesn’t really show any clear trend and looks like it may be noise.
Questions for authors
What is the motivation/benefit for having a latent diffusion language model?
What are the reference metrics for OWT?
Can you apply the llm-eval style scoring from https://arxiv.org/abs/2602.20528 to your model for Table 2? Potentially adopting the benchmarks from the shared prior work if those are more tractable.
What is the impact of removing self-conditioning?
What is the impact of the bottleneck dimension on the other metrics? Any clearer trend there?
3 comments
I appreciate the authors response. I think the direct scoring, additional ablations, and the clarified motivation and relationship with Cosmos strengthen the paper.
It does look like some diversity collpase is occuring which is likely strengthening the Gen-ppl and MAUVE results. However, most of my concerns are addressed, so I will raise my score.
Thank you for your thoughtful review
Dear Reviewer rJcM,
Thank you very much for your thoughtful reply and for taking the time to read through our responses. We were glad to hear that we had addressed most of your concerns.
Thank you again for the care and attention you’ve given our work throughout the review process.
Sincerely,
The Authors of Submission 10856
Follow-up on our response to Reviewer rJcM
Dear Reviewer rJcM,
Thank you very much for the time and care you devoted to reviewing our paper. We greatly appreciate the expertise you brought to your review. Your questions about our choice of continuous text representations and how we evaluate them helped us strengthen our evaluation.
Following your suggestion, we scored the answer options directly on the same 6,830 questions and included a 1B autoregressive model as a reference. We also reported generation metrics for real OpenWebText samples, clarified how our approach differs from COSMOS, and tested the effects of self-conditioning and the input bottleneck.
We hope these additional results and clarifications address your concerns. As the discussion period draws to a close, we would welcome any further feedback you may have.
Thank you once again for your thoughtful suggestions and for the time you have given our work.
Best regards,
The Authors