- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper analyses the effect of the encoder in continuous latent-diffusion language models, and proposes new encoder distillation strategies that improve generative performance. The key contributions are identifying gaps in the encoding aspect of ELF and proposing improvements for encoder training. Several clear ablations and experiments validate all claims.
Critical strengths
- Using soft labels to improve clustering of related words is a clever idea, and seems to work well.
- Fig 10 is a useful ablation.
Critical weaknesses
- The authors do not clearly explain the reasons for choosing a smaller encoder. Ablations show that encoder depth is not a major factor in performance, nor in E2E sampling latency. Shouldn't a larger encoder be better then?
- The work is narrow in scope. It is seemingly only applicable to ELF-class models with T5-class encoders. The continuous diffusion literature is exploring multiple paradigms in parallel and none has emerged as a clear winner yet. Moreover, a lot of recent work in image diffusion has been moving away from latent-spaces (JIT, pMF). It is unclear why latent spaces may remain viable for language diffusion.
- Embedding collapse is a problem that has not been clearly addressed. For eg, the reconstruction from t=0.3 flips a few words.
Questions for authors
- Continuous DLMs were reinvigorated with three key papers - categorical flow maps, langflow and FMLM. I would recommend that the authors cite these papers when introducing continuous diffusion for language.
- Can the distance to the nearest embedding be tied to errors in generation? For eg, Posterior Refinement finds that FMLM outputs are not one-hot when the model is making errors. Since ELF uses embeddings instead of one-hot vectors, can some similar notion be identified here as well.
- Why is the particular direction of the KL divergence used for training the student on line 283? An ablation considering both direction and explaining their effect would be helpful.
2 comments
Official Comment by Authors (2/2)
4) Embedding collapse. We believe this refers to the round-trip denoising at t = 0.3 (Fig. 7). We assume this is not a collapse, as a round trip is not expected to land at its starting point (SDEdit, Meng et al., 2021). And the changed words end up as meaningful alternatives ('reliability' 'accuracy', 'behavior' 'behaviour').
Embedding collapse can happen when the distillation is not done properly. In the ablation on the distillation objective (Fig. 10), when trained on hard labels, the generated text entropy stays below 4.18 (real text: 5.43). So distilling with soft labels also helps prevent collapse. We have clarified the wording in Fig. 7.
5) Can the distance to the nearest embedding be tied to errors in generation? Yes. A larger distance to the real embeddings confuses the decoder (see Fig. 9: for T5Gemma-2, the median top-1 decoded prob drops from 0.48 within 0.91 nn to 0.23 beyond 0.91 nn), and it affects the text quality. To verify this, we split the generated sequences by their invalid rate into five equal-sized groups, and compare their Gen. PPL. We find that Gen. PPL rises from 30 to 490 across groups for T5Gemma-2, and from 28 to 54 for the student (with Spearman correlation 0.68 and 0.59; see App. D.5 for the plot).
We agree that the embedding error is related to the non-one-hot predictions in FMLMs, since the distance to the nearest real embedding largely decides the decoder's top-1 probability, which resembles the posterior confidence. We think the self-conditioning in ELF plays a similar role to Posterior Refinement, as from sc = 1 to sc = 3 at NFE 32, the invalid positions per sequence decrease from 343.5 to 297.5 for T5Gemma-2 and from 34.3 to 21.0 for the student. We now note this connection in the conclusion.
6) Forward vs. reverse KL in distillation. We use the forward KL , which takes the teacher distribution as the target, and is also the standard setup in Hinton et al. (2015). Here is the teacher's distribution over the words at a position, and is the student's (decoded with the frozen teacher decoder). Forward KL is mode-covering, which means every word that the teacher finds plausible has to have non-zero probability in , or the loss will blow up. So all plausible candidates have to stay decodable from the same student embedding, which pulls the candidate embeddings together. The reverse direction is mode-seeking, so a student can concentrate on the teacher's top word only. We expect distilling with reverse KL to behave closer to the hard-label CE objective that collapses in our ablation Fig. 10. We have added this explanation to Sec. 4.
We thank the reviewer again for the suggestions and are happy to follow up on any remaining points.
Official Comment by Authors (1/2)
We thank the reviewer for the thoughtful and constructive review, and we address the weaknesses and questions below.
1) Why choose a 'smaller' encoder. If 'smaller' means distilling into a smaller student, it is because we started with a common distillation setup, where we initialize the student from every second layer of the teacher to save training cost. The reviewer is right that depth does not affect sampling latency, since the encoder is not involved in sampling. But in training, we encode the data at every step, where the 9-layer student takes 2.1 ms per sequence against 3.4 ms for the 18-layer teacher. We later found that deeper students can be better, as the reviewer suggests (also Reviewer 9tzG, point 3).
2) Missed key papers in continuous DLMs. We thank the reviewer for bringing up this point. In this paper, we look at continuous latent DLMs on pretrained embeddings, while there are also other strong lines of continuous DLMs, including flows on one-hot / simplex representations, and on learned token embeddings. We now cite and discuss the following key papers in the Introduction and Related Work sections,
Our work echoes those works on some key points, for example, the need to generate valid representations (e.g. generating one-hot vectors in FMLMs generating valid embeddings in this work). Please let us know if there are others that we should acknowledge.
3) Why we focus on continuous latent DLMs. As the paradigms have not yet converged, we find it important to test the limits of the continuous latent diffusion route, to help make the picture clearer. Our work is similar in spirit to the Representation Autoencoder (RAE) for image generation, where we test whether the progress of language embedding models (T5Gemma, EmbeddingGemma, Qwen Embedding, etc.) can benefit continuous DLMs. It turns out it can, and the model trained on them outperforms GPT-2-M at the same scale. For images, pixel-space diffusion is indeed advancing quickly (JiT, pMF), while representation encoders are being scaled up as well (RAE for text-to-image), so the question is still open and intriguing for both image and language generation.