acceptodds
Under review as a conference paper at ICLR 2027

Scaling and Distilling Text Embeddings for Better Diffusibility

Abstract

Diffusion language models (DLMs) offer a promising alternative to autoregressive (AR) language generation. Recent advances in continuous DLMs, which apply latent diffusion to continuous text embeddings, raise a practical question: which embedding makes the best latent space, i.e., the most diffusible? To answer this, we search through different embeddings and find that *scaling* up to the advanced and general-purpose T5Gemma-2 embeddings greatly improves generative performance. But such raw embeddings are not optimal, as they are overly discriminative and even the embeddings of plausible alternative words are separated. Consequently, continuous diffusion often fails to reach any of them and ends up at an invalid embedding in between. To address this, we *distill* T5Gemma-2 into a student encoder that learns the teacher's decoded probabilities as soft labels. Learning from such soft labels makes the student pull the alternative embeddings closer while maintaining the encoding-decoding mechanism. The distilled embeddings form a more connected and diffusible latent space, improving over the vanilla T5Gemma-2 embeddings. As a result, our medium-sized DLM achieves **Gen. PPL 17.8** (against real-text PPL 15.4) at real-text entropy on OpenWebText, outperforming GPT-2-M as an AR counterpart.

Primary area:
generative models
Keywords:
Diffusion Language Models, Text Embeddings, Knowledge Distillation, Language Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uro6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyses the effect of the encoder in continuous latent-diffusion language models, and proposes new encoder distillation strategies that improve generative performance. The key contributions are identifying gaps in the encoding aspect of ELF and proposing improvements for encoder training. Several clear ablations and experiments validate all claims.

Critical strengths
  • Using soft labels to improve clustering of related words is a clever idea, and seems to work well.
  • Fig 10 is a useful ablation.
Critical weaknesses
  • The authors do not clearly explain the reasons for choosing a smaller encoder. Ablations show that encoder depth is not a major factor in performance, nor in E2E sampling latency. Shouldn't a larger encoder be better then?
  • The work is narrow in scope. It is seemingly only applicable to ELF-class models with T5-class encoders. The continuous diffusion literature is exploring multiple paradigms in parallel and none has emerged as a clear winner yet. Moreover, a lot of recent work in image diffusion has been moving away from latent-spaces (JIT, pMF). It is unclear why latent spaces may remain viable for language diffusion.
  • Embedding collapse is a problem that has not been clearly addressed. For eg, the reconstruction from t=0.3 flips a few words.
Questions for authors
  • Continuous DLMs were reinvigorated with three key papers - categorical flow maps, langflow and FMLM. I would recommend that the authors cite these papers when introducing continuous diffusion for language.
  • Can the distance to the nearest embedding be tied to errors in generation? For eg, Posterior Refinement finds that FMLM outputs are not one-hot when the model is making errors. Since ELF uses embeddings instead of one-hot vectors, can some similar notion be identified here as well.
  • Why is the particular direction of the KL divergence used for training the student on line 283? An ablation considering both direction and explaining their effect would be helpful.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the thoughtful and constructive review, and we address the weaknesses and questions below.

1) Why choose a 'smaller' encoder. If 'smaller' means distilling into a smaller student, it is because we started with a common distillation setup, where we initialize the student from every second layer of the teacher to save training cost. The reviewer is right that depth does not affect sampling latency, since the encoder is not involved in sampling. But in training, we encode the data at every step, where the 9-layer student takes 2.1 ms per sequence against 3.4 ms for the 18-layer teacher. We later found that deeper students can be better, as the reviewer suggests (also Reviewer 9tzG, point 3).

2) Missed key papers in continuous DLMs. We thank the reviewer for bringing up this point. In this paper, we look at continuous latent DLMs on pretrained embeddings, while there are also other strong lines of continuous DLMs, including flows on one-hot / simplex representations, and on learned token embeddings. We now cite and discuss the following key papers in the Introduction and Related Work sections,

  • Roos et al., "Categorical flow maps," ICML 2026.
  • Lee et al., "Flow map language models: One-step language modeling via continuous denoising," arXiv 2026.
  • Davis et al., "Scaling categorical flow maps," arXiv 2026.
  • Agarwal et al., "Posterior refinement: Fast language generation via any-order flow maps," arXiv 2026.
  • Tang & Wang, "Discrete Beckmann transport models for one-step language modeling and reasoning," arXiv 2026.
  • Chen et al., "Langflow: Continuous diffusion rivals discrete in language modeling," arXiv 2026.
  • Deschenaux & Gulcehre, "Language modeling with hyperspherical flows," arXiv 2026.

Our work echoes those works on some key points, for example, the need to generate valid representations (e.g. generating one-hot vectors in FMLMs generating valid embeddings in this work). Please let us know if there are others that we should acknowledge.

3) Why we focus on continuous latent DLMs. As the paradigms have not yet converged, we find it important to test the limits of the continuous latent diffusion route, to help make the picture clearer. Our work is similar in spirit to the Representation Autoencoder (RAE) for image generation, where we test whether the progress of language embedding models (T5Gemma, EmbeddingGemma, Qwen Embedding, etc.) can benefit continuous DLMs. It turns out it can, and the model trained on them outperforms GPT-2-M at the same scale. For images, pixel-space diffusion is indeed advancing quickly (JiT, pMF), while representation encoders are being scaled up as well (RAE for text-to-image), so the question is still open and intriguing for both image and language generation.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

4) Embedding collapse. We believe this refers to the round-trip denoising at t = 0.3 (Fig. 7). We assume this is not a collapse, as a round trip is not expected to land at its starting point (SDEdit, Meng et al., 2021). And the changed words end up as meaningful alternatives ('reliability' →\to 'accuracy', 'behavior' →\to 'behaviour').

Embedding collapse can happen when the distillation is not done properly. In the ablation on the distillation objective (Fig. 10), when trained on hard labels, the generated text entropy stays below 4.18 (real text: 5.43). So distilling with soft labels also helps prevent collapse. We have clarified the wording in Fig. 7.

5) Can the distance to the nearest embedding be tied to errors in generation? Yes. A larger distance to the real embeddings confuses the decoder (see Fig. 9: for T5Gemma-2, the median top-1 decoded prob drops from 0.48 within 0.91 nn to 0.23 beyond 0.91 nn), and it affects the text quality. To verify this, we split the generated sequences by their invalid rate into five equal-sized groups, and compare their Gen. PPL. We find that Gen. PPL rises from 30 to 490 across groups for T5Gemma-2, and from 28 to 54 for the student (with Spearman correlation 0.68 and 0.59; see App. D.5 for the plot).

We agree that the embedding error is related to the non-one-hot predictions in FMLMs, since the distance to the nearest real embedding largely decides the decoder's top-1 probability, which resembles the posterior confidence. We think the self-conditioning in ELF plays a similar role to Posterior Refinement, as from sc = 1 to sc = 3 at NFE 32, the invalid positions per sequence decrease from 343.5 to 297.5 for T5Gemma-2 and from 34.3 to 21.0 for the student. We now note this connection in the conclusion.

6) Forward vs. reverse KL in distillation. We use the forward KL KL(pT‖pS)KL(p_T ‖ p_S), which takes the teacher distribution as the target, and is also the standard setup in Hinton et al. (2015). Here pTp_T is the teacher's distribution over the words at a position, and pSp_S is the student's (decoded with the frozen teacher decoder). Forward KL is mode-covering, which means every word that the teacher finds plausible has to have non-zero probability in pSp_S, or the loss will blow up. So all plausible candidates have to stay decodable from the same student embedding, which pulls the candidate embeddings together. The reverse direction KL(pS‖pT)KL(p_S ‖ p_T) is mode-seeking, so a student can concentrate on the teacher's top word only. We expect distilling with reverse KL to behave closer to the hard-label CE objective that collapses in our ablation Fig. 10. We have added this explanation to Sec. 4.

We thank the reviewer again for the suggestions and are happy to follow up on any remaining points.

Reviewer_6ekfOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study which frozen text encoder gives the best latent space for continuous diffusion LMs, keeping the diffusion model (ELF) fixed. They state three contributions: 1) scaling the encoder along T5, T5Gemma-1 and T5Gemma-2 improves generation; 2) an analysis showing that T5Gemma-2 keeps plausible alternative words far apart, so generated embeddings land between them and decode poorly ("invalid embeddings"); 3) distillation of T5Gemma-2 into a smaller student trained on the teacher decoder's soft labels, which pulls the alternatives closer. ELF on the student embeddings beats ELF on raw T5Gemma-2 on OWT and LM1B, and the medium model beats GPT-2-M in Gen. PPL at real-text entropy.

Critical strengths
  1. The paper addresses an important open problem for continuous DLMs: which latent space to diffuse in. The experimental design is sound in principle: the diffusion model and training recipe are fixed and only the encoder changes, so the effect of the latent space is isolated.
  2. The distillation is simple and cheap, and it gives a clear gain over the teacher at base scale.
Critical weaknesses
  1. The "invalid embedding" analysis is not well defined. The embeddings are contextual and the candidates are words, but the paper never explains how a candidate's embedding at a given position is obtained (Sec. 3.2).
  2. The thresholds (0.91 nn, top-1 < 0.6) look arbitrary and are tuned on the teacher's own distance distribution, which is then used to show that the teacher is worse.
  3. It is unclear how the Gen. PPL–entropy curves are built. Only Fig. 8 says what is swept; Fig. 1, 2, 10 and 11 do not.
  4. Small sample sizes and no error bars: 64 sequences for the analysis, and 256 samples from one seed for the ablations and for choosing operating points. At medium scale, the gain over the teacher is small (Table 1).
  5. The "scaling" claim is not really about scale. The compared encoders differ in pretraining data, objective, vocabulary and width, not only in size. Within families, the effect of size is mixed: T5Gemma-1-B clearly improves over T5Gemma-1-S, but T5-small and T5-base nearly overlap in Fig. 2, and T5Gemma-2 is tested in a single size (1B/4B are left for future work). What the paper shows is that a stronger modern encoder gives a better latent space, not a scaling trend.
Questions for authors
  1. How is the embedding of a candidate word computed in Sec. 3.2? What value of k is used there, and is it the same as the top-256 used for distillation?
  2. How do the invalid rates change with other thresholds and with a normalization shared between the two spaces?
  3. What is swept along each curve in Fig. 1, 2, 10 and 11?
  4. Gen. PPL and per-sample unigram entropy say little about diversity across samples. Could you report n-gram diversity metrics at the selected operating points?
  5. Could you add T5Gemma-2-1B/4B, or another size sweep within one family, to support the scaling claim?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the in-depth and constructive review. We address the weaknesses and questions below.

1) How we obtain the (contextual) candidate embeddings. We keep the context fixed, swap the candidate word into the position, and re-encode with the frozen encoder. So the candidate embeddings are real contextual embeddings. The candidates are the decoder's top-k tokens, and in the invalid analysis we use k = 4, while in distillation we use k = 256. This is now stated in Sec. 3.2.

2) Thresholds for the invalid embeddings. The exact thresholds 0.91 nn and 0.6 are indeed not special, which we show by sweeping them on 512 generated sequences per model (ELF-B, sc = 3, NFE 32, k = 4). T5Gemma-2 has a higher invalid rate than the student at every threshold, as in App. D.3.

invalid among uncertain positions (%)0.7 nn0.80.911.01.2
T5Gemma-289.378.163.752.735.6
Student76.757.336.624.39.5

The gap is 21–29% for any threshold between 0.8 and 1.2 nn, so 1 nn would also serve well (52.7% vs. 24.3%) as the threshold. We picked 0.91 nn as it divides the two modes of the teacher's distances (Fig. 9), but the analysis does not rely on a specific value, since T5Gemma-2 generates more invalid embeddings than the student at every threshold. The same holds for the uncertainty: with top-1 < 0.5 / 0.6 / 0.7, the invalid rates are 70% / 64% / 60% for T5Gemma-2 vs. 41% / 37% / 18% for the student.

On the shared normalization question, we standardize each space by its own global mean and std for diffusion training (as in ELF). We then measure the distances in that standardized space, where nn is 13.5 for the teacher and 12.4 for the student, so the two units differ by only 9%. When using one common threshold for both spaces (12.3, the teacher's 0.91 nn), the invalid rates are 63.7% for T5Gemma-2 vs. 25.3% for the student.

3) How we plot Fig. 1, 2, 10, and 11. They are all Pareto frontiers and give the lowest Gen. PPL at each entropy over a set of sampling settings. Fig. 1, 2, and 11 use the same grid (self-conditioning scale {1, 2, 3} × NFE {64, 128, 256, 512}, App. B.3). Fig. 8 shows the individual curves of this grid, and the frontier is taken by their lower envelope. Fig. 10 uses a wider set of settings, which are now listed in its caption. We have updated the captions to make this clearer.

4) Sample sizes and the medium-scale improvement. We use the 256 samples only as a quick preview to pick the operating points, and the main evaluations (Tables 1 and 2) use 3 seeds × 1024 samples. We use 64 sequences for the (token-level) embedding analysis, so this already covers 64 × 1024 tokens per model. We have rerun it with 512 sequences per model and the gap between the two models remains: 63.7% (T5Gemma-2) vs. 36.6% (student) of the uncertain positions are invalid, with more results in App. D.3.

We agree that the ablation curves (Figs. 10, 11) are single-seed, but the std from different seeds is normally 0.1–0.5 PPL (Table 1), far below the gaps in the ablations (~10 PPL). The improvement at medium scale is smaller because the model is getting close to the real-text PPL (15.4), and going from 19.3 to 17.8 at real-text entropy shrinks the distance to real text from 3.9 to 2.4, which is about 40% improvement.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

5) Diversity across samples. We compute distinct-n at the operating points of Table 1 (1024 samples; App. D.6):

distinct-n234
Real text0.4060.7450.884
T5Gemma-2 + ELF-B / ELF-M0.321 / 0.3180.675 / 0.6620.858 / 0.842
Student + ELF-B / ELF-M0.300 / 0.3060.654 / 0.6530.841 / 0.832
Student + ELF-B / ELF-M (same sampling configs as row 2)0.322 / 0.3180.682 / 0.6710.860 / 0.847

All models are below the n-gram diversity of real text. At the operating points of Table 1, the student is 0.01–0.02 below T5Gemma-2, because its operating points use more sampling steps, which trades diversity for fluency. When sampled at T5Gemma-2's settings (last row), the student is slightly higher. The student also has a higher MAUVE in Table 1, which represents diversity across samples as well.

6) Use of the term "scaling". We thank the reviewer for the catch, and we agree that "scaling" is not the precise word here. What we show, as the reviewer points out, is that stronger embedding models of the same line (T5 → T5Gemma-1 → T5Gemma-2) give a better latent space. These models are pretrained with more data and better recipes, not only larger in size. We now state this explicitly in the introduction, and have replaced "scaling" with "advancing" in the title and the text. The title on OpenReview will be updated after the discussion period, following the author guidelines. We would also appreciate the reviewer's suggestion on better wording.

As suggested, we also trained ELF-B on T5Gemma-2-1B in the same setting as Fig. 2 (with the Pareto front curves in Fig.12 of App. D.2). Here we standardize each embedding dimension separately, as one outlier dimension alone holds 27% of the variance of the 1B embeddings. The 1B embeddings show good potential, with a Gen. PPL of 13.7–30.0, lower than T5Gemma-2-270M at every sampling setting (18.1–33.3), and a slightly lower entropy (4.64–4.92 vs. 4.80–5.02); however, they do not clearly beat the 270M yet. We think a larger encoder needs a tuned recipe and a strong denoiser to fully realize its potential, which we leave for future work.

We thank the reviewer again for the suggestions, and are happy to follow up on any remaining points.

I thank the authors for the clarifications and additional experiments. The rebuttal addressed my main concerns and I will raise my score accordingly.

Thank you for the reply and for raising the score. We are truly encouraged and sincerely appreciate your suggestions, which made the paper clearer.

Reviewer_9tzGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the embedding space affects continuous diffusion language models while keeping the ELF diffusion framework fixed. It finds that T5Gemma-2 embeddings improve generation, but argues that denoising can still produce invalid embeddings between separated, plausible token representations. To address this, the authors distill the encoder by matching the teacher’s soft output distributions through a frozen decoder, aiming to make the embedding space easier to generate while preserving decodability. Experiments on OWT and LM1B show improved generation, with objective ablations supporting the importance of soft targets and embedding diagnostics showing fewer invalid generated representations.

Critical strengths

The method is simple and produces clear improvements on OWT and LM1B while keeping the diffusion architecture and training recipe fixed. This helps isolate the effect of the embedding space and makes the teacher–student comparison convincing. The objective ablations strengthen the empirical case by showing the benefit of soft targets over hard-label training and embedding matching. The gains on LM1B using an encoder distilled on OWT also suggest that the improvements extend beyond the distillation dataset.

Critical weaknesses

1.) The evaluation focuses on unconditional generation. A task-based experiment, such as training on TinyGSM and evaluating on GSM8K, would help establish whether the improvements also transfer to conditional generation.

2.) The embedding analysis provides a plausible explanation for the gains. A more direct comparison of distances between plausible alternatives before and after distillation would further strengthen this interpretation.

Questions for authors

1.) Can the authors compare the original and distilled embeddings in the TinyGSM training / GSM8K evaluation setting, reporting accuracy at matched sampling budgets?

2.) Can the authors quantify how distances between plausible alternative embeddings change after distillation?

3.) Could the authors elaborate on the quality–cost trade-off behind choosing the 9-layer student? Have they tested whether the stronger results from deeper students also carry over to the main experimental setting?

3 comments

We thank the reviewer for the thoughtful and in-depth review. We address the weaknesses and questions below.

1) TinyGSM results. We follow the standard protocol, train on TinyGSM and evaluate on the GSM8K test set by executing the generated Python code. We train ELF-B on TinyGSM with the teacher and student encoders, following the recipe of the official ELF code (we train both models for 144k steps, about 6 of the recipe's 11 epochs, due to limited time). Since the student in the paper is distilled on OWT only, we distill a new 9-layer student on TinyGSM text in the same way. We see that (1) T5Gemma-2-270M continues to be a good encoder for the reasoning task, and (2) the distilled student further improves over it at matched sampling budgets; we think the results can be even further improved, as the setting is new and has not been tuned yet.

GSM8K acc. (%, mean of 3 seeds)NFE 32641282565121024
T5Gemma-2 + ELF-B23.628.031.533.535.435.8
Student + ELF-B25.429.133.634.036.036.3
Held-out TinyGSM acc. (%, mean of 3 seeds)NFE 32641282565121024
T5Gemma-2 + ELF-B41.344.746.950.153.353.1
Student + ELF-B41.845.947.150.254.153.8

The held-out TinyGSM set is 500 problems left out of training. NFE 1024 is the sampler setting of the official ELF code. For reference, in this setting ELF reports GSM8K accuracies of 33.0% for MDLM, 36.0% for Duo, and 40.8% for T5-small + ELF-B trained for the full 11 epochs.

2) Direct comparison of how the candidates change under distillation. We take 256 held-out real sequences, obtain the contextual embeddings of the top-4 candidates at each position (as in Sec. 3.2). We then measure the distance from the input word's embedding to each of its three alternatives, in nn units of each space. This contains 247,483 positions and is in App. D.4.

alternatives on real textT5Gemma-2Student
mean distance (nn)1.451.35
alternatives farther than 1 nn (%)70.866.7
alternatives farther than 2 nn (%)20.713.8

On clean real text, distillation moves the alternatives moderately, but it indeed pulls together the far ones. For instance, the percentage of alternatives beyond 2 nn drops from 20.7% to 13.8% (row 3). The effect is much larger on the generated embeddings, which are never exact, as we showed in Fig. 4 and 6.

3) Quality–cost trade-off. We started with a common setup in distillation, using a 9-layer student initialized from every second layer of the teacher and then trained with the KL loss. This saves encoding compute in diffusion training (Table 3, student takes 2.1 ms per sequence vs. 3.4 ms for the teacher). Later we found that a deeper student indeed diffuses even better (Fig. 11), but the 9-layer student already surpasses the teacher, so we keep it as the main setting and report the depth ablation in the appendix.

We thank the reviewer again for the suggestions, and are happy to follow up on any remaining points.

Thank you for the additional experiments and clarifications. The TinyGSM/GSM8K results address my main concern about conditional generation. The gains are modest but consistent across sampling budgets. Reporting the variability across seeds would help assess their significance.

The distance analysis strengthens the proposed interpretation, and the encoding-cost comparison clarifies the choice of the 9-layer student. Whether deeper students improve the main experimental setting remains open, but I do not consider this essential for acceptance.

Overall, the response strengthens my support for acceptance. I will keep my score at 3 (weak acceptance).

Thank you for the engagement and support for the paper. Your comments helped us strengthen the paper, and we sincerely appreciate them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive reviews. We are encouraged to see that the reviewers agree that (1) the paper studies an important problem (Reviewer 6ekf) on the latents of continuous diffusion language models (DLMs), and (2) our method is clever (Reviewer uro6), simple, and effective (Reviewers 9tzG, 6ekf). We address the shared questions here, and mark all changes in blue in the revised PDF:

  1. A clearer definition of 'invalid' embeddings. The candidates at a position are the top-k words predicted by the decoding head (for the analysis we use k = 4). To get the embedding of a candidate, we keep the context fixed, swap the candidate word into that position, and re-encode the sentence. So the candidate embeddings are real contextual embeddings. We call a generated embedding invalid if it is farther than 0.91 nn from every candidate, where nn is the median nearest-neighbor distance between real embeddings. This definition is now in Sec. 3.2, and we also conduct a threshold sweep suggested by Reviewer 6ekf (App. D.3) which shows that the analysis does not depend on an exact threshold.

  2. Scope of the paper. We focus on T5Gemma-1 and T5Gemma-2 since they follow the original encoder-decoder text embedding models and serve as good plug-in encoders for continuous latent diffusion language models, and with them we test the limit of this route. We have added a discussion of the other continuous paradigms (FMLM, Langflow, Categorical Flow Maps, etc.) to Sec. 2.

  3. New experiments are added as App. D of the revised PDF, including a TinyGSM experiment (D.1; 9tzG), T5Gemma-2-1B + ELF-B (D.2; 6ekf), a threshold sweep of the invalid-embedding analysis on 512 sequences (D.3; 6ekf), how the distances between alternatives change under distillation (D.4; 9tzG, uro6), the correlation between embedding error and generation quality (D.5; uro6), n-gram diversity (D.6; 6ekf), and quantitative results for round-trip denoising (D.7; uro6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.