acceptodds
Under review as a conference paper at ICLR 2027

Training Latent Language Diffusion Models Requires Less Than You Think

Abstract

Contextual latent diffusion offers a way to generate language in a representation space that captures linguistic dependencies beyond token identity, but existing approaches typically rely on pretrained language features or a separate representation-learning stage. We introduce LDLM-s: Scratch Latent Language Diffusion Model, to our knowledge, the first latent language diffusion framework to achieve strong generation while jointly learning its contextual representation space and generator from random initialization, without pretrained language components or a separate representation-pretraining stage. LDLM-s uses a language-modeling autoencoder to provide linguistic supervision to the latent space, while the diffusion objective simultaneously shapes the encoder, allowing the representation and denoiser to co-adapt throughout training. On OpenWebText, LDLM-s outperforms most of the recent continuous and discrete diffusion baselines on the generative-perplexity–token-entropy tradeoff. It also retains substantially more repetition-clean generations in the low-perplexity regime than the competing continuous baselines. Relative to LDLM, LDLM-s extends the tradeoff toward lower generative perplexity, while LDLM reaches higher-diversity operating points, despite LDLM-s seeing roughly fewer training tokens. More broadly, our results show that strong contextual latent diffusion can be learned without external language representations, providing a step toward self-contained latent language generation directly from text.

Primary area:
generative models
Keywords:
diffusion language models, latent diffusion, language modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FE1kOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method called LDLM-s, which is a latent diffusion language model where the latent representation of the context is jointly trained with diffusion model without relying on any pretrained language model to get the latent representation. Next-token prediction objective is added as supervision in addition to the diffusion model training objective. Their results show that compared to LDLM, LDLM-s achieves better diversity while using 5x fewer training tokens, achieves a better perplexity entropy frontier and have less repetition.

Critical strengths
  1. The research question itself is interesting, it first tries to explore the possibility of learning a latent space without utilizing an existing language model. And the results show that this is doable.
  2. The novelty of the approach compared to LDLM is the introduction of the additional next-token prediction loss from the latent while LDLM is to use the clean latent from GPT-2 for the autoencoder setup. The next-token prediction loss enforces the latent to contain semantic meanings.
  3. Ablation studies show the role of different ingredients of the method: NTP, Latentnorm and selective diffusion-gradient routing are all important for training.
Critical weaknesses
  1. The paper does not provide a MAUVE comparison between LDLM-s and the competing diffusion models on the main OWT-1024 benchmark, which should characterize the distribution quality of the generated text.
  2. The authors point out that LDLM-s only need 5x fewer training tokens compared to LDLM. It would be more helpful to understand the training cost if the authors can provide some metrics from training FLOPs, GPU-hours and clarify its computational efficiency.
Questions for authors
  1. Could the authors provide some samples of the generated text beyond the qualitative analysis?
4 comments

We are grateful for your acknowledgement of the novelty of our work, and your positive assesment of our analysis. We are eager to answer your questions.

W1. MAUVE comparison with the baselines.

Method64 Steps128 Steps256 Steps512 Steps1024 Steps
LDLM0.893±0.0060.893_{\pm 0.006}0.911±0.0200.911_{\pm 0.020}0.854±0.0170.854_{\pm 0.017}0.847±0.0180.847_{\pm 0.018}0.844±0.0250.844_{\pm 0.025}
CoBit0.179±0.0340.179_{\pm 0.034}0.511±0.0800.511_{\pm 0.080}0.516±0.0170.516_{\pm 0.017}0.313±0.0240.313_{\pm 0.024}0.186±0.0240.186_{\pm 0.024}
ELF0.883±0.0190.883_{\pm 0.019}0.838±0.0400.838_{\pm 0.040}0.743±0.0380.743_{\pm 0.038}0.621±0.0060.621_{\pm 0.006}0.526±0.0170.526_{\pm 0.017}
LangFlow0.889±0.0310.889_{\pm 0.031}0.906±0.0220.906_{\pm 0.022}0.923±0.0280.923_{\pm 0.028}0.932±0.0190.932_{\pm 0.019}0.923±0.0220.923_{\pm 0.022}
MDLM0.746±0.0380.746_{\pm 0.038}0.820±0.0410.820_{\pm 0.041}0.797±0.0170.797_{\pm 0.017}0.859±0.0120.859_{\pm 0.012}0.860±0.0430.860_{\pm 0.043}
Duo0.867±0.0210.867_{\pm 0.021}0.867±0.0350.867_{\pm 0.035}0.887±0.0310.887_{\pm 0.031}0.888±0.0370.888_{\pm 0.037}0.895±0.0150.895_{\pm 0.015}
LDLM-s\textbf{LDLM-s}0.758±0.0140.758_{\pm 0.014}0.851±0.0070.851_{\pm 0.007}0.897±0.0230.897_{\pm 0.023}0.863±0.0400.863_{\pm 0.040}0.837±0.0000.837_{\pm 0.000}

For this comparison we did not use SC-CFG. It is hard to rank models' performance based on MAUVE alone, since it has a significant variance depending on the number of steps and depending on the generation random seed. We report the mean values and std of 5 generations of 1000 texts.

These results show that generally our model generates texts close to validation texts distribution. We will add these result to the final manuscript.


W2. Compute requirements.

Here we provide complete compute requirements both for our model and for baselines

Model (training updates)Trainable parameters PPProcessed tokens DDTokens / parameter D/PD/PEstimated training FLOPs (101810^{18})
LDLM-s324M102B315207.85
AR (from MDLM paper)162M261B1,614223.74
MDLM169M524B3,091448.00
Duo169M524B3,091488.47
LDLM~298M524B~1,759~1167.12
ELF-B (5 epochs)~105M45.2B~430~47.34-55.03
CoBit (750k)~134M393B~2,940~343.95
LangFlow (1M)~171M524B~3,070~631.00

For latent diffusion, we do not estimate the compute required for the pretrained models that induce the latent space. However, eliminating the need for such is one of our main motivations. Considering this, we find it relevant to account for these compute requirements.

Model (pretraining setup)Parameters PPProcessed tokens DDTokens / parameter D/PD/PEstimated pretraining FLOPs (101810^{18})
GPT-2-small (LDLM)~124M~120B~964~102.53
T5-small (ELF; 512/114)~60M1,049B~17,330~217.61

Since the exact training compute of GPT2-small is not reported, we set it as reasonable 120B tokens and estimate the required FLOPs based on this assumptions.

With these considerations, our method requires the least amount of training compute among all of the baselines. But even if we put them aside, the only less compute-extensive model is ELF which we outperform. Comparing to LDLM, the method is almost 6×6\times efficient in terms of FLOPs.

In our experiments, we trained our main model for around 5 days on a node of 8 A100 GPUs or around 900 GPU-hours. LDLM reports around 5 days on 64 A100 estimating around 7700 GPU-hours.

We additionally measure the full generation latency:

ModelSampling stepsDenoiser NFEBatch latency (s)Throughput (tokens/s)
LDLM (reported)10241024~40.00~409.60
LDLM-s + sc-CFG12825610.191,607.80
LDLM-s + sc-CFG25651220.18811.90
LDLM-s + sc-CFG512102440.17407.86

We compare with LDLM on the same setup: 1 A100 GPU, 1024 tokens per sample, BF16 model weights, batch size of 16. Since our sampling algorithm and and the backbone architecture inherit LDLM, the latency for 1 NFE is equal, but since we achieve out best results with less sampling steps, it brings the speedup of up to 4×4\times.


Q1. Generation Samples

Of course, we will add generation samples of our model


We hope that our answer covers all of your concerns. We thank you again for your involvement in the review of our paper and the assesement of its contribution.

AuthorsOpenReview ↗

Gentle follow-up

We again express our gratitude for your valuable review and the assessement of our work. In light of upcoming final of discussion period, we would like to know if our detailed responce addressed your concerns. We are eager to clarify any remaining questions. If your are satisfied with the provided answers, we would kindly ask you to consider them in your final assessment.

Thanks the author for the additional experimental results. The MAUVE comparison provides useful insights of the generation quality of LDLM and the latency numbers also clarify the efficiency of your approach. Overall my main concerns have been addressed and I would encourage the authors to integrate these results into a later version.

Dear Reviewer,

We are happy that our responce addressed your concerns. We have revised our manuscript in accordance with new results. We have integrated them in detail in Appendix B (Compute) and Appendix F (Mauve) and Appendix J (Generation Samples) along with other discussion results.

Reviewer_AfNKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LDLM-s— a diffusion language model architecture that extends latent diffusion language models (LDLM, ICML 2026). LDLM first converts tokens to continuous embeddings using the representations of a pretrained GPT2 model, then it trains a latent diffusion model and is then decoded. LDLM-s removes the need for the GPT2 model, replacing LDLM’s frozen GPT-2 with a next-token prediction objective, and replacing the warmup with LatentNorm plus a stop-gradient on the diffusion target. On OpenWebText it reaches lower generative perplexity than LDLM with about 5× fewer training tokens.

Critical strengths
  • The question is clear and worth asking: can contextual latents be learned without a pretrained language model? Experiments from th LDLM paper made this look hard, since diffusing over embeddings trained from scratch gave bad perplexity
  • The fact that comparable results with 5x fewer tokens is a strong positive result here and is interesting. The reduced diversity is an interesting result also
  • The recipe is in some respects simpler than LDLM: no warmup and less sensitivity to decoder noise
  • The use of next token prediction simultaneous to the denoising objective to create good quality representations seems original
Critical weaknesses
  • Some claims aren’t ablated. LatentNorm is credited with removing the need for warmup, yet no experiment removes LatentNorm or adds warmup back. The same goes for self-conditioning guidance, which is used for all LDLM-s results but never ablated, so it’s unclear how much of the improvement over LDLM comes from it
  • The paper’s title advertises a simplification of LDLM, though the training objective is quite complex and a small language model is effectively being learned within it.
  • The ablations are weak statistically. Each configuration is one trained model, and the error bars come only from generation seeds
Questions for authors
  • How good is the internal LM decoder as a language model on its own? Did you just train a small language model or is it actually biased towards the denoising task
  • What happens if LatentNorm is removed, or if LDLM’s warmup is added back? Does LatentNorm actually replace the warmup?
  • Does the advantage over LDLM persist or grow with larger models?
3 comments

We appreciate your acknowledgement of our paper strengths, such as the relevance of the posed research question, the novelty of the idea and the training efficiency in respect to tokens budget. We will answer to your questions below.

W1, Q2. On the ablation of the LatentNorm and the diffusion-to-encoder warmup.

We conducted your proposed experiments and provide them below.

MethodEntropy ↑Gen. PPL ↓MAUVE ↑Div
Baseline4.19±0.0094.19_{±0.009}73.8±0.673.8_ {± 0.6}0.80±0.020.80_{±0.02}0.48±0.0090.48_{±0.009}
Learnable LatentNorm4.278±0.0044.278_{±0.004}1011.5±16.81011.5_{±16.8}0.0060±0.00030.0060_{±0.0003}0.468±0.0030.468_{±0.003}
Diffusion-to-encoder warmup (50k)4.202±0.0034.202_{±0.003}78.6±0.978.6_{±0.9}0.718±0.0150.718_{±0.015}0.480±0.0030.480_{±0.003}

These results indeed support the role of the fixed LatentNorm. When it is learnable, the model collapses, performing extremely poorly. On the other hand, the addition of the Diffusion-to-Encoder warmup over the baseline setup does not yield any gains. If anything, it slightly degrades the quality.

We thank you for raising this question, since these experiments further clarify the role of fixed LatentNorm.


W1, Q2. The ablation of Self-Conditioning Classifier Free Guidance.

Our model slightly outperforms LDLM in generative perplexity with matched compute even without SC-CFG. However, it significantly helps to extend the frontier towards a lower perplexity end.

Method64 Steps128 Steps256 Steps512 Steps1024 Steps
LDLM56.73±0.3356.73_{\pm 0.33}43.24±0.1443.24_{\pm 0.14}35.09±0.1935.09_{\pm 0.19}31.37±0.1531.37_{\pm 0.15}28.59±0.1228.59_{\pm 0.12}
LDLM-s52.85±0.1852.85_{\pm 0.18}42.07±0.0642.07_{\pm 0.06}34.81±0.1634.81_{\pm 0.16}29.68±0.2329.68_{\pm 0.23}26.26±0.0026.26_{\pm 0.00}
Guidance Scale64 Steps128 Steps256 Steps512 Steps1024 Steps
1.0052.85±0.1852.85_{\pm 0.18}42.07±0.0642.07_{\pm 0.06}34.81±0.1634.81_{\pm 0.16}29.68±0.2329.68_{\pm 0.23}26.26±0.0026.26_{\pm 0.00}
1.2538.83±0.1338.83_{\pm 0.13}31.05±0.1031.05_{\pm 0.10}25.63±0.1125.63_{\pm 0.11}22.06±0.0422.06_{\pm 0.04}—
1.5031.13±0.0731.13_{\pm 0.07}25.72±0.0425.72_{\pm 0.04}21.26±0.0521.26_{\pm 0.05}18.14±0.0218.14_{\pm 0.02}—
1.7527.19±0.0927.19_{\pm 0.09}23.02±0.1723.02_{\pm 0.17}19.12±0.0219.12_{\pm 0.02}16.07±0.0316.07_{\pm 0.03}—
2.0025.10±0.0425.10_{\pm 0.04}21.68±0.0321.68_{\pm 0.03}17.99±0.0617.99_{\pm 0.06}15.16±0.0715.16_{\pm 0.07}—

Larger guidance scale also allows for reaching lower generative perplexity points with significantly less steps. We hypothesize that since it strengthens the self-conditioning impact, it essentially simulates more generation steps with self-conditioning. This allows to reach simillar generative perplexity of around 26 points with 4×4\times NFEs.

Q2. On the simplification of the framework.

We agree that the training objective itself is not simplistic. However, as you fairly mention, the simplification comes with relieving the burden of excessive hyperparameters tuning. Both the decoder noise sensitivity and the need of diffusion-to-encoder warmup are eliminated in our method. And, of course, there is no need for pretrained encoder.

The decoder itself can be used as a small AR-model, however its performance is far from remarkable and does not hold significan value. That said, it is most certainly very oriented for diffusion generation. This is supported by our additional ablation experiment of two-staged training. We first trained just the NTP autoencoder, and then trained a diffusion model over these representations.

MethodEntropy ↑Gen. PPL ↓MAUVE ↑Div
Baseline4.19±0.0094.19_{±0.009}73.8±0.673.8_ {± 0.6}0.80±0.020.80_{±0.02}0.48±0.0090.48_{±0.009}
Two-staged training4.24±0.0034.24_{±0.003}101.5±1.34101.5_ {± 1.34}0.69±0.020.69_{±0.02}0.543±0.0020.543_{±0.002}

The resulting performance clearly shows the autoencoder adaptation to diffusion training when trained jointly. This confirms the bias towards the denoising task.


Q3. On the performance with larger models.

To confidently establish the advantage in generative perplexity over LDLM with larger models, we would need to train both larger LDLM and LDLM-s. However, even training our base model on 1024 token length context would require around 5 days on 8 A100 node. With increased model size, it would take even longer.

However, we started the training of a model with diffusion backbone of around 300M parameters. We will add these results in the final draft, and will gladly present them on the rebuttal if the model converges on time.


W4. On multiple training seeds.

Unfortunately, as much as we would want it, training a diffusion language model model, even a small one, requires substantial amount time. Hence there is no common practice of training on multiple random seeds in the relevant literature. To our knowledge, all of the baseline models only reported generations over multiple seeds, not retraining.

Due to relatively short rebuttal period, we would not retrain all of the ablations from a number of random seeds in time.


We again thank you for the efforts in reviewing our work. We found your notes helpful in clarifing our paper's contribution. As we answered your questions, we hope that all of your concerns are satisfied.

AuthorsOpenReview ↗

Gentle follow-up

As the discussion period is ending shortly, we again thank you for your thoughtful and constructive review.

We also would like to provide additional results on scaling our model. We have trained a medium-sized (300M backbone, same 160M autoencoder) model for 150k training updates on OWT keeping all other variables the same. The scaled up model significantly improves upon its base (130M backbone) variant. Here are the numbers (PPL / Entropy):

Modelw=0.75w=1w=1.25w=1.5
Base DiT59.1 / 5.4342.07 / 5.3531.1 / 5.2925.7 / 5.26
DiT-M (300M)49.1 / 5.4333.16 / 5.3725.1 / 5.3121.2 / 5.29

The larger model, despite being trained less, exhibits higher entropy and lower perplexity under the same sampling settings. We find this results to support the scalability of our approach.

We hope that our detailed responce addressed your concerns. We would respectfully ask you to consider it in your final assessment.

Reviewer_akHwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LDLM-s, a continuous latent diffusion language model that jointly learns its contextual representations and diffusion generator from random initialization, without pretrained language-model weights. Building on LDLM, the authors introduce next-token prediction supervision and investigate/incorporate architectural and objective choices such as a detached reconstruction loss, latent normalization, and selective gradient propagation through the diffusion objective. These modifications enable joint training without the warmup stage required by LDLM.

The authors evaluate unconditional generation on OpenWebText using generative perplexity, token entropy, and repetition metrics. They report competitive quality–diversity tradeoffs using approximately 102B training tokens, roughly 5× fewer than LDLM. Ablations investigate the contributions of NTP, gradient routing, and decoder noise.

Critical strengths

This paper proposes LDLM-s, a continuous latent diffusion language model that jointly learns its contextual representations and diffusion generator from random initialization, without pretrained language-model weights. Building on LDLM, the authors introduce next-token prediction (NTP) supervision, detached reconstruction loss, latent normalization, and selective gradient propagation through the diffusion objective. These modifications enable joint training without the warmup stage required by LDLM.

The authors evaluate unconditional generation on OpenWebText using generative perplexity, token entropy, and repetition metrics. They report competitive quality–diversity tradeoffs using approximately 102B training tokens, roughly 5× fewer than LDLM. Ablations investigate the contributions of NTP, gradient routing, and decoder noise.

Critical weaknesses
  1. The reported generative-perplexity improvements come at the expense of sample entropy. My main concern is Figure 3. OpenWebText has entropy 5.436, whereas LDLM-s reaches only approximately 5.35, with its lowest-perplexity operating points having substantially lower entropy. At comparable entropy near 5.35, the original LDLM also appears to achieve better generative perplexity. Consequently, I do not find the claimed improvement in the quality–diversity frontier convincing in the regime closest to the data distribution. The repetition analysis is useful, but in my mind a low four-gram repetition alone does not establish that the model preserves the diversity of real text.

  2. The central training contribution and efficiency claims are insufficiently isolated. Much of the architecture and diffusion machinery are inherited from LDLM, while CoLa-DLM has previously explored joint training from scratch, so the main contribution of this work is a training recipe. For a training recipe work, I would like questions around architectural efficiency to be more of a central focus and relevant empirics to be explored. For example, the paper does not compare joint training against separately pretraining the same encoder with NTP before diffusion training under matched compute. Furthermore, the reported reductions in training tokens and denoiser evaluations do not directly establish improvements in training FLOPs or wall-clock inference efficiency. These omissions make it difficult to assess the substantive advantage of the proposed method. The authors should cite and better position their contributions relative to SEDD (Lou et al., ICML 2024), BD3LM (Arriola et al., ICLR 2025), and Scaling Up Masked Diffusion Models on Text (Nie et al., ICLR 2025), particularly when discussing training efficiency and scaling. The original MAUVE paper (Pillutla et al., NeurIPS 2021) should also be cited when introducing that evaluation metric.

Questions for authors
  1. Generation quality at realistic entropy: Can the authors report their strongest achievable generative perplexity near the OpenWebText entropy of 5.436 and compare against LDLM and other baselines at matched entropy? If LDLM-s cannot reach this regime, the authors should explicitly acknowledge this limitation. Please also clarify which language model is used to compute generative perplexity.

  2. Value of joint training: Under matched total training FLOPs, how does the proposed simultaneous training compare with training the same NTP encoder first, followed by latent diffusion training? This would help isolate the contribution of joint optimization from the benefit of NTP-based representation learning.

  3. Training and inference efficiency: Could the authors report total training FLOPs and measured end-to-end generation latency, including guidance and decoding costs? I computed that LDLM-s processes approximately 315 tokens per trainable parameter, but the paper does not justify this training budget. At a minimum, the authors should report the tokens-per-parameter ratios for relevant architectures at comparable model sizes (e.g., LDLM, MDLM, Duo, and autoregressive baselines) and discuss how their chosen ratio compares. Ideally, they could provide validation performance as a function of training compute.

3 comments

Thank your for acknowledging the clarity of our writing and the informativeness of our work. We will gladly address your concerns further.


W1, Q1. On the performance in higher entropy regime


Generations with the diversity of around dataset level are indeed important to report. We provide the comparison at around-dataset level of entropy below:

MethodEntropy ↑Gen. PPL ↓
LDLM5.455.4535.09±0.1935.09_{ ±0.19}
CoBit5.405.4067.14±1.0167.14_ {± 1.01}
LangFlow5.425.4259.16±0.2859.16_{± 0.28}
Ours5.435.4359.10±0.0559.10_{±0.05}

Our added numbers show very competetive performance with continuous baselines at higher entropy levels. We did not include ELF since it does not reach high entropy with the sampling setting reported in the original paper. With these results in mind, we agree to weaken our claim: our method outperforms most continuous baselines in the lower entropy regime, while performs on par in the higher-entropy zone. We highlight that we did not claim to outperform LDLM: we rather establish a compromise of reaching lower generative perplexity operating points trading off the entropy

We use GPT2-Large to compute generative perplexity.


W2, Q2. On the value of joint training.


In accordance with your request, we trained a two-staged pipeline on OWT-128 to fully complete the ablation. We trained the NTP-autoencoder for 100k training updates, then the diffusion model over these frozen representations for 200k training updates to ensure the proper convergence of this setup. Total number of training FLOPS of the two-staged setup is around 17∗101817 * 10^{18} FLOPs, while the baseline e2e setup takes around 12∗101812 * 10^{18}. We provide the results below.

MethodEntropy ↑Gen. PPL ↓MAUVE ↑Div
Baseline4.19±0.0094.19_{±0.009}73.8±0.673.8_ {± 0.6}0.80±0.020.80_{±0.02}0.48±0.0090.48_{±0.009}
Two-staged training4.24±0.0034.24_{±0.003}101.5±1.34101.5_ {± 1.34}0.69±0.020.69_{±0.02}0.543±0.0020.543_{±0.002}

These results strongly support that the diffusion objective gradient to autoencoder significantly improves the quality of the resulting model, isolating the contribution of joint training. Thank you for proposing this experiment since it indeed solidifies our findings. We will add it to the main ablation.


W2, Q3. On the training and inference efficiency.


We would like to accentuate that the main focus of our work was to explore the possibility of training a strong latent diffusion model without the requirement for a pretrained large model. This question has risen in the light of potential scaling, which would be hardly burdened by the need of an exogenous language model. Considering this motivation, we view the need for fewer training tokens as a pleasant side effect.

However, we agree that providing more thorough compute report and comparison would strengthen the contribution.

The table below compares the OpenWebText training recipes. DD counts all processed token positions, including repeated presentations; PP counts unique trainable parameters across the entire model, including embeddings and autoencoder/decoder components, but excluding frozen pretrained parameters. Tokens per parameter are D/PD/P.

Model (training updates)Trainable parameters PPProcessed tokens DDTokens / parameter D/PD/PEstimated training FLOPs (101810^{18})
LDLM-s324M102B315207.85
AR (from MDLM paper)162M261B1,614223.74
MDLM169M524B3,091448.00
Duo169M524B3,091488.47
LDLM~298M524B~1,759~1167.12
ELF-B (5 epochs)~105M45.2B~430~47.34-55.03
CoBit (750k)~134M393B~2,940~343.95
LangFlow (1M)~171M524B~3,070~631.00

FLOPs are our analytical estimates of forward/backward matrix operations, using two FLOPs per multiply-add and dense attention counting, including for causal layers. We account for vocabulary projections, self-conditioning, and Duo's continuous-phase embedding computation; elementwise operations, optimizer updates, recomputation, evaluation, and I/O are excluded.

These results show that not only the method requires less training tokens to achieve strong performance, it also uses almost 6×6\times less FLOPs than LDLM and 2×2\times less FLOPs than the discrete diffusion baselines.

It is worth noting that these calculations only accounted for the diffusion models training itself. But latent diffusions such as LDLM and ELF, which are our main competitors, also require a pretrained language model.

The pretrained-model compute is estimated separately below. For T5-small, we count the full encoder-decoder model used in pretraining.

Model (pretraining setup)Parameters PPProcessed tokens DDTokens / parameter D/PD/PEstimated pretraining FLOPs (101810^{18})
GPT-2-small (LDLM)~124M~120B~964~102.53
T5-small (ELF; 512/114)~60M1,049B~17,330~217.61

Since the exact number of seen tokens for GPT2-small is unknown, we set it as roughly half of AR baseline from the MDLM paper.

If we add the compute required for pretraining these models, the gains of our model become more drastic.

We also would like to justify our token to parameter ratio. As discussed in Section 6, even after 100k training updates our model exhibits very strong and competetive generation, outperforming ELF. It continues to further improve after 100k more updates, however, the model seem to converge at around 200k training updates with further training not yielding significant improvements.

For inference, we compare batch generation latency on one NVIDIA A100 with batch size 16 and sequence length 1024. LDLM reports approximately 40 seconds at 1024 NFEs in Table 3 and Section 8. Our measurements use an A100, BF16 model computation. Each guided sampling step requires two denoiser evaluations.

ModelSampling stepsDenoiser NFEBatch latency (s)Throughput (tokens/s)
LDLM (reported)10241024~40.00~409.60
LDLM-s + sc-CFG12825610.191,607.80
LDLM-s + sc-CFG25651220.18811.90
LDLM-s + sc-CFG512102440.17407.86

The one token decoding pass is included in these latency computations. These results are expected, since both the sampling process and the model backbone are exaclty the same as in LDLM. The only difference lies in the sc-CFG, which requires one additional NFE per step.


W2. On the positioning and contribution.

We are happy to clarify our positioning. As mentioned earlier, our main focus was to explore the fully scratch end-to-end training of a latent language diffusion models. Scaling of the framework is a serious self-contained direction for the future work.

All of the mentioned works (SEDD, BD3LM and Scaling Up Masked Diffusion Models on Text) explore discrete diffusion methods and/or their scaling behaviour.

SEDD explores the influence of inference time compute of their model. BD3LM studies the dependence of the discrete diffusion training objective variance on performance and the means to reduce it. Nie. et al dedicated their work to thorough IsoFLOPs analysis of the scaling behaviour of discrete diffusion. We argue that our work explores quite different research direction.

However, we will add these papers into related works section to represent the state of literature more completely. We, of course, will cite the original MAUVE paper.


We again thank you for the time and effort you put in reviewing our work. We hope that your concerns were answered and sincerely find this discussion contributed positively to our work.

AuthorsOpenReview ↗

Gentle Follow-up

We once again thank you for your informative review and valuable feedback. As discussion period is coming to an end, we would be happy to know if our detailed rebuttal has addressed your concerns. We would be happy to continue the discussion and provide required evidence and clarifications. If you are satisfied with our detailed responce, we would greatly appreciate you considering them in the final decision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank reviewers for their deeply thoughtful reviews, constructive feedback and fruitful discussions. We would like to summarize the results of the rebuttal period. We have also updated the manuscript to incorporate all additional experiments, analyses and clarifications presented during the discussion.

(i) Joint learning of the latent space [akHw, AfNK]. We presented an additional comparison of two-stage training of an autoencoder augmented with next-token prediction and our baseline end-to-end model. Joint training has shown to significantly outperform the separate training regime, despite requiring less training compute, which confirms the critical value of this choice.

(ii) Generation near the dataset entropy [akHw]. LDLM-s performance at near-dataset entropy level has shown competitive results compared to the baselines, losing only to LDLM among the evaluated continuous models. This further supports the strength of the method across different regions of the quality--diversity frontier.

(iii) Scaling the diffusion backbone [AfNK]. We have trained an additional model with a larger 300M DiT backbone. The model performance has significantly improved the frontier, providing encouraging evidence of the method's scalability. We additionally compared it with ELF-M in the manuscript, which has a similarly sized backbone.

(iv) Latent normalization and diffusion-to-encoder warmup [AfNK]. We presented extended ablations on our choice of freezing LatentNorm and removing the diffusion-to-encoder warmup. The results of the experiments supported the role of frozen LatentNorm in stabilizing the method, while adding the warmup back did not bring improvements.

(v) Self-conditioning guidance ablation [AfNK]. Self-conditioning guidance ablation was meant to show to what extent this technique improved our model. Its results confirmed that our method's gains in generative perplexity remain even without guidance, while SC-CFG further improves the low-perplexity operating points.

(vi) Training compute and inference latency [akHw, FE1k]. We provided thorough estimations of training FLOPs and measured end-to-end inference latency. Our method requires roughly 2--6 times fewer training FLOPs than most diffusion baselines. ELF is the only diffusion baseline with a lower standalone training cost, but its estimated total exceeds that of LDLM-s once the cost of T5 pretraining is included. We also showed that our method can achieve competitive generation quality with up to 4 times lower inference latency.

(vii) MAUVE comparison [FE1k]. We provided additional MAUVE numbers comparing our method with the baselines across different sampling steps. They confirm that our method exhibits competitive performance in generating texts close to the data distribution.

All the additional experimental results have been included in the updated manuscript, alongside detailed compute estimations, generation samples, and extended discussions of our findings.

We once again thank all reviewers for their time and effort throughout the discussion. We sincerely hope that our answers and the updated manuscript have addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.