- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method called LDLM-s, which is a latent diffusion language model where the latent representation of the context is jointly trained with diffusion model without relying on any pretrained language model to get the latent representation. Next-token prediction objective is added as supervision in addition to the diffusion model training objective. Their results show that compared to LDLM, LDLM-s achieves better diversity while using 5x fewer training tokens, achieves a better perplexity entropy frontier and have less repetition.
Critical strengths
- The research question itself is interesting, it first tries to explore the possibility of learning a latent space without utilizing an existing language model. And the results show that this is doable.
- The novelty of the approach compared to LDLM is the introduction of the additional next-token prediction loss from the latent while LDLM is to use the clean latent from GPT-2 for the autoencoder setup. The next-token prediction loss enforces the latent to contain semantic meanings.
- Ablation studies show the role of different ingredients of the method: NTP, Latentnorm and selective diffusion-gradient routing are all important for training.
Critical weaknesses
- The paper does not provide a MAUVE comparison between LDLM-s and the competing diffusion models on the main OWT-1024 benchmark, which should characterize the distribution quality of the generated text.
- The authors point out that LDLM-s only need 5x fewer training tokens compared to LDLM. It would be more helpful to understand the training cost if the authors can provide some metrics from training FLOPs, GPU-hours and clarify its computational efficiency.
Questions for authors
- Could the authors provide some samples of the generated text beyond the qualitative analysis?
4 comments
Gentle follow-up
We again express our gratitude for your valuable review and the assessement of our work. In light of upcoming final of discussion period, we would like to know if our detailed responce addressed your concerns. We are eager to clarify any remaining questions. If your are satisfied with the provided answers, we would kindly ask you to consider them in your final assessment.
Thanks the author for the additional experimental results. The MAUVE comparison provides useful insights of the generation quality of LDLM and the latency numbers also clarify the efficiency of your approach. Overall my main concerns have been addressed and I would encourage the authors to integrate these results into a later version.
Dear Reviewer,
We are happy that our responce addressed your concerns. We have revised our manuscript in accordance with new results. We have integrated them in detail in Appendix B (Compute) and Appendix F (Mauve) and Appendix J (Generation Samples) along with other discussion results.
Rebuttal
We are grateful for your acknowledgement of the novelty of our work, and your positive assesment of our analysis. We are eager to answer your questions.
For this comparison we did not use SC-CFG. It is hard to rank models' performance based on MAUVE alone, since it has a significant variance depending on the number of steps and depending on the generation random seed. We report the mean values and std of 5 generations of 1000 texts.
These results show that generally our model generates texts close to validation texts distribution. We will add these result to the final manuscript.
Here we provide complete compute requirements both for our model and for baselines
For latent diffusion, we do not estimate the compute required for the pretrained models that induce the latent space. However, eliminating the need for such is one of our main motivations. Considering this, we find it relevant to account for these compute requirements.
Since the exact training compute of GPT2-small is not reported, we set it as reasonable 120B tokens and estimate the required FLOPs based on this assumptions.
With these considerations, our method requires the least amount of training compute among all of the baselines. But even if we put them aside, the only less compute-extensive model is ELF which we outperform. Comparing to LDLM, the method is almost 6× efficient in terms of FLOPs.
In our experiments, we trained our main model for around 5 days on a node of 8 A100 GPUs or around 900 GPU-hours. LDLM reports around 5 days on 64 A100 estimating around 7700 GPU-hours.
We additionally measure the full generation latency:
We compare with LDLM on the same setup: 1 A100 GPU, 1024 tokens per sample, BF16 model weights, batch size of 16. Since our sampling algorithm and and the backbone architecture inherit LDLM, the latency for 1 NFE is equal, but since we achieve out best results with less sampling steps, it brings the speedup of up to 4×.
Of course, we will add generation samples of our model
We hope that our answer covers all of your concerns. We thank you again for your involvement in the review of our paper and the assesement of its contribution.