- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a continuous diffusion/flow-based language model called Rectified Language Flow (ReLF). It aims to address several challenges associated with modeling text in continuous space. Continuous diffusion/flow models offer several potential advantages over their discrete counterparts. Most notably, they can more readily adopt techniques developed in the continuous domain, such as ODE/SDE sampling, numerical solvers, and distillation methods, as has been demonstrated in image generation.
However, applying continuous diffusion/flow models to language introduces several challenges. One is token commitment: continuous flows produce continuous-valued vectors, whereas language generation ultimately requires discrete tokens. To address this issue, the authors propose singular logits, which encourage the posterior to converge to a one-hot distribution.
Another important challenge is embedding geometry: what kind of token representations are suitable for continuous flow-based language models? This differs from the representation-learning problem in autoregressive language models. In denoising models, tokens with highly similar embeddings may be more difficult to distinguish, making the denoising task more challenging. To address this, the authors propose Hellinger–SVD to construct token embeddings.
The authors conduct experiments on OpenWebText, training three models ranging from 138M to 784M parameters. Their approach achieves improvements in generative perplexity over previous baselines.
Critical strengths
Token commitment and embedding geometry are two important design considerations when developing better diffusion/flow-based language models, particularly if the ultimate goal is to build an alternative backbone that can potentially outperform autoregressive models.
The paper clearly identifies and motivates these challenges and proposes corresponding solutions. The experimental results also demonstrate improvements over existing baselines.
Critical weaknesses
- If our goal is to develop scalable language models, I am concerned that relying on hand-crafted embeddings may eventually become a limitation. Is Hellinger–SVD really a better choice than embeddings that can be learned end-to-end? More importantly, does the proposed embedding geometry actually make the representations more suitable for denoising? This reminds me of a similar issue in visual tokenization, where better reconstruction quality does not necessarily translate into better generation quality. I think a similar distinction should be made here. From the paper, I can see that the proposed embeddings satisfy the first requirement: they are sufficiently expressive to represent all tokens. However, this alone is not particularly convincing, since even a simple lookup table can achieve this. The second requirement is more important: are these representations actually well-suited for denoising? The paper mainly provides intuition-level arguments for this, and it remains unclear whether the observed improvements come from easier posterior estimation, the training-time noise schedule, or other aspects of the overall design.
- Another concern is that GenPPL and entropy can be improved or manipulated in various ways without necessarily improving the actual quality of generated text. Therefore, I do not think these metrics alone are sufficient for evaluating generation quality. I would encourage the authors to include additional evaluation metrics to provide a more comprehensive assessment.
- The authors introduce weight interpolation during the reflow stage, but I could not find any corresponding ablation studies. It would be helpful to understand how much this design choice contributes to the final performance.
- According to the paper, training ReLF-Small requires approximately 300 GPU-hours. However, the reflow procedure may introduce substantial additional computational costs, since it typically requires generating a large number of samples. Does the reported training budget include the cost of reflow? If not, how much additional compute does reflow require, and how does the total training cost compare with ELF and other baselines? I could not find a clear discussion of this in the paper.
Questions for authors
- Hellinger–SVD indeed outperforms Gaussian and one-hot baselines. but why is token geometry constructed based on local context statistics particularly well-suited for learning the full denoising posterior? in other words, can the authors disentangle the contribution of embedding geometry from that of the corresponding noise schedule?
- Please give us more evidence on how Hellinger–SVD is suitable for flow language models.