acceptodds
Under review as a conference paper at ICLR 2027

Rectified Language Flow

25 views

Abstract

Continuous flow language models had lagged behind discrete-state diffusion models. They face challenges absent from discrete models: linking a continuous velocity field to a categorical vocabulary, committing to a single token as noise vanishes, and choosing the embedding geometry in which denoising takes place. We present Rectified Language Flow (ReLF), a rectified-flow language model trained with cross-entropy. We introduce singular logits, which analytically capture the divergent component of token posteriors, leaving the network a bounded residual and guaranteeing convergence to valid token embeddings. The flow runs in a compact, context-informed embedding space, and reflow straightens it for few-step generation. On OpenWebText with 64 function evaluations, ReLF attains lower generative perplexity than autoregressive baselines at every model size tested under matched diversity.

Primary area:
generative models
Keywords:
rectified flow, flow matching, continuous diffusion, language modeling, text generation, few-step generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZE2xOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a continuous diffusion/flow-based language model called Rectified Language Flow (ReLF). It aims to address several challenges associated with modeling text in continuous space. Continuous diffusion/flow models offer several potential advantages over their discrete counterparts. Most notably, they can more readily adopt techniques developed in the continuous domain, such as ODE/SDE sampling, numerical solvers, and distillation methods, as has been demonstrated in image generation.

However, applying continuous diffusion/flow models to language introduces several challenges. One is token commitment: continuous flows produce continuous-valued vectors, whereas language generation ultimately requires discrete tokens. To address this issue, the authors propose singular logits, which encourage the posterior to converge to a one-hot distribution.

Another important challenge is embedding geometry: what kind of token representations are suitable for continuous flow-based language models? This differs from the representation-learning problem in autoregressive language models. In denoising models, tokens with highly similar embeddings may be more difficult to distinguish, making the denoising task more challenging. To address this, the authors propose Hellinger–SVD to construct token embeddings.

The authors conduct experiments on OpenWebText, training three models ranging from 138M to 784M parameters. Their approach achieves improvements in generative perplexity over previous baselines.

Critical strengths

Token commitment and embedding geometry are two important design considerations when developing better diffusion/flow-based language models, particularly if the ultimate goal is to build an alternative backbone that can potentially outperform autoregressive models.

The paper clearly identifies and motivates these challenges and proposes corresponding solutions. The experimental results also demonstrate improvements over existing baselines.

Critical weaknesses
  1. If our goal is to develop scalable language models, I am concerned that relying on hand-crafted embeddings may eventually become a limitation. Is Hellinger–SVD really a better choice than embeddings that can be learned end-to-end? More importantly, does the proposed embedding geometry actually make the representations more suitable for denoising? This reminds me of a similar issue in visual tokenization, where better reconstruction quality does not necessarily translate into better generation quality. I think a similar distinction should be made here. From the paper, I can see that the proposed embeddings satisfy the first requirement: they are sufficiently expressive to represent all tokens. However, this alone is not particularly convincing, since even a simple lookup table can achieve this. The second requirement is more important: are these representations actually well-suited for denoising? The paper mainly provides intuition-level arguments for this, and it remains unclear whether the observed improvements come from easier posterior estimation, the training-time noise schedule, or other aspects of the overall design.
  2. Another concern is that GenPPL and entropy can be improved or manipulated in various ways without necessarily improving the actual quality of generated text. Therefore, I do not think these metrics alone are sufficient for evaluating generation quality. I would encourage the authors to include additional evaluation metrics to provide a more comprehensive assessment.
  3. The authors introduce weight interpolation during the reflow stage, but I could not find any corresponding ablation studies. It would be helpful to understand how much this design choice contributes to the final performance.
  4. According to the paper, training ReLF-Small requires approximately 300 GPU-hours. However, the reflow procedure may introduce substantial additional computational costs, since it typically requires generating a large number of samples. Does the reported training budget include the cost of reflow? If not, how much additional compute does reflow require, and how does the total training cost compare with ELF and other baselines? I could not find a clear discussion of this in the paper.
Questions for authors
  1. Hellinger–SVD indeed outperforms Gaussian and one-hot baselines. but why is token geometry constructed based on local context statistics particularly well-suited for learning the full denoising posterior? in other words, can the authors disentangle the contribution of embedding geometry from that of the corresponding noise schedule?
  2. Please give us more evidence on how Hellinger–SVD is suitable for flow language models.
Reviewer_bXMbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose ReLF, a continuous rectified-flow LM. They state three main contributions: 1) singular logits, an analytic unigram-reference posterior addition to model’s own predictions; 2) fixed 32-d Hellinger–SVD embeddings; 3) regularized reflow for few-step sampling. On OWT ReLF beats same-size AR and ELF at matched entropy, and after reflow it beats ELF+PD from 8 NFE. The method also reaches data-level metrics in Large model size.

Critical strengths
  1. Singular logits are an interesting, cheap, well-motivated idea with a clear ablation gain.
  2. Strong unconditional results. The continuous model beats AR at every scale.
Critical weaknesses
  1. Singular logits closely resemble CoBit’s analytic matched-filter residual [1]. And it isn’t cited.
  2. Hellinger PCA embedding is a known technique. The context smoothing may crowd rare tokens near the mean. The attached samples are consistent with that – there are malformed rare words like “Explosvers”, “provinticisticist”, “dialhetic”. The metrics can’t detect this.
  3. Diversity is only measured within each sample – there is no inter-sample diversity or distributional metric such as MAUVE.
  4. The few-step comparison is limited, stronger baselines such as FMLM* are missing [2].
  5. Key components are not ablated: regularized vs. standard reflow, logit- vs. velocity-space guidance, and the arc-length time schedule.

[1] Batzolis et al. “CoBit: Language Modeling with Bitstream Diffusion”. 2026

[2] Yoo et al. “Self-conditioned Flow Map Language Models via Fixed-point Flows”. 2026

Questions for authors
  1. Could you report inter-sample diversity and MAUVE? Can you also report a metric for the model’s ability to generate rare tokens?
  2. Could you include comparison with FMLM*?
  3. Could you show results on conditional tasks such as TinyGSM -> GSM8K or Sudoku?
  4. Could you add the reflow, guidance-space, and schedule ablations?
Reviewer_bfUdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReLF, a continuous rectified-flow language model that generates text by running an ODE from Gaussian noise to token embeddings, trained with cross-entropy on token posteriors. There are three main pieces. First, the posterior has to become one-hot at the clean endpoint, which would require unbounded logits, so the authors compute this divergent part in closed form and let the network predict only a bounded residual. Second, the flow runs in a fixed 32-dimensional space built upon each token’s context distribution, so tokens with similar contexts have close embeddings. Third, for few-step generation, they adapt reflow and interpolate the new weights back toward the base model after each round. Their experiments run on unconditional OpenWebText generation with three model sizes, scored by generative perplexity under GPT-2 Large together with unigram entropy. At 64 function evaluations and matched entropy, ReLF reaches lower generative perplexity than AR baselines with the same width and depth.

Critical strengths

Continuous flow beats size-matched AR at three scales. At matched entropy, ReLF reaches Gen PPL lower than AR baselines with the same widths and depths. Continuous-state models have trailed AR for several years, so this is a meaningful attempt and data point.

The commitment guarantee is informative. Theorem 3.1 bounds the exact residual uniformly in t, and Proposition 3.3 concludes convergence of the trajectory to a valid token embedding. Because this holds for any uniformly bounded residual, the model can remove the rounding step or separate decoder that prior works need.

The Ablation Study is Sound. Figure 4 isolates different design choices proposed in the paper, such as singular logits, CE v.s. MSE, and the flow dimension. These studies provide evidence that the building blocks of ReLF contribute to final performance improvement, consistent with their theoretical analysis.

Critical weaknesses

The evaluation is limited to a single benchmark with one pair of metrics, and the benchmark is already saturated. The paper evaluates only unconditional generation on OpenWebText using Gen PPL and unigram entropy, and ReLF-Large already matches real text (17.3 at 5.51 vs. 17.3 at 5.49). I understand that this is a standard evaluation convention, but finding new benchmarks that can better distinguish model performance is important. The abstract also motivates the work by contrasting it with discrete-state diffusion, yet the paper includes no discrete diffusion baseline.

The formulation supports only unconditional generation. Time is a single scalar shared by all positions (Eqs. 1 and 4), so there is no mechanism to hold a prefix fixed while a suffix denoises. The conclusion nonetheless positions continuous flows as “a promising alternative to autoregressive language modeling”, but no conditional generation, where AR models are good, is discussed and compared in the paper.

The reflow procedure rests on four hand-picked constants: Couplings are generated at ω = 1.3 in the first round and ω = 1 in the second, and the interpolation uses λ = 0.7 then 0.55, with no stated selection criterion. An ablation study or criterion may strengthen these choices. And although L298–300 claims θk+1θ_{k+1} beats the unregularized one, Figure 1(b) only varies the round count, so a λ = 0 control is needed to attribute the Table 6 gains to regularized reflow rather than to plain reflow.

Questions for authors

Can the formulation be extended beyond unconditional generation? In the current scope, time is a single scalar shared by all positions, so it is not clear to me how a prefix could be held fixed while the rest of the sequence denoises. A short description of what would change, or one conditioned generation result, would make the comparison with AR models much more meaningful.

How were the reflow guidance scales and interpolation weights selected? See Weakness 3, appendix B.2 gives ω = 1.3 / 1 and λ = 0.7 / 0.55 without saying how they were found, how is it selected? If these were chosen by looking at generative perplexity, the few-step numbers would be optimistically biased.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.