Rectified Language Flow

Continuous flow language models had lagged behind discrete-state diffusion models. They face challenges absent from discrete models: linking a continuous velocity field to a categorical vocabulary, committing to a single token as noise vanishes, and choosing the embedding geometry in which denoising takes place. We present Rectified Language Flow (ReLF), a rectified-flow language model trained with cross-entropy. We introduce singular logits, which analytically capture the divergent component of token posteriors, leaving the network a bounded residual and guaranteeing convergence to valid token embeddings. The flow runs in a compact, context-informed embedding space, and reflow straightens it for few-step generation. On OpenWebText with 64 function evaluations, ReLF attains lower generative perplexity than autoregressive baselines at every model size tested under matched diversity.