acceptodds
Withdrawn from ICLR 2027

SimpLM: Simplex-Based Flow Language Modeling via Autoregressive Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive language models remain the dominant language modeling paradigm, yet their sequential decoding limits efficiency. While recent efforts have explored discrete diffusion language models for parallel decoding, these approaches suffer from severe degradation in text quality when decoding multiple tokens simultaneously due to token-wise independence assumptions. Continuous-space diffusion and flow models offer a compelling alternative, but existing methods are often restricted to unconditional full-sequence generation, and some rely heavily on learned embedding encoders to map discrete tokens into continuous latent states, requiring additional pretrained encoders. In this work, we propose SimpLM, a continuous flow language modeling framework that operates directly on the probability simplex in the vocabulary space. By applying Gaussian interpolation to the probability states, we derive a closed-form posterior logit correction that yields an analytic skip connection for logit transfer, bypassing the hidden-state information bottleneck in Transformer backbones. Beyond this, SimpLM natively supports conditional and block-based semi-autoregressive generation via a vectorized time formulation that assigns separate time variables to block positions, providing a principled foundation for continuous-time conditional generation. Furthermore, SimpLM can adapt a pretrained autoregressive model into a continuous flow model, reducing the cost of training non-autoregressive models from scratch. Empirically, with only 30B training tokens, SimpLM achieves a generative perplexity below 20 at block size and generation length of 128.

Primary area:
generative models
Keywords:
continuous flow, simplex representation, pretrained language model adaptation, blockwise generation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_gYgMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SimpLM, a conditional generation framework for flow language models that can adapt pretrained autoregressive (AR) models. To adapt AR models effectively, the authors propose a blockwise flow formulation with curriculum learning, where the block size is initially set to 1, as in AR models, and then gradually increased to larger block sizes. In addition to its blockwise AR design, the authors propose two improvements to the model parameterization. The analytic posterior-based logit skip connection allows the model to predict residual logits while directly incorporating an analytic term derived from the Gaussian posterior, rather than predicting the entire posterior with the network. Moreover, the probability-simplex representation allows the model to use simplex-based inputs instead of noisy Euclidean embeddings.

Critical strengths
  • Compared to its ablated versions, the analytic posterior-based logit skip connection substantially improves the model's performance by leveraging the structure of Gaussian corruption.
Critical weaknesses
  1. The novelty of the paper is limited.

    • Lee et al. (2026) already report conditional generation results with FMLM. The novelty of this paper should therefore come from a clear and rigorous formulation of its blockwise flow. However, the formulation itself is described largely through verbal explanations rather than a rigorous mathematical formulation of conditional flow generation.
    • The recipe for the proposed blockwise flow design largely overlaps with existing methods. Diffusion Forcing (Chen et al., 2024) already proposed independently controlled noise levels for flexible denoising schedules, and AR-Diffusion (Wu et al., 2023) adopted position-dependent diffusion timesteps for language modeling to leverage the benefits of AR modeling. Moreover, blockwise autoregression that unifies AR and diffusion was already proposed in BD3-LMs (Arriola et al., 2025). The authors should clarify why the proposed design is not simply a combination of existing methods and what nontrivial challenges arise when combining them. The paper should also clearly distinguish the contributions of prior work from its own novel contributions.
  2. The effectiveness of the probability-simplex representation is not fully validated.

    • While the authors argue for its effectiveness based on validation loss and ELBO-based perplexity, its improvement in actual generation performance, measured by generative perplexity, is marginal or even slightly negative. If the authors wish to argue that generative perplexity is an incomplete evaluation metric, I strongly recommend evaluating generation performance with MAUVE (Pillutla et al., 2021) or performing a GenPPL-entropy frontier analysis, as proposed in Generative Frontiers (Pynadath et al., 2026).

    • I am not fully convinced by the authors' claim that the probability-simplex representation improves generation performance, as the NELBO does not directly account for the inference-time errors accumulated during finite-step sampling. Moreover, the gap between the NELBO and the exact negative log-likelihood can depend on the model's design. Therefore, the current likelihood-based evaluation alone does not provide sufficient evidence for the effectiveness of the probability-simplex representation.

Lee et al., Flow Map Language Models: One-step Language Modeling via Continuous Denoising, 2026.
Chen et al., Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion, 2024.
Wu et al., AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation, 2023.
Arriola et al., Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models, 2025.
Pillutla et al., MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers, 2021.
Pynadath et al., CANDI: Hybrid Discrete-Continuous Diffusion Models, 2025.
Pynadath et al., Generative Frontiers: Why Evaluation Matters for Diffusion Language Models, 2026.

Questions for authors
  • Self-conditioning citation: While FLM (Lee et al., 2026) does not use self-conditioning, the paper cites it as an example of a self-conditioned model (L178–180). This citation should be revised.

  • Euclidean baseline: In L327, the paper cites FLM to explain its Euclidean-space baseline, but the implementation omits a key design component of FLM: time reparameterization. The authors should clearly state this difference and discuss its potential impact on the baseline's performance.

  • Clarity of the method section: The writing of the method section could be improved. The current method section is understandable at a high level, but the exact formulation is not clearly stated, making some explanations ambiguous. This could be improved by (1) explaining the blockwise flow formulation first (Section 4.2) and then (2) introducing the curriculum learning algorithm (Section 4.1) with clear and concise mathematical notation. The current formulation also makes it difficult to distinguish between tokens used as clean contexts and tokens undergoing denoising.

  • Concurrent work: CoBit (Batzolis et al., 2026) also uses an analytic logit skip connection based on the Gaussian posterior. Although the paper was first released as an arXiv preprint in May 2026, it would be helpful to cite it as related or concurrent work and briefly compare the proposed method with CoBit, particularly regarding the analytic posterior correction.

Batzolis et al., CoBit: Language Modeling with Bitstream Diffusion, 2026.

Reviewer_pYUwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new method based on block-wise continuous flow matching in vocabulary space. The authors show analytically that, when one-hot token representations are corrupted with Gaussian noise, the logits of the posterior over the clean token decompose exactly, up to a token-independent constant, into two terms. The first is the evidence contained in the noisy state itself, which is available in closed form. The second is the contextual log-prior of the token. Building on this, the network is parameterized to model only the contextual term, while the analytic term is added explicitly to its output as a logit skip connection. Instead of the raw noisy vector, the network receives the posterior over the vocabulary computed from the analytic term alone (i.e., under a uniform prior), projected through the embedding matrix.The model is initialized from a pretrained autoregressive LM (Pythia-160M) and fine-tuned with a gradually increasing block size. Experiments on OpenWebText consist of ablations of these components, evaluated by validation loss, ELBO-based perplexity, and generative perplexity.

Critical strengths
  1. The method is derived analytically rather than designed heuristically. Because the noise is Gaussian, the part of the prediction that depends on the noisy state itself can be computed exactly. The authors use this to let the network model only the contextual part of the prediction, while the rest is supplied in closed form. This adds no parameters and is simple to implement.
  2. Experiments demonstrate a benefit of the proposed parameterization. Compared with the standard vocabulary-space flow model, it reduces validation loss across noise levels (Fig. 2), ELBO perplexity (Fig. 3), and generative perplexity from 127 to 45 (Table 1). The gain holds consistently across all block sizes considered (1, 4, and 32).
Critical weaknesses

1.Limited experimental comparison. All experiments are ablations on a single small model (Pythia-160M). There is no comparison with the AR model itself, with discrete diffusion LMs (e.g., MDLM, BD3-LM), or with other continuous flow LMs (e.g., LangFlow, ELF, Lee et al., 2026), so it is unclear where the method stands relative to existing approaches. It is also unclear whether the gains hold for larger models.

  1. No evidence of efficiency. Parallel decoding is the main motivation, but wall-clock time and the number of forward passes are not compared with AR decoding. The best reported result (block size 128) uses 256 sampling steps for 128 tokens, i.e., more forward passes than AR generation would need, while with 64 steps the quality degrades sharply (Appendix D).

  2. No benchmark evaluation. Quality is measured only by generative perplexity under GPT-2 Large and unigram entropy, which are indirect and can be traded off against each other. Results on standard benchmarks would make the quality claims much more convincing.

  3. Presentation and reproducibility issues. The training budget is inconsistent across Sec. 5.1 (tokens), Appendix B (steps), and the step counts in Fig. 2–4. The noise scale σ and t_clean are not reported. Appendix D uses an old method name (“FlowLLM”). The title is somewhat misleading: the flow itself is Gaussian in Euclidean space, and the simplex appears only in the input representation.

Questions for authors
  1. How does SimpLM compare in wall-clock time and number of forward passes with AR decoding of the same Pythia-160M model at matched quality?
  2. Can you add comparisons with the base AR model and at least one discrete diffusion LM (e.g., MDLM or BD3-LM) and one continuous flow LM under the same evaluation protocol?
  3. What are the values of σ and t_clean, the number of ELBO samples n?
  4. Why does the simplex representation alone yield the worst generative perplexity (616) despite a better ELBO than the baseline?
  5. Please reconcile the training budget across Sec. 5.1, Appendix B, and Figs. 2–4.
Reviewer_ytFPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents SimpLM −- a flow matching model on one-hot token vectors initialized with a pretrained autoregressive model. The authors derive two key techiniques that amplify the performance of the method. First, their analytic derivation of the forward process likelihood motivates the logit skip-connection, which helps by explicitly adding noisy-input information to the model outputs, thus simplifying the task. Second, they introduce vectorized time, which allows for conditional generation. SimpLM architectural choices are ablated on OpenWebText. Both generative and likelihood metrics are used for the ablations.

Critical strengths

The paper is clearly written and structured. I had and easy time with getting familiar with it. The authors introduce logit skip and vectorized time to Flow Matching on simplex, which was not done before.

Critical weaknesses

There are two main critiques to the manuscript.

The first one is very weak experimental verification of the method. The paper lacks clear comparison with strong language diffusion baselines. OpenWebText suits as the defalut benchmark for language modeling. However, the sequence length of 512 tokens is rather unconventional, thus making it hard to compare.

Euclidean baseline is very different to FLM −- conceptually the closest model. First of all, it is initialized with the autoregressive model and is trained in the block-causal fashion. Second, as far is I understood, it does not include time reparameterization but includes self-conditoning. Considering all of the above, it is not the FLM. One way to introduce more fair comparison is to retrain the model on the conditional generation with a more thorough FLM recipe. But, of course, training on a conventional sequence length of 1024 is much more preferred.

Also, there are no numbers on performance of Pythia on the task. The authors state that they "show how to convert pretrained autoregressive language models into continuous flow models with much less training cost than pretraining from scratch" (line 85). However, it is not possible to infer from the manuscript if this convertation is even worth it. The performance may or may not suffer (but I would assume that it suffers). If the former is true, the potential benefits of the convertation are unclear. There is no measurements of latency gains (or losses). Since the method my require 2x steps per new token (line 419), I would not hope to see much improvements in this direction.

There is also no convincing ablation on effective converting of the autoregressive model. The authors eplore only one curriculum to adapt Pythia. Wether it is even required remains unclear.

The model trained from scratch is also very undertrained. Usually it takes much more than 100k iterations for the diffusion language model to reach it full potential and to see better perfromance from the pretrained models with equated training compute seems to be expected.

Assesing a generative model quality with the ELBO and the loss on trajectory I also consider flawed. Tighter ELBO does not necessarily imply better generation.

While the lower loss on the trajectory might indicate a better model, in this particular case I would be concerned that half of the trajectory virtually does not contribute to the training or the generative process. The loss drastically decreases to zero at around t ~ 0.5.

The second major critique is the lack of novelty. The techniques that the paper presents mostly are slight reformulations of previously explored ideas. To my knowledge, the idea similar to logit skip was introduced in Plaid, and even concurrent works such as CoBit explore it for different modalities. This also goes for the vectorized time. The attempts to implement similar techniques were made extensively in the literature, and this particular paper does not explore the potential of different non-constant noise functions.

Considering all of the aforementioned weaknesses, I would rate the paper as clear rejection.

Questions for authors

Presenting experiments with more baselines and/or with sequence length of 1024 could potentially make me raise my score

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.