- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SimpLM, a conditional generation framework for flow language models that can adapt pretrained autoregressive (AR) models. To adapt AR models effectively, the authors propose a blockwise flow formulation with curriculum learning, where the block size is initially set to 1, as in AR models, and then gradually increased to larger block sizes. In addition to its blockwise AR design, the authors propose two improvements to the model parameterization. The analytic posterior-based logit skip connection allows the model to predict residual logits while directly incorporating an analytic term derived from the Gaussian posterior, rather than predicting the entire posterior with the network. Moreover, the probability-simplex representation allows the model to use simplex-based inputs instead of noisy Euclidean embeddings.
Critical strengths
- Compared to its ablated versions, the analytic posterior-based logit skip connection substantially improves the model's performance by leveraging the structure of Gaussian corruption.
Critical weaknesses
-
The novelty of the paper is limited.
- Lee et al. (2026) already report conditional generation results with FMLM. The novelty of this paper should therefore come from a clear and rigorous formulation of its blockwise flow. However, the formulation itself is described largely through verbal explanations rather than a rigorous mathematical formulation of conditional flow generation.
- The recipe for the proposed blockwise flow design largely overlaps with existing methods. Diffusion Forcing (Chen et al., 2024) already proposed independently controlled noise levels for flexible denoising schedules, and AR-Diffusion (Wu et al., 2023) adopted position-dependent diffusion timesteps for language modeling to leverage the benefits of AR modeling. Moreover, blockwise autoregression that unifies AR and diffusion was already proposed in BD3-LMs (Arriola et al., 2025). The authors should clarify why the proposed design is not simply a combination of existing methods and what nontrivial challenges arise when combining them. The paper should also clearly distinguish the contributions of prior work from its own novel contributions.
-
The effectiveness of the probability-simplex representation is not fully validated.
-
While the authors argue for its effectiveness based on validation loss and ELBO-based perplexity, its improvement in actual generation performance, measured by generative perplexity, is marginal or even slightly negative. If the authors wish to argue that generative perplexity is an incomplete evaluation metric, I strongly recommend evaluating generation performance with MAUVE (Pillutla et al., 2021) or performing a GenPPL-entropy frontier analysis, as proposed in Generative Frontiers (Pynadath et al., 2026).
-
I am not fully convinced by the authors' claim that the probability-simplex representation improves generation performance, as the NELBO does not directly account for the inference-time errors accumulated during finite-step sampling. Moreover, the gap between the NELBO and the exact negative log-likelihood can depend on the model's design. Therefore, the current likelihood-based evaluation alone does not provide sufficient evidence for the effectiveness of the probability-simplex representation.
-
Lee et al., Flow Map Language Models: One-step Language Modeling via Continuous Denoising, 2026.
Chen et al., Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion, 2024.
Wu et al., AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation, 2023.
Arriola et al., Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models, 2025.
Pillutla et al., MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers, 2021.
Pynadath et al., CANDI: Hybrid Discrete-Continuous Diffusion Models, 2025.
Pynadath et al., Generative Frontiers: Why Evaluation Matters for Diffusion Language Models, 2026.
Questions for authors
-
Self-conditioning citation: While FLM (Lee et al., 2026) does not use self-conditioning, the paper cites it as an example of a self-conditioned model (L178–180). This citation should be revised.
-
Euclidean baseline: In L327, the paper cites FLM to explain its Euclidean-space baseline, but the implementation omits a key design component of FLM: time reparameterization. The authors should clearly state this difference and discuss its potential impact on the baseline's performance.
-
Clarity of the method section: The writing of the method section could be improved. The current method section is understandable at a high level, but the exact formulation is not clearly stated, making some explanations ambiguous. This could be improved by (1) explaining the blockwise flow formulation first (Section 4.2) and then (2) introducing the curriculum learning algorithm (Section 4.1) with clear and concise mathematical notation. The current formulation also makes it difficult to distinguish between tokens used as clean contexts and tokens undergoing denoising.
-
Concurrent work: CoBit (Batzolis et al., 2026) also uses an analytic logit skip connection based on the Gaussian posterior. Although the paper was first released as an arXiv preprint in May 2026, it would be helpful to cite it as related or concurrent work and briefly compare the proposed method with CoBit, particularly regarding the analytic posterior correction.
Batzolis et al., CoBit: Language Modeling with Bitstream Diffusion, 2026.