acceptodds
Under review as a conference paper at ICLR 2027

Improving Few-Step Language Flows with Untied Self-Conditioning

Abstract

Flow-matching language models refine all token positions in parallel and can trade sampling steps for latency, yet generation quality still degrades sharply with few sampling steps. We trace a source of this degradation to how self-conditioning reuses predictions during sampling. Training constructs the conditioning input from the current noisy state. During sampling, self-conditioning ties two distinct uses to the same clean prediction: advancing the latent through the solver, and conditioning the next model evaluation. A correction that improves generation when supplied to the solver can become harmful when also used for self-conditioning. We introduce Untied Self-Conditioning, a training-free method that unties these two uses by adapting a shared prediction to their different requirements, using only one model evaluation per sampling step. The past prediction already contributes to the latent, we therefore scale down its conditioning components along directions shared by the model's input projections. For the solver update, we instead use prediction history to estimate how the clean prediction changes during a sampling step and scale the resulting correction using empirical trajectory statistics. Across several pretrained language flow baselines, our method consistently improves generation quality: with only 8 function evaluations (NFE), it reduces generative perplexity on OpenWebText from to  () for LangFlow, and from to  for ELF-B. Under an adapted Arena-Hard-Auto v2 evaluation on LangFlow trained on OpenWebText, our generations win of non-tied comparisons with the official sampler at the same NFE. Improvements hold from 8 to 256 sampling steps across all tested models and datasets, suggesting that our method offers a promising direction for efficient sampling in continuous diffusion language models.

Primary area:
generative models
Keywords:
flow matching, language modeling, self-conditioning, few-step generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_KPGTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Untied Self-Conditioning, a training-free approach to improve few-step generation in continuous flow-matching language models. The key idea is to separate the use of the model's clean prediction for self-conditioning and numerical solver updates. The method introduces a projection-based conditioning transform and a history-based solver correction without additional model evaluations. Experiments on ELF and LangFlow demonstrate substantial improvements in generative perplexity, particularly at low sampling steps.

Critical strengths
  1. Practical training-free improvement: The method improves generation quality without retraining or additional model evaluations. The large improvements at low NFE are particularly promising for efficient diffusion language model inference.

  2. Strong experimental evidence: The paper evaluates multiple model sizes and sampling budgets. The ablation studies demonstrate the importance of separating solver corrections from self-conditioning and provide useful evidence supporting the proposed design.

Critical weaknesses
  1. Limited generalizability of the proposed correction: While the paper provides a theoretical motivation for separating conditioning and solver updates, the implemented correction relies on prediction histories, offline trajectory statistics, and configuration-dependent coefficients. Although the authors provide supporting diagnostics, it remains unclear how well the method generalizes to different architectures, sampling schedules, or checkpoints without additional calibration.

  2. Limited evaluation of broader generation quality: The improvements are primarily demonstrated through generative perplexity on unconditional text generation. While the authors also provide LLM-judge evaluations and diversity measurements, it remains unclear whether these improvements extend to broader language-generation capabilities. Additionally, Appendix C.3 reports higher word 4-gram repetition rates across the evaluated LangFlow configurations, suggesting a potential trade-off that warrants further investigation.

Questions for authors
  1. How sensitive are the improvements to the offline trajectory statistics and configuration-specific coefficients? Can the authors demonstrate whether the same calibration transfers across checkpoints or sampling schedules?

  2. The paper provides strong improvements in generative perplexity and complementary LLM-judge evaluations. Can the authors further clarify how these gains translate to broader text-generation quality or downstream capabilities (Math, Coding etc), where supported by the evaluated models?

  3. Appendix C.3 reports consistently higher word 4-gram repetition rates for Untied Self-Conditioning. Can the authors explain this increase and whether it indicates a trade-off between generative perplexity and repetition?

  4. While the projection-overlap ablations support the proposed conditioning transform, could the authors clarify under what assumptions geometric overlap between input projections is a reliable indicator of redundant conditioning information?

Reviewer_GyMoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Untied Self-Conditioning, a training-free method for improving few-step sampling in self-conditioned flow language models. It argues that using the same prediction for both self-conditioning and the solver update is suboptimal, since the solver needs the average prediction over each step, while the previous prediction is already included in the noisy input. Motivated by this, it proposes 1) scaling down the self-conditioning input and 2) an EMA-based solver correction, and uses them to untie the prediction supplied to the solver from the one used for self-conditioning.

Critical strengths

Strengths

  1. The experiments are extensive and consistent. The proposed method is evaluated on different model and sizes, including multi-seed average results and LLM-judge results.
  2. The proposed is simple and requires no additional training cost or model evaluation.
Critical weaknesses
  1. The major weakness is the methodological novelty. The solver-correction part has nothing to do with self-conditioning: it could be used in any ODE solver, with or without self-conditioning, and seems to extend existing higher-order multistep solvers, but the current explanation blurs the connection. For the self-conditioning transformation, the equations contain unnecessary jargons, but the final method ends up simply reducing the scale heuristically. Importantly, the authors does not properly discuss what their baseline methods, especially advanced solvers, do in detail.
  2. Most of the claimed few-step improvement on gen.ppl comes together with severe entropy drop. Therefore, the current gen.ppl-based performance of the model is unreliable.
Questions for authors
  1. Can the authors evaluate their method using distribution-based quality metrics, such as MAUVE, or GM [1] ?
  2. Can the authors clearly and specifically explain how their EMA-based solver correction strategy is different from existing ode solvers, especially with the methods that the authors cited in the related works section? Especially, how is the author's proposed method different from HiGS's EMA-based correction?

[1] Hoogeboom et al., Beyond Single Tokens: Distilling discrete diffusion models via discrete MMD , arXiv 2026

Reviewer_HfbsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies self-conditioning in flow-matching language models, where the clean prediction at each step is used both to advance the latent through the solver and as the conditioning input for the next step. The authors argue that these two uses have different requirements and propose Untied Self-Conditioning, a training-free sampler that separates them. The conditioning input is scaled down along directions where the input projections of the latent and conditioning input overlap, and the solver receives a corrected prediction built from prediction histories and trajectory statistics.

The authors evaluate the method on ELF (three model sizes) and LangFlow on OpenWebText and LM1B. It improves Gen-PPL at every tested NFE, with the largest gains at low NFE, for example from 531 to 61 for LangFlow and from 71 to 44 for ELF-B at 8 NFE on OpenWebText. The paper also reports an LLM-judge comparison against the official sampler, ablations of the method's components, and a comparison with other training-free samplers.

Critical strengths
  1. The method is training-free and uses only one model evaluation per step, retaining about 95% of the official sampler's throughput. It can therefore be applied directly to existing pretrained flow language models.

  2. The evaluation is extensive, covering two model families and two datasets. The gains are especially clear at low NFE, where the official samplers perform poorly. The LLM-judge comparison is also a useful addition.

Critical weaknesses
  1. The paper is difficult to read, and the motivation for scaling down the self-conditioning input is unclear. Since the network is trained to combine both inputs, redundancy should be handled by the learned weights. This would break if there is a train-test mismatch, but the paper does not measure it, and the transform relies on projection geometry rather than any measured property. For LangFlow, it reduces to uniformly scaling the conditioning input.

  2. Self-conditioning is simple and easy to implement, but the proposed sampler adds many components, including the principal-angle transform, two prediction histories, offline trajectory statistics, and a base exponent derived from a finite-difference Gram matrix. It also introduces several coefficients (ηR, ηT, ηO, α) that are set differently for each model and NFE (Table 6), without describing how they were selected or how sensitive the results are to them.

  3. The improvements in Gen-PPL come with a drop in sample entropy, especially at low NFE (e.g., 5.81 to 5.32 for LangFlow on OWT at 8 NFE, and 5.31 to 5.06 for ELF-L), and the entropy falls below the data reference of about 5.4. Word repetition also increases as seen in Table 5. Since entropy can be traded for lower Gen-PPL, it is unclear how much of the improvement reflects better generation, and the paper does not report MAUVE.

Questions for authors
  1. Can the authors report Gen-PPL numbers for the baselines at matched entropy? MAUVE score would also be very useful.
  2. How were the coefficients in Table 6 selected? Can the authors report ablation sweeps on these hyperparameters?
Reviewer_YFFUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors identify two main problems in existing self-conditioning methods: the correlation between the self-conditioning input and the latent input can lead to suboptimal model predictions when using fixed embedding layers, and discretization errors along the ODE trajectory become significant in few-step generation. To address these issues, the authors propose reducing the components of the previous prediction input along the overlapping row-space directions of the latent and self-conditioning embedding layers. They also propose refining the model prediction at each timestep using EMA, solver-weighted prediction history of previous predictions, and precomputed step-wise statistics (e.g. μk\mu_k and σk\sigma_k) obtained from a fixed reference sampler. By applying these methods to pretrained ELF and LangFlow models, the authors empirically demonstrate that their approach can competitive with or outperform existing methods without any additional training.

Critical strengths

The proposed untied self-conditioning method empirically demonstrates substantially improves generative perplexity (GenPPL), enabling pretrained models to match or outperform few-step distilled models without any additional training. Furthermore, the method consistently reduces GenPPL across different pretrained models like ELF and LangFlow.

Critical weaknesses

Overall, this paper lacks sufficient theoretical justification for the proposed methods.

  1. Although Sections 3.1 and 3.2 provide theoretical motivation for reducing redundancy between the previous prediction input and the latent input, it remains unclear how reducing the projection overlap energy in Section 4.1 leads to improved predictions from a pretrained model.
  2. Section 3.3 argues that adding a correction direction BkB_k that is positively aligned with the difference Ck∗C^\ast_k between the step-average prediction and the current model prediction can reduce the local discretization error with approciate scaling. However, the paper does not theoretically establish whether the proposed correction AkA_k in Section 4.2 satisfies this condition. In particular, there is no theoretical guarantee that AkA_k is positively aligned with Ck∗C_k^\ast at the current latent state zkz_k.
Questions for authors
  1. The current evaluation metrics do not seem to adequately capture the trade-off between entropy and GenPPL. Could the authors provide a GenPPL-entropy trade-off frontier curve, as used in Pynadath et al. [1], to better illustrate this relationship?

  2. Could the authors evaluate whether the proposed method affects the diversity and distributional quality of generated samples, for example using MAUVE?

  3. The caption of Figure 2 appears to incorrectly cite Yoo et al. [2] as the source of the LM1B reference results, although their original paper does not report experiments on LM1B. Could the authors clarify the source of these results?

[1] Pynadath et al., "CANDI: Hybrid Discrete-Continuous Diffusion Models", 2025

[2] Yoo et al., "Self-conditioned Flow Map Language Models via Fixed-point Flows", 2026

Reviewer_banCOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a train–inference mismatch in self-conditioned continuous diffusion language models. During sampling, the previous denoiser prediction influences the next model evaluation both through the updated latent and as an explicit self-conditioning input, whereas training constructs these inputs differently.

The authors propose Untied Self-Conditioning, a training-free sampler that attenuates redundant conditioning directions using the geometry of frozen input projections, while separately correcting the solver update using prediction history and calibrated trajectory statistics. The method requires no additional neural network evaluations.

Experiments on ELF-B/M/L and LangFlow across OpenWebText and LM1B show consistent generative perplexity improvements, including 531.2 to 60.8 for LangFlow at 8 NFE. The paper also provides analytical motivation, component ablations, entropy and repetition measurements, LLM-judge comparisons, and selected throughput benchmarks.

Critical strengths
  1. Compelling diagnosis and breakdown and meaningful ablations. I found the distinction between using a prediction for numerical integration versus future self-conditioning insightful. Table 1 provides particularly strong evidence: applying the correction only to the solver achieves 45.63 GenPPL, whereas also feeding it into self-conditioning increases GenPPL to 334.19. Additional controls in Appendices C.6–C.9 support the importance of the proposed correction direction and conditioning transform.

  2. Improvements seem substantial without retraining. The experiments preserve the pretrained checkpoint, sampling grid, and number of model evaluations. Improvements extend across multiple model sizes, datasets, and sampling budgets. The blind LLM-judge results provide useful evidence beyond GenPPL, and Table 14 demonstrates relatively modest runtime overhead for the evaluated ELF configurations.

Critical weaknesses
  1. I found that several GenPPL improvements coincide with lower unigram entropy and increased repetition. For example, on ELF-B at 8 NFE, entropy decreases from 5.233 to 5.074, moving farther from OpenWebText's reference entropy of approximately 5.4 (Table 3). For LangFlow on OpenWebText, word 4-gram repetition increases from 0.11 × 10^-3 to 1.90 × 10^-3 at 8 NFE (Table 5). As discussed by Zhang et al. (2026), GenPPL can reward repetition. I would like to see a quality–diversity frontier as reported in LDLM (Meshchaninov et al., 2026, Figure 1) and repetition measurements relative to real text to establish whether improvements persist at comparable diversity. Although the blind LLM-judge evaluation provides complementary evidence, I could not find the exact adapted evaluation prompt. The authors should include it to clarify how repetition and diversity are assessed and to make these results reproducible.

  2. Appendix B.2 shows that LangFlow's input projections are full rank, so the proposed geometry-aware conditioning transform reduces to uniform scaling. However, the key component ablation is performed on ELF-B, leaving the source of LangFlow's much larger gains less clear. Furthermore, the method uses model- and NFE-specific calibration statistics and hyperparameters. Although Table 14 reports throughput for ELF at 8 and 16 NFE, I found no corresponding LangFlow measurements, peak memory usage, or accounting for offline calibration cost. This would be needed to fully establish end-to-end computational advantages.

Questions for authors
  1. Can the authors provide GenPPL–entropy trade-off curves for the proposed and official samplers at fixed NFE, together with ground-truth 4-gram repetition rates? I would particularly like to know whether the GenPPL improvements persist when both methods operate near the reference entropy (approximately 5.4 on OpenWebText). LDLM (Meshchaninov et al., 2026, Figure 1) provides a useful precedent for presenting these quality–diversity frontiers. This experiment would materially affect my assessment of the generation-quality claim.

  2. Since the conditioning transform reduces to uniform scaling for LangFlow (Appendix B.2), could the authors report the contributions of uniform conditioning attenuation and the solver correction separately? An additional control using fixed rather than timestep specific correction statistics would clarify whether the proposed normalization and temporal adaptation are necessary.

  3. Could the authors extend Table 14 to LangFlow and specify the accelerator, batch size, latency, additional solver FLOPs, and peak HBM usage? Please also report the one-time cost of collecting calibration trajectories and clarify how the model/NFE-specific hyperparameters in Table 6 were selected, including whether calibration and evaluation samples were disjoint.

Comments on the paper

1 comment

Dear Reviewers,

The author-reviewer discussion phase is ending soon. If you have not done so yet, please read the author rebuttals and respond.

Please recall that, as a reviewer, you have agreed to engage in discussion with the authors to clarify and improve the paper. Failure to do so may result in penalties on your own submissions, as per the official ICLR policy. Given the high volume of submissions, your participation is crucial.

Thank you for your time and help.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.