- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper interprets self-conditioning in continuous flow language models as a fixed-point iteration of the denoiser over its own prediction. Replacing the self-conditioning state with its fixed point yields “fixed-point flows”: an ordinary ODE whose velocity depends only on (t, x), which admits a well-defined flow map. Under this view, conventional self-conditioned sampling is a warm-started, single-iteration approximation of the fixed point. The authors show that ELF and LangFlow approximately learn fixed-point iterations, and distill ELF into a self-conditioning-free flow (ELF*) and a flow map (FMLM*), using either a two-stage (offline) or one-stage (online) recipe. FMLM* achieves strong few-step results on OpenWebText.
Critical strengths
- Treating self-conditioning as a fixed-point iteration turns it into an ordinary flow, which opens the door to flow-map distillation. FMLM* shows this leads to strong few-step quality.
- The fixed-point behavior isn’t specific to one model: the authors show it on LangFlow as well as ELF.
- Strong few-step results on OpenWebText.
- I appreciate offering both online and offline distillation, a practical choice between training cost and quality.
- The paper is clean, well structured, and coherent.
Critical weaknesses
- Table 3 (and online vs. offline in Table 1) compares FMLM* variants at different entropy levels, e.g., 73.86 / 5.35 vs. 80.26 / 5.44 at 4 steps. Since a small entropy drop can easily buy lower gPPL, these comparisons aren’t conclusive. Fig. 3 points the same way: the many-FPI frontier nearly matches warm-started single-FPI sampling, just at lower entropy. Frontier or matched-entropy comparisons would settle this.
- Per-sample entropy can’t detect reduced diversity across samples, which is a natural concern for fixed-point iteration and distillation. An inter-sample diversity metric (e.g., Self-BLEU) and a distributional metric (e.g., MAUVE) would help here.
- All experiments are unconditional. A conditional task, e.g. training on TinyGSM and evaluating on GSM8K, would show the approach goes beyond open-ended generation.
Questions for authors
- Does the benefit of more FPIs hold at matched entropy?
- Could you show FMLM* frontiers for different numbers of FPIs?
4 comments
Response to Reviewer s1dh (2/2)
W2. Per-sample entropy can’t detect reduced diversity across samples, which is a natural concern for fixed-point iteration and distillation. An inter-sample diversity metric (e.g., Self-BLEU) and a distributional metric (e.g., MAUVE) would help here.
We agree with the reviewer that per-sample entropy cannot detect reduced diversity across samples. Following the reviewer's suggestion, we measure MAUVE and Self-BLEU for the teacher across FPIs and for the distilled models.
As shown in Table B, MAUVE increases consistently with the number of FPIs (from 0.19 to 0.81 for ELF), while Self-BLEU moves from below that of the data to slightly above it. The change in Self-BLEU is small compared to the gain in MAUVE, suggesting that the gain from more FPIs does not come from a loss of diversity.
As shown in Table C, FMLM$^\star$ achieves a much higher MAUVE than all baselines at every step count, with Self-BLEU slightly above that of the data and well below that of FMLM. Some baselines have Self-BLEU below that of the data but low MAUVE, as does ELF with a single FPI in Table B. This suggests that their samples share fewer n-grams because they are less coherent rather than more diverse. Overall, fixed-point iteration and distillation bring the samples substantially closer to the data distribution in MAUVE while keeping Self-BLEU near that of the data.
We will add these results to Appendix D in the next revision.
Table B. MAUVE and Self-BLEU across FPIs with cold-start sampling (1024 samples; 32 steps for ELF and 64 for LangFlow).
| ELF | FPI=1 | FPI=2 | FPI=3 | FPI=4 | FPI=5 | Data |
|---|---|---|---|---|---|---|
| MAUVE | 0.192 | 0.467 | 0.691 | 0.764 | 0.807 | 1.000 |
| Self-BLEU | 0.245 | 0.267 | 0.290 | 0.305 | 0.315 | 0.274 |
| LangFlow | FPI=1 | FPI=2 | FPI=3 | FPI=4 | FPI=5 | Data |
|---|---|---|---|---|---|---|
| MAUVE | 0.851 | 0.919 | 0.940 | 0.953 | 0.956 | 1.000 |
| Self-BLEU | 0.270 | 0.281 | 0.294 | 0.300 | 0.306 | 0.282 |
Table C. MAUVE and Self-BLEU at 1 / 2 / 4 sampling steps for the models in Table 1 (1024 samples).
| 1 / 2 / 4 steps | MAUVE | Self-BLEU |
|---|---|---|
| Duo + DCD | 0.055 / 0.063 / 0.150 | 0.060 / 0.156 / 0.271 |
| Duo + Di4C | 0.054 / 0.075 / 0.166 | 0.406 / 0.470 / 0.418 |
| MDLM + SDTT | 0.053 / 0.057 / 0.090 | 0.200 / 0.191 / 0.265 |
| MDLM + Di4C | 0.053 / 0.057 / 0.107 | 0.196 / 0.205 / 0.292 |
| FMLM | 0.078 / 0.165 / 0.351 | 0.401 / 0.397 / 0.399 |
| FMLM$^\star$ (offline) | 0.692 / 0.884 / 0.865 | 0.322 / 0.324 / 0.349 |
| FMLM$^\star$ (online) | 0.679 / 0.883 / 0.882 | 0.312 / 0.316 / 0.339 |
| Data | 1.000 | 0.282 |
W3. All experiments are unconditional. A conditional task, e.g. training on TinyGSM and evaluating on GSM8K, would show the approach goes beyond open-ended generation.
As the reviewer suggested, we evaluate our method on a conditional generation task. We train an ELF teacher on TinyGSM, distill it into FMLM$^\star$ with online distillation (8 FPIs), and evaluate both zero-shot on GSM8K. As shown in Table D, FMLM$^\star$ outperforms the teacher and baselines at every matched step count. This suggests that our distillation extends to conditional generation.
We have added this result to Appendix D.4 (Table 7) of the revised manuscript.
Table D. GSM8K accuracy (%) of models trained on TinyGSM, following the setting of [1, 2, 3, 4]. The teacher uses warm-start sampling. denotes results taken from the original papers.
| Sampling steps | 1 | 2 | 4 |
|---|---|---|---|
| FMLM+ (Init) [3] | 0.3 | 0.7 | 5.1 |
| MDLM$^\dagger$ [3] | 0.0 | 0.0 | 0.1 |
| ELF (teacher) | 0.08 | 0.15 | 1.82 |
| FMLM$^\star$ (ours) | 1.14 | 2.96 | 6.22 |
[1] Kim et al. "Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training." 2026.
[2] Deschenaux and Gulcehre. "Language Modeling with Hyperspherical Flows." 2026.
[3] Agarwal et al. "Posterior Refinement: Fast Language Generation via Any-Order Flow Maps." 2026.
[4] Hu et al. "ELF: Embedded Language Flows." 2026.
Thank you for the detailed response and for running the additional experiments.
Thank you for the detailed response and for running the additional experiments. I especially appreciate the GSM8K evaluation, which directly addresses my concern about conditional generation.
I still have a question about the matched-entropy comparison in Table 4. More fixed-point iterations lower gPPL at an entropy of 5.30. Could the authors explain why they chose this value and whether the improvement holds closer to the OpenWebText data entropy of 5.44?
I would also qualify the claim about diversity. MAUVE improves with more iterations, but the samples also share more phrases. For ELF, Self-BLEU rises from 0.245 to 0.315, above the real-data value of 0.274. This suggests that the improvement may come with some loss of lexical diversity across samples. Overall, I’ve decided to keep my original rating.
We thank the reviewer for the follow-up, and we are glad that the GSM8K evaluation addressed the concern about conditional generation.
Matched entropy. The entropy value 5.30 came from one of our sampling configurations used during development, which we set as a reference with no particular intent.
Following the reviewer's request, we repeat the comparison at the data entropy of 5.44 (Table A). We find that the trend in Table 4 holds at entropy 5.44, where the gPPL improves with every additional FPI for both models.
Lexical diversity. We agree that more FPIs may come with some loss of lexical diversity, as Self-BLEU increases beyond that of the data. Yet, the MAUVE score, which measures distributional diversity with respect to true data, improves consistently with every additional FPI, in line with the trend in Table A. As such, we believe that although some lexical phrases are shared, the diversity of the sample distribution is not substantially reduced.
In the final version, we will add the comparison at 5.44 to Appendix D.2 and distinguish lexical diversity across samples (Self-BLEU) from diversity measured by MAUVE in our claim on diversity.
Table A. gPPL at a matched entropy of 5.44 (the data entropy). The samples are generated with 32 steps for ELF and 64 for LangFlow, without damping. Warm start with a single FPI corresponds to conventional self-conditioned sampling.
| Warm start, 1 FPI | Cold start, 1 FPI | 2 FPIs | 3 FPIs | 4 FPIs | 5 FPIs | |
|---|---|---|---|---|---|---|
| ELF | 67.50 | 99.77 | 67.86 | 64.01 | 63.80 | 63.36 |
| LangFlow | 69.97 | 108.69 | 83.21 | 74.26 | 68.25 | 64.71 |
Response to Reviewer s1dh (1/2)
We thank the reviewer for this comment and agree that matched-entropy comparisons are more conclusive. Following the reviewer's suggestion, we plot the gPPL-entropy frontiers of online FMLM$^\star$ with different numbers of FPIs and of offline FMLM$^\star$ at one, two, and four steps, sweeping the self-conditioning guidance weight at inference. Where the frontiers overlap, they allow a comparison at matched entropy. The number of FPIs refers to the teacher evaluations in online distillation, and FMLM$^\star$ uses no FPIs at inference (Appendix C.3).
As shown in Figure 12 in the revised Appendix D.3, with four steps, where the frontiers overlap over a wide range of entropy, more FPIs give lower gPPL at the same entropy. Offline FMLM$^\star$ achieves the lowest gPPL, and the frontier of online FMLM$^\star$ approaches it as the number of FPIs grows. With one and two steps, the guidance sweep changes entropy only slightly, so the frontiers overlap less, but they show the same trend.
In the revised manuscript, we have added these frontiers to Appendix D.3 (Figure 12) and will revise the comparison between online and offline FMLM$^\star$ in Section 4.2 to refer to them.
As the reviewer suggested, we compare generation quality across numbers of FPIs at a matched entropy of 5.30 (Table A). We use cold start, where the iteration at every step starts from zero, so that the number of FPIs counts all iterations at each step, as in our distillation (Equation (18)). We also include conventional self-conditioned sampling, which uses warm start with a single FPI, as a reference.
As shown in Table A, gPPL with cold start decreases consistently as the number of FPIs increases for both models, so the benefit of more FPIs holds at a matched entropy. Conventional sampling reaches a gPPL close to that of cold start with three FPIs, consistent with the reviewer's observation on Figure 3, while many FPIs still achieve a lower gPPL. We attribute this to the initialization provided by warm start (Section 3.3): the estimate from the previous flow step initializes the iteration close to the fixed point, so that fewer FPIs are needed (Appendix B.5).
We have added this comparison to Appendix D.2 (Table 4) of the revised manuscript and will clarify in Section 4.1 that Figure 3 compares the two initializations.
Table A. gPPL at a matched entropy of 5.30 (32 steps for ELF and 64 for LangFlow, without damping). Warm start with a single FPI corresponds to conventional self-conditioned sampling.