- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work tackles the problem of incorporating token order into the training of one-step continuous flows of text. The motivation is that token order is critical in getting good generation quality, e.g., by using confidence-based samplers with masked diffusion models, and therefore the authors posit that it could improve performance even for one-step continuous flow LMs if incorporated during distillation. They introduce a general framework, asynchronous flow maps, which supports a vector of per-token times rather than a single scalar time , which may depend on the values of intermediate tokens during generation. They prove out this idea in small-scale settings on Sudoku and GSM8K, showing that (1) their framework enables sampling with confidence-based schedules using a model trained with diffusion forcing, and (2) even more interestingly, they find that using confidence-based per-token schedules during distillation outperforms simple scalar schedules when evaluated with one or few steps at test time.
Critical strengths
The experiment in Section 5.2 is compelling evidence that this idea works---in particular that more complex token-dependent schedules can improve one-step generation performance after distillation, even when diffusion forcing is used to train the teacher. This shows that the "axis" of asynchronous scheduling actually matters in the flow LM setting, where in some sense the whole point is to distillation followed by one- or few-step generation. I found it non-obvious that this would work, so this is a very cool result.
Critical weaknesses
The main weakness of the paper is that the experimental settings are quite toy. Could you train at a few larger compute budgets to demonstrate that your improvements in one- and few-step performance as in S5.2 don't diminish with scale? For example, I'm concerned that your compute scale is so low that FLMs are very poor at few-step generation, but this could change at somewhat larger scale and close the gap with FMLMs. Similarly it's not clear to me that EB versus Sync. during the distillation phase matters at larger scale.
Questions for authors
- See the critical weakness above for experiments I'd like to see run.
- Clarification: in the Section 5.1 Fig. 1 experiment, would it be possible to use a train-time schedule based on EB, and see if this improves performance beyond applying EB at inference-time only on top of diffusion forcing? Or is there some reason why you cannot use EB, but you are able to use the "alg. order" schedule during training?
4 comments
Rebuttal for Reviewer GesC [2/2]
Q2. Clarification: in the Section 5.1 Fig. 1 experiment, would it be possible to use a train-time schedule based on EB, and see if this improves performance beyond applying EB at inference-time only on top of diffusion forcing? Or is there some reason why you cannot use EB, but you are able to use the "alg. order" schedule during training?
Yes, there is a reason why EB is hard to use during training while the algorithmic order can be used. The difference comes from whether the schedule can be fixed before generation.
The algorithmic order (Alg. order) is a prescribed schedule (Equation (16)). It assigns each cell a predefined path of the tokenwise time from the round in which a rule-based solver resolves the cell (Appendix C and D.2), so the path does not depend on the denoiser prediction. Therefore, the states along this schedule can be sampled directly from the interpolant (Proposition 3), and the denoiser can be trained along it.
In contrast, EB is a state-dependent schedule, where the tokenwise times along the sampling path depend on the outputs of the denoiser. As we did with the algorithmic order, to train the denoiser only along the EB schedule we would need the states that EB actually visits. However, since the EB trajectory depends on the current state, obtaining these states requires simulating the scheduler many steps from the initial noise for every training sample, which is extremely costly. Moreover, even for the same sample, these states change throughout training because the denoiser is kept updated. As a result, the training distribution would keep shifting, which makes the training unstable.
For this reason, we instead first train the denoiser with diffusion forcing, which does not depend on the scheduler and, in principle, covers every state that the EB scheduler can visit. We then apply the EB scheduler to the trained denoiser at inference time and then distill it, which we believe is a practical way to bring the EB schedule into training, since the frozen teacher gives EB trajectories that no longer change.
I thank the authors for their rebuttal and larger-scale experiment at 422M parameters on TinyGSM. I am still a bit concerned that there are no standard pretraining experiments, such as compute-optimal pretraining on a corpus such as DCLM or FineWeb. But I think the paper is a reasonable proof-of-concept and therefore I will maintain my score recommending acceptance.
We thank the reviewer for acknowledging the additional experiment, and for the recommendation for acceptance.
Regarding the remaining concern, we agree that adding pretraining experiments and scaling analysis on the corpora that the reviewer mentioned would be a valuable addition to our work. However, such large-scale pretraining requires multi-node training over a long period, which is difficult to address within a single work that focuses on the methodological contribution. Therefore, as we explained in the previous response, we focused on the standard protocol at academic scale. Moreover, scaling studies of continuous language models have only begun very recently [1-3], so the experimental setup at this scale is not yet established and requires considerable effort. Nevertheless, we showed that the improvement of our method persists at a larger scale (Table A), and we leave the large-scale pretraining and compute-optimal scaling studies as future work.
We thank the reviewer again for the time and effort in the review process, and we would appreciate it if the reviewer could take these points into account for the final assessment.
[1] Yang et al., Continuous diffusion scales competitively with discrete diffusion for language, NeurIPS, 2026
[2] Yang et al., Large language continuous diffusion models, arXiv, 2026
[3] Chavan et al., Large flow language models, arXiv, 2026
Rebuttal for Reviewer GesC [1/2]
We appreciate the reviewer’s time and effort in the careful review. We address the weakness and the question below.
We thank the reviewer for this insightful comment. We respectfully note that we follow a standard setting in recent work for evaluating non-autoregressive language models at academic scale. In particular, we follow the setup of S-FLM [1] for Sudoku and the setup of [2] for TinyGSM, and other concurrent work also evaluates on the same settings [3, 4].
On the concern that few-step improvements may diminish with scale, we argue that larger scale would not fundamentally close the gap between the flow and the flow map. Flow matching learns from randomly coupled pairs of noise and data, so it learns the conditional mean of the velocity. As a result, its trajectory is curved in general [5], and approximating it with straight steps for numerical ODE solvers (e.g., Euler) induces error in the few-step regime. Importantly, these errors come from the learning objective and the data rather than from compute or capacity, meaning that scaling the model alone would not remove the gap between few-step and many-step generation of (a)FLMs. In contrast, flow map distillation directly learns the solution operator of the curved ODE, which optimizes for few-step performance. This explains the fundamental gap between aFLM and aFMLM. We remark that, even in the image domain, an 8B flow model still requires distillation for few-step sampling [6].
Nevertheless, we agree that results at a larger compute budget strengthen our claims. We therefore trained aFLM and aFMLM on TinyGSM with a medium-size transformer of 422M parameters, which is about 2.5 times larger than the 168M-parameter model in the paper.
Table A. aFLM and aFMLM on TinyGSM with the 422M transformer (accuracy, %).
Table A supports our claim. Even with a 2.5 times larger model, aFLM still fails in few-step generation under both schedulers, reaching at most 2.4% at 4 NFEs, while aFMLM (EB) reaches 14.1%. Moreover, the few-step accuracy of aFLM does not improve with longer training (Appendix D.3 in the updated manuscript), so in our experiments neither a larger model nor longer training closed the gap between the flow and the flow map.
Regarding EB versus Sync, it is true that the gap between EB and Sync would vanish with a perfect denoiser of infinite capacity. However, we argue that the approximation error of the denoiser remains at essentially all practical scales, as recent large diffusion language models such as LLaDA2.0 (100B) [7] and DiffusionGemma (26B) [8] still adopt confidence sampling that relies on approximation errors. Consistently, the gain from EB for aFLM persists with the 2.5 times larger model (32.4% vs. 7.2% at 32 NFEs in Table A). Furthermore, this gain transfers to aFMLM at this scale through the proposed distillation objective, where aFMLM (EB) achieves 14.1% at 4 NFEs while aFMLM (Sync) achieves 3.2%.
[1] Deschenaux et al., Language modeling with hyperspherical flows, TMLR, 2026
[2] Kim et al., Stop training for the worst: Progressive unmasking accelerates masked diffusion training, ICML, 2026
[3] Agarwal et al., Posterior refinement: Fast language-generation via any-order flow maps, arXiv, 2026
[4] Tang et al., Discrete Beckmann transport models for one-step language modeling and reasoning, arXiv, 2026
[5] Liu et al., Flow straight and fast: Learning to generate and transfer data with rectified flow, ICLR, 2023
[6] Sauer et al., Fast high-resolution image synthesis with latent adversarial diffusion distillation, arXiv, 2024
[7] Bie et al., LLaDA2.0: Scaling up diffusion language models to 100B, arXiv, 2025
[8] DiffusionGemma Team, DiffusionGemma technical report, arXiv, 2026