acceptodds
Under review as a conference paper at ICLR 2027

Asynchronous Flow Maps for Flow Language Models

10 views

Abstract

The choice of how data elements are ordered and scheduled for modeling has a profound impact on deep generative models. This effect is pronounced in categorical data such as language and code with procedural underlying generative processes. As such, discrete methods, such as autoregressive models and masked diffusion models, can naturally improve by exploiting these orderings during training and inference. However, it is less clear whether this effect can be incorporated into continuous flows for language and categorical data, which have a distinguished advantage of efficient inference through flow map distillation. We introduce asynchronous flows and flow maps, a framework for leveraging token orderings in flow language models by moving each token at a different speed with either prescribed or learned schedules, along with their mathematical formulation and distillation objectives. The resulting asynchronous flow maps are capable of one-step generation while harnessing the benefit of ordered generation by internalizing the procedure into a single forward pass. We train asynchronous flows and flow maps on Sudoku solving, and mathematical coding tasks and achieve state-of-the-art one-step performance.

Primary area:
generative models
Keywords:
Flow language model, flow maps, diffusion language model, few-step generative modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GesCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work tackles the problem of incorporating token order into the training of one-step continuous flows of text. The motivation is that token order is critical in getting good generation quality, e.g., by using confidence-based samplers with masked diffusion models, and therefore the authors posit that it could improve performance even for one-step continuous flow LMs if incorporated during distillation. They introduce a general framework, asynchronous flow maps, which supports a vector of per-token times τ∈RL\tau \in \mathbb{R}^L rather than a single scalar time t∈Rt \in \mathbb{R}, which may depend on the values of intermediate tokens during generation. They prove out this idea in small-scale settings on Sudoku and GSM8K, showing that (1) their framework enables sampling with confidence-based schedules using a model trained with diffusion forcing, and (2) even more interestingly, they find that using confidence-based per-token schedules during distillation outperforms simple scalar schedules when evaluated with one or few steps at test time.

Critical strengths

The experiment in Section 5.2 is compelling evidence that this idea works---in particular that more complex token-dependent schedules can improve one-step generation performance after distillation, even when diffusion forcing is used to train the teacher. This shows that the "axis" of asynchronous scheduling actually matters in the flow LM setting, where in some sense the whole point is to distillation followed by one- or few-step generation. I found it non-obvious that this would work, so this is a very cool result.

Critical weaknesses

The main weakness of the paper is that the experimental settings are quite toy. Could you train at a few larger compute budgets to demonstrate that your improvements in one- and few-step performance as in S5.2 don't diminish with scale? For example, I'm concerned that your compute scale is so low that aaFLMs are very poor at few-step generation, but this could change at somewhat larger scale and close the gap with aaFMLMs. Similarly it's not clear to me that EB versus Sync. during the distillation phase matters at larger scale.

Questions for authors
  • See the critical weakness above for experiments I'd like to see run.
  • Clarification: in the Section 5.1 Fig. 1 experiment, would it be possible to use a train-time schedule based on EB, and see if this improves performance beyond applying EB at inference-time only on top of diffusion forcing? Or is there some reason why you cannot use EB, but you are able to use the "alg. order" schedule during training?
4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer GesC [1/2]

We appreciate the reviewer’s time and effort in the careful review. We address the weakness and the question below.

W1. The main weakness of the paper is that the experimental settings are quite toy. [...] For example, I'm concerned that your compute scale is so low that FLMs are very poor at few-step generation [...] Similarly it's not clear to me that EB versus Sync. during the distillation phase matters at larger scale..

We thank the reviewer for this insightful comment. We respectfully note that we follow a standard setting in recent work for evaluating non-autoregressive language models at academic scale. In particular, we follow the setup of S-FLM [1] for Sudoku and the setup of [2] for TinyGSM, and other concurrent work also evaluates on the same settings [3, 4].

On the concern that few-step improvements may diminish with scale, we argue that larger scale would not fundamentally close the gap between the flow and the flow map. Flow matching learns from randomly coupled pairs of noise and data, so it learns the conditional mean of the velocity. As a result, its trajectory is curved in general [5], and approximating it with straight steps for numerical ODE solvers (e.g., Euler) induces error in the few-step regime. Importantly, these errors come from the learning objective and the data rather than from compute or capacity, meaning that scaling the model alone would not remove the gap between few-step and many-step generation of (a)FLMs. In contrast, flow map distillation directly learns the solution operator of the curved ODE, which optimizes for few-step performance. This explains the fundamental gap between aFLM and aFMLM. We remark that, even in the image domain, an 8B flow model still requires distillation for few-step sampling [6].

Nevertheless, we agree that results at a larger compute budget strengthen our claims. We therefore trained aFLM and aFMLM on TinyGSM with a medium-size transformer of 422M parameters, which is about 2.5 times larger than the 168M-parameter model in the paper.

Table A. aFLM and aFMLM on TinyGSM with the 422M transformer (accuracy, %).

Method \ NFE12481632
aFLM (Sync)0.20.42.44.96.17.2
aFLM (EB)0.20.41.08.125.532.4
aFMLM (Sync)0.91.83.24.45.46.4
aFMLM (EB)7.010.114.116.722.528.2

Table A supports our claim. Even with a 2.5 times larger model, aFLM still fails in few-step generation under both schedulers, reaching at most 2.4% at 4 NFEs, while aFMLM (EB) reaches 14.1%. Moreover, the few-step accuracy of aFLM does not improve with longer training (Appendix D.3 in the updated manuscript), so in our experiments neither a larger model nor longer training closed the gap between the flow and the flow map.

Regarding EB versus Sync, it is true that the gap between EB and Sync would vanish with a perfect denoiser of infinite capacity. However, we argue that the approximation error of the denoiser remains at essentially all practical scales, as recent large diffusion language models such as LLaDA2.0 (100B) [7] and DiffusionGemma (26B) [8] still adopt confidence sampling that relies on approximation errors. Consistently, the gain from EB for aFLM persists with the 2.5 times larger model (32.4% vs. 7.2% at 32 NFEs in Table A). Furthermore, this gain transfers to aFMLM at this scale through the proposed distillation objective, where aFMLM (EB) achieves 14.1% at 4 NFEs while aFMLM (Sync) achieves 3.2%.


[1] Deschenaux et al., Language modeling with hyperspherical flows, TMLR, 2026

[2] Kim et al., Stop training for the worst: Progressive unmasking accelerates masked diffusion training, ICML, 2026

[3] Agarwal et al., Posterior refinement: Fast language-generation via any-order flow maps, arXiv, 2026

[4] Tang et al., Discrete Beckmann transport models for one-step language modeling and reasoning, arXiv, 2026

[5] Liu et al., Flow straight and fast: Learning to generate and transfer data with rectified flow, ICLR, 2023

[6] Sauer et al., Fast high-resolution image synthesis with latent adversarial diffusion distillation, arXiv, 2024

[7] Bie et al., LLaDA2.0: Scaling up diffusion language models to 100B, arXiv, 2025

[8] DiffusionGemma Team, DiffusionGemma technical report, arXiv, 2026

AuthorsOpenReview ↗

Rebuttal for Reviewer GesC [2/2]

Q2. Clarification: in the Section 5.1 Fig. 1 experiment, would it be possible to use a train-time schedule based on EB, and see if this improves performance beyond applying EB at inference-time only on top of diffusion forcing? Or is there some reason why you cannot use EB, but you are able to use the "alg. order" schedule during training?

Yes, there is a reason why EB is hard to use during training while the algorithmic order can be used. The difference comes from whether the schedule can be fixed before generation.

The algorithmic order (Alg. order) is a prescribed schedule (Equation (16)). It assigns each cell a predefined path of the tokenwise time from the round in which a rule-based solver resolves the cell (Appendix C and D.2), so the path does not depend on the denoiser prediction. Therefore, the states along this schedule can be sampled directly from the interpolant (Proposition 3), and the denoiser can be trained along it.

In contrast, EB is a state-dependent schedule, where the tokenwise times along the sampling path depend on the outputs of the denoiser. As we did with the algorithmic order, to train the denoiser only along the EB schedule we would need the states that EB actually visits. However, since the EB trajectory depends on the current state, obtaining these states requires simulating the scheduler many steps from the initial noise for every training sample, which is extremely costly. Moreover, even for the same sample, these states change throughout training because the denoiser is kept updated. As a result, the training distribution would keep shifting, which makes the training unstable.

For this reason, we instead first train the denoiser with diffusion forcing, which does not depend on the scheduler and, in principle, covers every state that the EB scheduler can visit. We then apply the EB scheduler to the trained denoiser at inference time and then distill it, which we believe is a practical way to bring the EB schedule into training, since the frozen teacher gives EB trajectories that no longer change.

I thank the authors for their rebuttal and larger-scale experiment at 422M parameters on TinyGSM. I am still a bit concerned that there are no standard pretraining experiments, such as compute-optimal pretraining on a corpus such as DCLM or FineWeb. But I think the paper is a reasonable proof-of-concept and therefore I will maintain my score recommending acceptance.

We thank the reviewer for acknowledging the additional experiment, and for the recommendation for acceptance.

Regarding the remaining concern, we agree that adding pretraining experiments and scaling analysis on the corpora that the reviewer mentioned would be a valuable addition to our work. However, such large-scale pretraining requires multi-node training over a long period, which is difficult to address within a single work that focuses on the methodological contribution. Therefore, as we explained in the previous response, we focused on the standard protocol at academic scale. Moreover, scaling studies of continuous language models have only begun very recently [1-3], so the experimental setup at this scale is not yet established and requires considerable effort. Nevertheless, we showed that the improvement of our method persists at a larger scale (Table A), and we leave the large-scale pretraining and compute-optimal scaling studies as future work.

We thank the reviewer again for the time and effort in the review process, and we would appreciate it if the reviewer could take these points into account for the final assessment.


[1] Yang et al., Continuous diffusion scales competitively with discrete diffusion for language, NeurIPS, 2026

[2] Yang et al., Large language continuous diffusion models, arXiv, 2026

[3] Chavan et al., Large flow language models, arXiv, 2026

Reviewer_kiFhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Asynchronous Flow Maps for Flow Language Models (aFMLM), a framework for distilling asynchronous flow-based language generation into a one-step model. The method combines token-wise asynchronous scheduling with flow map distillation by jointly modeling token states and their corresponding denoising times. The proposed training objective includes denoiser supervision, scheduler supervision, and flow-map consistency losses. Experiments on Sudoku and GSM8K demonstrate that asynchronous scheduling substantially improves multi-step generation. The proposed aFMLM also achieves higher one-step accuracy than existing flow-map language model baselines.

Critical strengths
  1. The paper addresses an important problem: distilling state-dependent asynchronous generation dynamics into a one-step model. Extending the flow map state to include token-wise denoising times is mathematically sound.

  2. The proposed method consistently outperforms conventional flow map language models, particularly on challenging reasoning tasks.

Critical weaknesses
  1. Prior work has explored the main components of the proposed framework, including flow map distillation, token-wise asynchronous scheduling, and confidence-based scheduling strategies. The primary contribution is extending the flow map formulation to jointly model token states and tokenwise denoising times. While technically reasonable, this appears to be an incremental extension of existing methods rather than a fundamentally new distillation approach.

  2. Although the paper compares synchronous and asynchronous distillation, these experiments do not isolate the contribution of jointly predicting tokenwise denoising times. In particular, it remains unclear whether the improvements arise primarily from the stronger asynchronous teacher trajectories or from the proposed joint-state flow map formulation.

  3. Despite improvements over existing one-step baselines, the distilled model remains substantially weaker than multi-step generation methods. For example, on Sudoku-Hard, the asynchronous teacher achieves 95.6% accuracy with 32 NFEs, whereas the proposed one-step student achieves only 15.9%. Even increasing the student to four NFEs substantially improves accuracy to 65.1%, suggesting the distilled model still relies heavily on iterative computation. This large performance gap is particularly concerning given the paper's objective of compressing asynchronous generation into a single forward pass. Successful one-step and few-step distillation methods in image generation have shown that much smaller performance gaps are achievable, although the tasks differ. The paper does not sufficiently investigate whether the observed degradation results from limitations of the distillation objective, model capacity, or the inherently sequential nature of the evaluated reasoning tasks.

Questions for authors
  1. What is the specific methodological contribution beyond combining existing flow map distillation and asynchronous scheduling? Could the authors clarify which aspects of the proposed formulation existing distillation techniques cannot achieve?

  2. Is joint tokenwise-time prediction necessary for the observed improvements? Can the authors provide an ablation that distills the same asynchronous teacher without explicitly predicting token-wise denoising times? This would help distinguish the benefits of asynchronous teacher trajectories from those of the proposed joint-state formulation.

  3. Why is the performance gap between the multi-step teacher and the one-step student so large? For example, Sudoku-Hard accuracy decreases from 95.6% to 15.9%. Can the authors analyze the sources of this degradation and explain why increasing the number of student inference steps produces such substantial improvements?

5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer kiFh [1/3]

We appreciate the reviewer’s careful review of our work. We address each weakness and question below.

W1. Prior work has explored the main components of the proposed framework, including flow map distillation, token-wise asynchronous scheduling, and confidence-based scheduling strategies. The primary contribution is extending the flow map formulation to jointly model token states and tokenwise denoising times. While technically reasonable, this appears to be an incremental extension of existing methods rather than a fundamentally new distillation approach.

Q1. What is the specific methodological contribution beyond combining existing flow map distillation and asynchronous scheduling? Could the authors clarify which aspects of the proposed formulation existing distillation techniques cannot achieve?

While confidence-based asynchronous sampling and flow map distillation have been studied independently (Section 4), simply combining these methods cannot produce our best method that distills state-dependent schedules (Fig. 1). This is because, under these schedules, the tokenwise times depend on the token states along the trajectory and are not known in advance. To distill such a schedule, we first have to express the token states and the tokenwise times as one joint dynamics, so that the trajectory is a deterministic flow for which a flow map can be defined. The student then has to predict the tokenwise times together with the token states, since the tokenwise times at the target time cannot be computed without following the trajectory. Therefore, our formulation and the proposed method are not incremental design choices but a mathematical necessity for distilling state-dependent schedules. This formulation is new and has not been proposed in prior work as far as we are aware. We elaborate on the details below.

A flow map over joint dynamics jumps from base time ss to an arbitrary later base time tt, and how far each token moves during this jump is determined by how far its tokenwise time advances. We have proven that these tokenwise times are given by the integral of the scheduler,

τt=τs+∫stωr(xr,τr) dr.\tau_t=\tau_s+\int_s^t \omega_r({\bf x}_r,\tau_r)\,{\rm d}r .

For state-dependent schedules (Equation (15)), the scheduler depends on the token states xr{\bf x}_r that change along the trajectory, so this integral can be obtained only by following the trajectory. Therefore, to jump over the trajectory, we train the student to predict the tokenwise times at tt in addition to the token states, from the base times s,ts,t and the joint state (xs,τs)({\bf x}_s,\tau_s), using an extension of the flow map loss. This requires adding a separate time head to predict a scalar time value for each token. Existing flow map distillation methods do not predict the tokenwise times, so they cannot distill state-dependent schedules, which we also show empirically in our response to W2/Q2.

AuthorsOpenReview ↗

Rebuttal for Reviewer kiFh [2/3]

W2. Although the paper compares synchronous and asynchronous distillation, these experiments do not isolate the contribution of jointly predicting tokenwise denoising times. In particular, it remains unclear whether the improvements arise primarily from the stronger asynchronous teacher trajectories or from the proposed joint-state flow map formulation.

Q2. Is joint tokenwise-time prediction necessary for the observed improvements? Can the authors provide an ablation that distills the same asynchronous teacher without explicitly predicting token-wise denoising times? This would help distinguish the benefits of asynchronous teacher trajectories from those of the proposed joint-state formulation.

As explained in our response to W1/Q1, a state-dependent schedule cannot be distilled without predicting the tokenwise times, so the requested ablation does not have an exact counterpart. Nevertheless, we construct the closest approximation, which computes τt\tau_t by keeping the infinitesimal rate of the aFLM scheduler ωs(xs,τs)\omega_s({\bf x}_s,\tau_s) fixed during each jump, in both distillation and generation. This approximation is incorrect since the rate changes along the trajectory. For example, one jump from ss to tt and two jumps through uu reach different tokenwise times, since the second jump uses the rate at uu. Therefore, the self-consistency target asks the student to match token states that are reached at different tokenwise times, which is inconsistent.

We verify this by comparing aFMLM (EB) with the variant that is distilled without the time head as described above, together with aFMLM (Sync), in Table A and Table B. With the time head, aFMLM (EB) achieves 4.1% at 1 step on GSM8K, while aFMLM (EB) without the time head achieves only 0.4%. Moreover, the student without the time head even falls below aFMLM (Sync) at 2 and 4 steps, although it is distilled from the stronger asynchronous teacher. Thus, the few-step improvements of aFMLM do not come from the asynchronous teacher alone but from correctly distilling the state-dependent trajectory, and the time head is necessary for this.

Table A Ablation of the time head on Sudoku (exact match, %).

Sudoku-Hard \ NFE124
aFMLM (Sync)9.025.941.6
aFMLM (EB, without time head)11.925.733.4
aFMLM (EB, with time head)15.949.374.3

Table B. Ablation of the time head on GSM8K (accuracy, %).

GSM8K \ NFE124
aFMLM (Sync)0.20.61.3
aFMLM (EB, without time head)0.40.51.1
aFMLM (EB, with time head)4.15.57.1
AuthorsOpenReview ↗

Rebuttal for Reviewer kiFh [3/3]

W3. Despite improvements over existing one-step baselines, the distilled model remains substantially weaker than multi-step generation methods.[...] Even increasing the student to four NFEs substantially improves accuracy to 65.1%, suggesting the distilled model still relies heavily on iterative computation. [...] The paper does not sufficiently investigate whether the observed degradation results from limitations of the distillation objective, model capacity, or the inherently sequential nature of the evaluated reasoning tasks.

Q3. Why is the performance gap between the multi-step teacher and the one-step student so large? Can the authors analyze the sources of this degradation and explain why increasing the number of student inference steps produces such substantial improvements?

We thank the reviewer for the feedback. We first explain why one-step image models are not a direct reference point, and then we analyze where the gap comes from on Sudoku-Hard.

First, the small gaps of recent one-step image models are the result of several years of engineering. Early one-step distillation methods also showed large gaps. For example, on ImageNet 64×64, progressive distillation [1] reaches an FID of 15.39 with one step, while the EDM teacher reaches 2.44 with 79 NFEs [2]. Moreover, much of the later progress relies on components that are specific to images, such as LPIPS losses that use features of networks pretrained on ImageNet [2, 3] and adversarial losses that use image diffusion models [4]. These techniques do not transfer directly to discrete data due to its categorical nature and large dimensionality, as studied in FMLM [5]. Thus, one-step language modeling is still at an early stage, and most of this work appeared this year [5, 6].

Second, we agree with the reviewer’s comment that our paper does not separate the possible sources of the gap. We find that the gap mainly comes from the depth of the current model, not from the proposed distillation objective, and that it shrinks consistently as the model gets deeper. Puzzles like Sudoku-Hard require sequential reasoning steps that propagate the constraints, as shown by the results of Alg. order in Fig. 1, where solving the puzzles in a clue-dependent algorithmic order achieves better performance than synchronous denoising. However, compressing such sequential computation into a single parallel forward pass is known to be difficult [7], especially when the depth of the model is limited, since the model has to internalize these sequential steps into its hidden computation without layer-wise supervision [8]. To support this, we show the one-step aFMLM (EB) results on Sudoku-Hard where we scale only the depth of the model.

Table C Effect of model depth on Sudoku-Hard (exact match, %).

Num. Transformer blocksaFMLM (EB), One stepaFLM (EB), 32 steps
8 (original)15.995.6
1636.694.1
2464.096.6

As Table C shows, scaling only the depth reduces the gap between one-step and many-step generation to 32.6 points. Hence, the gap mainly comes from the limited depth for internalizing the sequential computation that the task requires, and we expect it to shrink further as the model scales. This also explains why more student steps help so much, since each additional forward pass increases the effective depth of the computation.

Apart from the depth, the proposed distillation objective itself successfully distills the teacher trajectory. We note that the 65.1% that the reviewer cites is the accuracy of the teacher with 4 NFEs, and aFMLM (EB) reaches a higher accuracy of 74.3% with the same 4 NFEs (Figure 2). In contrast, without our proposed objective, the gap is much larger, since the student reaches only 33.4% with 4 NFEs under the same teacher and depth (Table A).


[1] Salimans and Ho, Progressive distillation for fast sampling of diffusion models, ICLR, 2022

[2] Song et al., Consistency models, ICML, 2023

[3] Kim et al., Consistency trajectory models: Learning probability flow ODE trajectory of diffusion, ICLR, 2024

[4] Yin et al., Improved distribution matching distillation for fast image synthesis, NeurIPS, 2024

[5] Lee et al., Flow map language models: One-step language modeling via continuous denoising, NeurIPS, 2026

[6] Roos et al., Categorical flow maps, ICML, 2026

[7] Liu et al., Serial scaling hypothesis, ICLR, 2026

[8] Yang et al., Do large language models latently perform multi-hop reasoning?, ACL, 2024

Reviewer_CdvHOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes aFMLM, a framework for combining diffusion forcing with continuous diffusion language models and distilling the resulting multi-step models into few-step generators. The main contribution of this paper is to combine the confidence-based sampling in masked diffusion to the flow map distillation in continuous diffusion and validate it on math and sudoku benchmarks.

Critical strengths

1, It is interesting to see that confidence based tokenwise time scheduling can help the sample quality of continuous diffusion language models.

2, The framework is sound and consistent.

3, Empirical results for the proposed method and the included baselines show the method's superiority clearly.

Critical weaknesses

1, The novelty is limited. The key method combines 1) tokenwise confidence based asynchronous sampling of diffusion language models and 2) distillation, while both of them originates from existing methods. The only novel design is the predicted head for the κ\kappa. However, no experiments show the necessity or the advantage of using this head in aFMLM instead of the original scheduler in aFLM, nor explanation.

2, Key ablations are missing. Although aFMLM achieves state-of-the-art performance, there is no result about the performance of aFLM before distillation. Hence, it is not clear whether the gain originates from the stronger performance of the teacher with a better sampling method or the ease of distillation.

3, Missing baselines. There are also existing frameworks of continuous DLMs [1, 2, 3] that should have be included in the comparison. Also, whether this method can generalize to these frameworks is also unclear.

Reference:

[1] Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

[2] LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

[3] ELF: Embedded Language Flows

Questions for authors

1, Can you include the result of aFLM under the multi-step generation setting for comparison with FLM? This is critical to demonstrate where the advantage of aFMLM comes from.

2, I wonder whether a high-confidence districts could change back to lower-confidence ones. Can you show me quantitative results?

3, The proposed method resembles the confidence-based sampling used in masked diffusion. However, this kind of sampling will hurt the data diversity and then the potential of post training in masked diffusion. I wonder if the author could report some intuitive metrics like entropy to analyze the impact of aFLM sampling to the data diversity.

7 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer CdvH [1/4]

We thank the reviewer for the thoughtful and detailed feedback. Below, we respond to each of the weaknesses and questions.

W1. The novelty is limited. The key method combines 1) tokenwise confidence based asynchronous sampling of diffusion LMs and 2) distillation, while both of them originate from existing methods. The only novel design is the predicted head for the κ\kappa. However, no experiments show the necessity or the advantage of using this head in aFMLM instead of the original scheduler in aFLM, nor explanation.

While confidence-based asynchronous sampling and flow map distillation have been studied independently (Section 4), we would like to clarify that simply combining these methods cannot produce our best method that distills state-dependent schedules (Figure 1). This is because the tokenwise time now has its own flow map, so it is mathematically necessary to have a separate prediction head for tokenwise time. This formulation is new and hasn’t been proposed in prior work as far as we are aware. The specific parameterization with κ\kappa in equation (28) is an added novelty on top of this already novel formulation. We elaborate on the details below.

The time head is required by formulation. A flow map jumps from time ss to tt, and in the asynchronous setting, how far each token moves during this jump is determined by how far its tokenwise time advances. We have proven that these tokenwise times are given by the integral of the scheduler, τt=τs+∫stωr(xr,τr) dr.\tau_t=\tau_s+\int_s^t \omega_r({\bf x}_r,\tau_r)\,{\rm d}r . For state-dependent schedules (equation (15)), the scheduler depends on the token states xr{\bf x}_r that change along the trajectory, so this integral can be obtained only by following the trajectory. Therefore, to jump over the trajectory, we train the student to predict the tokenwise times at t in addition to the token states, from the base times ss,tt and the joint state (xs,τs)({\bf x}_s,\tau_s), using an extension of flow map loss. This requires adding a separate time head to predict a scalar time value for each token.

The original aFLM scheduler cannot replace the head. The aFLM scheduler cannot give correct tokenwise times since it outputs only the infinitesimal rate ωs(xs,τs)\omega_s({\bf x}_s,\tau_s) at the start of each jump. Therefore, the closest way to distill state-dependent schedules without the time head is to compute the tokenwise times at tt by keeping this rate fixed during the jump, in both distillation and generation. However, this is incorrect since the rate changes along the trajectory. For example, one jump from ss to tt and two jumps through uu reach different tokenwise times, since the second jump uses the rate at uu. Therefore, the self-consistency target asks the student to match token states that are reached at different tokenwise times, which is mathematically wrong.

The time head is empirically crucial. As requested, we show in Table A and Table B that distillation without the time head fails to distill the state-dependent schedule and degrades the few-step generation quality.

Table A Ablation of the time head on Sudoku (exact match, %).

Sudoku-Hard \ NFE124
aFMLM (Sync)9.025.941.6
aFMLM (EB, without time head)11.925.733.4
aFMLM (EB, with time head)15.949.374.3

Table B. Ablation of the time head on GSM8K (accuracy, %).

GSM8K \ NFE124
aFMLM (Sync)0.20.61.3
aFMLM (EB, without time head)0.40.51.1
aFMLM (EB, with time head)4.15.57.0

With the time head, aFMLM (EB) achieves 4.1% at 1 step, while the approximation with the aFLM scheduler achieves only 0.4%. Moreover, the approximation even falls below aFMLM (Sync) at 2 and 4 steps. Thus, without the time head, the student loses the advantage of the asynchronous teacher, which confirms that the time head is necessary to distill state-dependent schedules.

AuthorsOpenReview ↗

Rebuttal for Reviewer CdvH [2/4]

W2. Key ablations are missing. Although aFMLM achieves state-of-the-art performance, there is no result about the performance of aFLM before distillation. Hence, it is not clear whether the gain originates from the stronger performance of the teacher with a better sampling method or the ease of distillation.

Q1. Can you include the result of aFLM under the multi-step generation setting for comparison with FLM? This is critical to demonstrate where the advantage of aFMLM comes from.

We kindly note that Fig. 1 already shows the performance of aFLM before distillation across two datasets, multi-step generation, and the choice of schedulers. Nonetheless, we report the exact numbers in Table C, together with the multi-step results of FLM as requested. The names in parentheses denote the schedulers applied to aFLM at test-time, as in our experiment section.

Table C. Multi-step results of FLM and aFLM under different schedulers on GSM8K (accuracy, %).

GSM8k12481632
FLM (Sync)0.00.10.71.51.82.5
aFLM (Sync)0.00.11.22.63.04.0
aFLM (EB)0.00.00.44.918.324.6

As shown in Table C, aFLM paired with our proposed state-dependent scheduler (EB) substantially outperforms FLM at 32 NFEs (24.6% vs. 2.5%), whereas aFLM with FLM-like scheduler (Sync) only marginally outperforms FLM (4.0% vs. 2.5%).

The advantage of aFMLM does come from the stronger performance of the teacher with a better sampling method. Our Table 1 in the main text shows that, after flow map distillation, the relative competences between aFLM (EB), aFLM (Sync), and FLM at 32 NFEs (Figure 1 and Table C) indeed translates to the performance of the respectively distilled aFMLM (EB), aFMLM (Sync), and FMLM at one-step generation. We remark that since EB is state-dependent, its distillation is only possible through our novel asynchronous flow map formulation with the time head, as explained in our response to W1. This is the main contribution of our work, which is not achievable with existing flow map distillation methods.

AuthorsOpenReview ↗

Rebuttal for Reviewer CdvH [3/4]

W3. Missing baselines. There are existing continuous DLMs that should have be included in the comparison. Also, whether this method can generalize to these frameworks is unclear.

As requested, we show the results of additional continuous DLMs. For RePlaid, neither the code nor the checkpoints are released, so we could not include it. Therefore, we report LangFlow and ELF on GSM8K, and only LangFlow on Sudoku-Hard (Table D), since ELF relies on the contextual embeddings of a T5 encoder that is pretrained on natural language, so it is not straightforward to apply it to Sudoku. Like the other many-step models in Table 1, these models serve as references and not as direct comparison to one-step models. We will add these results to the final version of the paper.

Table D. Results of LangFlow and ELF on GSM8K and Sudoku-Hard (accuracy, %).

MethodGSM8K (1024 NFEs)Sudoku-Hard (180 NFEs)
FLM2.545.1
LangFlow2.064.7
ELF38.2–

Our asynchronous flow map formulation generalizes to other continuous DLMs. Importantly, our formulation does not depend on the one-hot representation. If we define the flow state between noise and the token embeddings instead of between noise and the one-hot vectors, the tokenwise times, the schedulers, and the asynchronous flow maps transfer straightforwardly. Therefore, mathematically, the generation of LangFlow, ELF, and RePlaid can follow tokenwise schedules and be distilled with our framework.

For empirical demonstration, since LangFlow and RePlaid both use vocabulary embeddings, we take LangFlow as representative and show this generalization on Sudoku-Hard (Table E) and GSM8K (Table F). For ELF, we find that the confidence from the decoder output is not a reliable signal for the scheduler. Because the decoder is trained to decode noisy latents into clean tokens, its confidence saturates at very early steps while its top-1 predictions keep changing afterwards. Using our method with ELF would instead require a measure of uncertainty that is defined on the latent space itself, which we leave for future work.

Table E. Results of LangFlow with our framework on Sudoku-Hard (exact match, %)

NFE124
FMLM4.928.743.1
FMLM+4.126.071.2
LangFlow (Sync)0.83.615.1
LangFlow (EB)0.814.861.3
LangFlow-aFMLM (Sync)2.48.515.7
LangFlow-aFMLM (EB)15.847.772.0

Table F. Results of LangFlow with our framework on GSM8K (accuracy, %)

NFE124
FMLM0.00.10.1
FMLM+0.10.13.9
LangFlow (Sync)0.00.00.0
LangFlow (EB)0.00.22.1
LangFlow-aFMLM (Sync)0.00.00.1
LangFlow-aFMLM (EB)1.83.96.1

Applying the EB scheduler improves LangFlow at 4 NFEs (from 15.1% to 61.3% on Sudoku-Hard and from 0.0% to 2.1% on GSM8K). Moreover, the asynchronous flow map that is distilled along the EB scheduler outperforms the one that is distilled along the synchronous schedule at every NFE (72.0% versus 15.7% on Sudoku-Hard and 6.1% versus 0.1% on GSM8K at 4 NFEs). These results show that our proposed method generalizes to continuous DLMs that are based on learned embeddings.

Comments on the paper

1 comment

Dear reviewers,

The author-reviewer discussion period is ending soon. If you have not done so yet, please read the authors’ rebuttal and reply to it. Tell the authors whether it addresses your concerns, and update your score if your assessment has changed.

The ICLR 2027 Reviewer Guidelines list « discussing with the authors to clarify and improve the paper » as one of the core duties you accepted as a reviewer. With the high number of submissions this year, your input matters more than ever. Recall that reviewers that do not fulfill their duties may also face penalties on their own submissions.

Thank you for your time and help.

Best regards,


Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.