acceptodds
Under review as a conference paper at ICLR 2027

Discrete Beckmann Transport Models for One-Step Language Modeling and Reasoning

21 views

Abstract

Discrete diffusion and flow models are a promising alternative to autoregressive language models, but compressing many-step sampling into fewer steps typically requires distilling a pretrained teacher model. This caps the student at the teacher's quality and requires a costly two-stage training pipeline. We introduce **Discrete Beckmann Transport Models** (DBTM), built on a time-independent flow whose autonomous transport map provably carries any point in the ambient space to a fixed point on the vertices of the simplex in a single step. We show that this fixed-point property is characterized by a conservation equation whose residual can be minimized directly from data, removing the requirement for a teacher flow and time conditioning. Under this construction, a partially trained map corresponds to the flow truncated at finite time, so generation reduces to iterating one map until it reaches a fixed point. We further extend the map to a partial-context interpolant where additional function evaluations act as refinement steps rather than ODE integration steps. On language modeling and reasoning tasks, DBTM enables one- and few-step generation that improves quality and accuracy over discrete diffusion and continuous flow baselines.

Primary area:
generative models
Keywords:
discrete generative models, language modeling, reasoning, few-step language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_WyKQOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors tackle the problem of developing a one-step flow generative model of text that can be trained without a teacher distillation or bootstrapping objective. Their key insight is to average over all interpolants over time to get a time-independent autonomous flow, and to learn this flow directly from the data by defining it as the solution of a conservation equation. The authors provide early results in unconditional language modeling where their DBTM starts to outperform FMLM at higher NFE, and outperforms FMLM on Sudoku and GSM8K.

Critical strengths
  • The time-independent autonomous flow paradigm is conceptually very nice and allows the authors to avoid using either distillation or bootstrapping to train one-step flow LMs. A single stage is much cleaner and means that the teacher is no longer the performance ceiling, nor does it contribute approximation error during student training.
  • It's also quite nice to explore how this paradigm can be used for few- rather than one-step generation, and the fixed-point refinement idea makes a lot of sense for this.
Critical weaknesses
  • Could the authors provide training FLOP counts for the training of the various baselines and DBTM? It is very well-accepted at this point that the meaningful quantity to control for pretraining language models is compute, not solely parameters. This would make the empirical claims much more sound.
  • Overall, the paradigm involves quite a lot of different loss terms, and it is sometimes unclear when the terms are required for having a consistent objective, versus when they happen to empirically help but there is no deep reason they are necessary: e.g., the anchor loss, semi-group loss, and refinement-in-the-loop objective. Could you comment on which of the losses are actually necessary for consistency, and for those loss terms which are not required for consistency, could you provide some more intuition in the paper on why they're empirically useful?
Questions for authors

Please see the weaknesses above for the two main asks (FLOPs accounting, and more discussion on consistency of the objective / when loss terms are necessary versus happen to help empirically).

3 comments
AuthorsOpenReview ↗

Author Response To Reviewer WyKQ (1/1)

Dear Reviewer WyKQ,

Thank you for your thoughtful and encouraging review. We are happy to hear you found the autonomous-flow paradigm conceptually appealing, valued the possibility of training a one-step map directly from data without an external pretrained teacher, and found the fixed-point refinement idea sensible for few-step generation.

We hope to clear up your remaining concerns with the additional experiments and responses below:

[W1] Could the authors provide training FLOP counts for DBTM and the baselines?

Yes, we have done so! The LM1B training FLOP estimates include teacher pretraining and student distillation for FMLM (See Table R10 in General Response). Although a base DBTM step costs 2.7× an FLM step per sample, the reported DBTM + ril pipeline uses approximately 6.8× fewer total training FLOPs, and plain DBTM uses approximately 14× fewer, owing to fewer training steps and smaller batches. To evaluate quality under a controlled training budget, we additionally trained both pipelines from scratch on OWT with the same backbone and approximately matched total training node-hours, counting all stages (See Tables R12–R13 in General Response). DBTM + ril achieves lower Gen-PPL at most evaluated matched-entropy comparisons and [W2] details the results for each backbone and budget.

[W2] Which losses are necessary for a consistent objective, and which only help empirically? E.g. the anchor, semigroup, and refinement-in-the-loop terms.

Thanks for the great suggestion, we agree this is a useful way to organize the objective! Table R9 now separates all eight terms. The accompanying numbered list in the appendix explains each term and its significance.

Empirical Role of the Anchor

In the exact toy setting, training the softmax map on the transport and boundary losses alone reaches a degenerate solution: the softmax saturates, so the measured residual is near zero, but the basins are wrong, and agreement with the true TT is only 37% with weights (0.6, 0.3, 0.1) and 49% with two correlated positions. Adding the anchor at t⋆t^\star from Theorem 4.1 restores it to 98% and 89% (See Table R3 in General Response). At scale on LM1B, the anchor also improves generation: without it, Gen-PPL remains above 650 at every NFE, over a frontier entropy range of 4.04 to 4.69 (See Table R7 in General Response).

Contribution of the Transport Loss

Removing the transport loss raises Gen-PPL at matched entropy at every budget (289.6 vs. 208.6 at NFE 1, and 291.6 vs. 87.6 at NFE 16, both at entropy 4.0). Pure x1\boldsymbol{x}_1 prediction without it does not produce useful one- or two-step samples in the evaluated sweep, yet with the same refinement sampler it is better than DBTM at 16 NFE (43.8 vs. 87.6). See Table R7 in General Response and our reply to Reviewer Ngbi [W3].

Anchor Placement

Theorem 4.1 gives t⋆=0.82t^\star=0.82 for LM1B while our default is 0.75. Training at exactly 0.82 gives worse quality after 100K steps (117.6 vs. 103.1 at NFE 8 and entropy 4.0), and anchoring well below t⋆t^\star collapses (0.60 in Table 12). The revision says the theorem sets the scale of tanchort_{\text{anchor}} and that the best value sits slightly below it.

We now also clarify that a map is uniquely determined within the stated solution class if it remains constant along each flow trajectory and approaches the correct vertex at the trajectory’s endpoint. These conditions characterize the ideal map but do not by themselves guarantee that neural network training recovers it.

Table R9: Roles of the training objectives.

LossSatisfied by the exact TT?Needed for consistency?Role in practice
Transportyes (stationary where b⋅∇T=0b\cdot\nabla T=0)yesdefines TT, and removing it hurts at every NFE
Boundaryyesyespins T=idT=\mathrm{id} on clean token sequences
Anchor (t>tanchort>t_{\text{anchor}})when ItI_t is in x1\boldsymbol{x}_1’s basin, likely after t⋆t^\starempirically important, without a necessity theoremhelps avoid degenerate sequences in the tested runs
Semigroupyesno: the identity map also satisfies itoff (λsemi=0\lambda_{\text{semi}}=0) in the base training recipe, removed from Eq. (25)
Quality headnot applicable (detached from TT)noscores tokens for commit only
ril-commitdepends on the paired targetnotrains the map on its own partial-context states
ril-qualitynot applicable (detached from TT)nocalibrates token scores on rollout states
Self-distillationtargets the model’s own many-round outputnocompresses refinement into fewer evaluations

If we have addressed your concerns, we hope you consider raising your score in stronger support of acceptance, and if any remain, we are happy to continue the discussion!

Thank you,

The Authors

Thanks to the authors for a nice rebuttal. I appreciate the OWT pretraining experiment with controlled train FLOPs, which looks fairly convincing. Thanks also for the disambiguation of the loss terms into those that are needed for consistency or otherwise useful empirically.

I can appreciate that novel generative model work often starts out complicated, with many auxiliary losses, and is slowly simplified over time. I think the notion of training a one-step flow from scratch, without distillation or bootstrapping, is an exciting enough proposition that any successful training of such a model, even if it requires 5+ auxiliary losses, is a worthwhile contribution.

So I am happy to raise my score to clear accept.

AuthorsOpenReview ↗

Thank you for your support!

We are happy to hear that we have resolved your remaining concerns, and your suggestions have been super helpful in strengthening our paper! We also thank you for recognizing that our paper proposes a novel generative method, and we are also excited about the potential to build upon it.

Sincerely, The Authors

Reviewer_NgbiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

DBTM learns a time-independent transport map TθT_\theta on the token simplex that is meant to send any ambient point to a vertex (a one-hot sequence) in one step. The map is trained with a stop-gradient conservation residual b⋅∇T=0b\cdot\nabla T=0 derived from the time-averaged (autonomous) flow-matching field, plus boundary, anchor, quality-head and refinement-in-loop (ril) losses, without a pretrained teacher or time conditioning. Few-step generation iterates a confidence-based commit-and-renoise sampler. Experiments cover LM1B and OpenWebText generative-perplexity/entropy frontiers, Sudoku, and TinyGSM (GSM8K).

Critical strengths
  1. A time-unconditional one-step map trained directly from data, without a teacher flow or (s, t) conditioning, would simplify training of flow-map language models considerably. This is a good attempt on discrete sequences.

  2. The flow-level theory is clean where it is proved.

  3. Sudoku results are real and large. DBTM+ril reaches 84.6–97.5% on Sudoku at 4–16 NFE, far above the baselines.

  4. Honest reporting of unfavorable numbers. (e.g. Table 10 reports that plain map iteration fails).

Critical weaknesses
  1. The learned map and the sampler are not the objects the theory describes.

    • The argmin claim does not select TT. TT is piecewise constant (one vertex per basin), so ∇T=0\nabla T=0 a.e. Any locally constant map that is the identity on M1\mathcal M_1, e.g. the argmax map x↦earg⁡max⁡xx\mapsto e_{\arg\max x}, satisfies Eq. (11) a.e. and makes Eq. (13) exactly zero, yet it does not push μ0\mu_0 to μ1\mu_1. A further explanation or clean statement here would be better.
    • How to prove that the partially trained map is a truncated flow? Is that true? For s<τs<\tau, Xs(x0)X_s(x_0) lies off the simplex (its per-position coordinate sum is 1+u(0)e−∫Z≠11+u(0)e^{-\int Z}\neq1), whereas Tθ=softmax(fθ)T_\theta=\mathrm{softmax}(f_\theta) is always on it, so "partially trained Tθ≈XsT_\theta\approx X_s, hence iterating gives XksX_{ks}" cannot hold. The paper's Table 10 confirms this: iterating TθT_\theta worsens LM1B Gen-PPL from 178.2 (1 NFE) to 501.2 (8 NFE), while the sampler actually used is renoise-and-remap, for which no theory is given.
    • Suggestion. It might be more appropriate to present "generation = iterating one map" as motivation, and describe the renoise-and-remap sampler in the abstract.
  2. OWT architecture mismatch: linear attention vs DiT baselines.

    • Table 1 and Fig. 1 use the linear-attention (DeltaNet hybrid) variant of DBTM, while FMLM, FMLM+ and DFM use DiT; Fig. 1's caption does not say so.
    • On LM1B, where backbones match, DBTM+ril does not beat DFM at matched entropy at any NFE. Is there any further analysis/explanation of the results?
  3. The training recipe is far from the "simple, teacher-free" framing, and its components are not isolated. Eq. (25) has seven loss terms (transport, semigroup, boundary, anchor, quality head, ril-commit, ril-quality), plus a late self-distillation loss. There is no ablation that removes the transport loss while keeping the partial-context interpolant, sampler and ril, so the contribution of the Beckmann objective itself is not measured.

    • Suggestion. Add (i) transport-loss-off, (ii) tanchor=0t_{\text{anchor}}=0 (pure x1x_1 prediction) with the same interpolant and sampler, (iii) ril components separately (self-distillation, temperature, OT coupling), and (iv) sensitivity to the activation schedule and rollout count kk. Say "no external pretrained teacher" rather than "no teacher".
  4. The refinement sampler is close to existing confidence-based decoding and is not attributed where it is introduced. The partial-context interpolant, threshold commit with a top-nn floor, and fresh renoising (Eqs. 19–23) closely follow FMLM+ posterior refinement (Agarwal et al., 2026), and confidence-based commit-and-remask is the mechanism of MaskGIT. Relatedly, DBTM+ril (attn) on OWT improves at 2 NFE (109.6) but then degrades monotonically to 121.0 at 16 NFE, contradicting "quality improves with budget" (Sec. 6.2).

    • Suggestion. Explain the non-monotone OWT behavior, ideally with a common backbone and comparable training compute
Questions for authors
  1. Lines 139–140 states "yielding straighter dynamics", and point to App. C.1, but I didn't find the proof. Which statement in App. C.1 proves that autonomous trajectories are straighter, and in what metric?
  2. How to prove that the partially trained map is a truncated flow? Is that true? Table 10 shows that iterating the map degrades LM1B Gen-PPL from 178.15 at 1 NFE to 501.24 at 8 NFE, whereas the renoise-and-remap sampler reaches 61.90. Could the authors explain this degradation or give some theoretical analysis?

Additional comments and suggestions are given under Weaknesses.

5 comments
AuthorsOpenReview ↗

Author Response To Reviewer Ngbi (1/2)

Dear Reviewer Ngbi,

Thank you for your careful and detailed review, and for the helpful suggestions. We are happy to hear you found the flow-level theory clean, recognized the potential of a time-unconditional map trained directly from data to simplify training, and highlighted the strong Sudoku results.

We hope to clear up your remaining concerns with the additional experiments and responses below:

[W1a] The argmin claim does not select TT: the argmax map makes Eq. (13) zero a.e. but does not push μ0\mu_0 to μ1\mu_1.

You are correct in your counterexample, and we have now made two clarifications:

  1. Characteristic invariance. Uniqueness requires T(Xs(x))=T(x)T(X_s(\boldsymbol{x}))=T(\boldsymbol{x}) and the endpoint trace, now stated explicitly. A zero gradient almost everywhere is insufficient: argmax can jump across decision boundaries along a trajectory, whereas the true map’s basin boundaries are invariant under the flow.
  2. Regression versus parameter optimization. The stop-gradient update has conditional mean proportional to −(b⋅∇Tθ)⋅∂θTθ-(b\cdot\nabla T_\theta)\cdot\partial_\theta T_\theta. Zero residual implies stationarity, but parameter gradients can also cancel. Sec. 4.1 separates ideal regression from neural optimization. Characteristic invariance plus endpoint trace uniquely identifies the ideal map within the stated solution class. We also state that recovery of the optimal map with neural network training remains empirical.

Table R3 of the General Response checks the counterexample numerically. With non-uniform data the trained map follows TT (0.976 and 0.889 agreement) and not argmax, whose pushforward is far from the data (TV 0.259 and 0.456). For uniform one-position data, symmetry makes TT equal to argmax. The revision therefore uses non-uniform data to distinguish them. We also tested smooth approximations to argmax: for the smoothed argmax softmax(x/ε)\mathrm{softmax}(\boldsymbol{x}/\varepsilon) the residual Eν∣b⋅∇Tε∣2\mathbb{E}_\nu\vert b\cdot\nabla T_{\varepsilon}\vert^2 grows like 1/ε1/\varepsilon with non-uniform data (0.40, 1.04, 3.40, 10.6 at ε=0.3,0.1,0.03,0.01\varepsilon=0.3,0.1,0.03,0.01) but shrinks with uniform data, where argmax is TT.

[W1b] For s<τs<\tau, Xs(x0)X_s(\boldsymbol{x}_0) is off the simplex while Tθ=softmax(fθ)T_{\theta}=\mathrm{softmax}(f_{\theta}) is on it, so “partially trained Tθ≈XsT_\theta\approx X_s” cannot hold. The sampler actually used has no theory. Present iteration as motivation and describe the sampler in the abstract.

You are right about the softmax head, and we have adopted your suggestion to distinguish the ideal map from the sampler used in practice. Sec. 4.1 now explains that unconstrained, identity-initialized population regression has the continuous-update limit ∂sTs=b⋅∇Ts\partial_sT_s=b\cdot\nabla T_s, whose solution is Ts=XsT_s=X_s under suitable regularity. The semigroup interpretation follows if an iterate approximates XhX_h at one common time and errors remain controlled. In the toy experiment, the nearest trajectory time varies across inputs (Table R2), and the softmax approximation TθT_\theta stays on the simplex rather than following the finite-time ambient trajectory.

For TθT_\theta, the theory instead motivates re-encoding before remapping. With the exact map, t=0t=0 gives a fresh sample and t=1t=1 preserves a hard vertex. Intermediate re-encoding with fresh noise can reopen token decisions. Table R4 also supports this intuition, showing that at t=0.75t=0.75 Gen-PPL improves from 178.4 to 92.5, while t=0.9t=0.9 helps less. The sampler we use combines committed tokens with fresh noise at the remaining positions, with the same interpolant form as training, so the map's predictions are reliable. However, we acknowledge that the generated context can still introduce exposure bias, so this is motivation rather than a convergence theorem for our sampler.

The abstract now describes renoise-and-remap, Sec. 4.1 states the ideal regression argument and its limits, and the appendix gives a conditional endpoint-error bound. The commit floor separately guarantees that sampling finishes within the NFE budget.

[Q1] Which statement in App. C.1 proves that autonomous trajectories are straighter, and in what metric?

Thank you for suggesting this, and we agree that the "straighter paths" claim needs to be clarified. App. C.1 now defines the Euclidean arc length and chord ratio (arc length divided by endpoint distance), and Proposition C.6 derives local curvature formulas for both flows. The time-dependent formula includes an explicit time-derivative term absent from the autonomous one. That term can reinforce or cancel the spatial contribution, and averaging also changes the spatial derivatives, so these identities alone do not prove straighter trajectories. The evidence that autonomous trajectories are straighter in our setting remains the empirical mean path length and chord ratio given in Fig. 3E-G and we have removed “provably” from that claim.

AuthorsOpenReview ↗

Author Response To Reviewer Ngbi (2/2)

[W2] OWT uses linear attention for DBTM, but DiT for the baselines, and Fig. 1 does not say so. On LM1B, DBTM + ril does not beat DFM at matched entropy at any NFE.

Thanks for the suggestion, and we have now made the backbone-matched comparisons. On OWT, Figure 1 now points to both backbones. At approximately matched training node-hours (Tables R12–R13), DBTM + ril has lower matched-entropy Gen-PPL on the linear backbone in all comparisons but one. On DiT, it is lower in the overlap except at 1 NFE and C1.

On LM1B, DBTM + ril is competitive with DFM (PSD) and behind DFM (ESD) at reported nearby entropies. Gen-PPL measures coherence under a reference model, while unigram entropy captures only one aspect of diversity, so isolated operating points cannot establish a ranking across the full quality–diversity trade-off. Our DFM comparison uses reported points rather than a full generative frontier.

In addition, reasoning provides complementary task-correctness measures which are not reported in DFM: DBTM + ril solves 84.6% and 97.5% of hard Sudoku puzzles at 4 and 16 NFE. Verifier-guided post-training reaches 34.4% GSM8K accuracy at 32 NFE (Table R15).

[W3] The recipe has seven-plus loss terms, and nothing isolates the transport loss. Suggested ablations: (i) transport off, (ii) tanchor=0t_{\text{anchor}}=0, (iii) ril components, (iv) activation schedule and kk.

We added transport-off, all-time-anchor and anchor-placement ablations (See Table R7 in General Response), plus self-distillation temperature, coupling and rollout comparisons (Table R8). Existing curves, including ril versus no ril on Sudoku, show the overall recipe’s benefit at matched optimizer steps. Individual ril terms and activation schedules are not fully isolated, and matched steps do not imply matched FLOPs.

Removing only the transport loss hurts at every tested budget (1.4× at NFE 1, 2.0× at NFE 2, about 3× at NFE 8 and 16), keeping the interpolant, sampler, anchor and boundary losses. But pure x1\boldsymbol{x}_1 prediction has no useful one- or two-step frontier in the sampled range, while giving lower Gen-PPL near entropy 4 at 8–16 NFE (43.8 vs. 87.6 at NFE 16). This supports the transport objective’s role at one and two NFE in the tested base recipe, while showing that a time-free predictor with the same sampler can be stronger after many rounds. The 100K-step base ablation does not establish the transport contribution within the full +ril pipeline. We also adopt “no external pretrained teacher”.

Table R8: Refinement-in-loop training ablations. Measured Gen-PPL (entropy) at each frontier's point nearest 4.10, without interpolation. All arms share a 100K checkpoint and train to 150K with one seed. Max. entropy covers NFE 1, 2, 4, 8 and 16. Bold marks each column's best displayed value.

Self-distillation settingNFE 2NFE 4NFE 8NFE 16Max. entropy
Default (τ=0.7\tau=0.7, OT coupling, k=4k=4)97.6 (4.10)99.3 (4.08)83.3 (4.10)77.0 (4.13)4.31
Target temperature τ=1.0\tau=1.095.1 (4.09)83.0 (4.08)73.4 (4.09)68.7 (4.08)4.19
Independent coupling95.9 (4.10)98.3 (4.08)82.2 (4.10)76.3 (4.12)4.31
Rollout count k=2k=2126.8 (4.12)112.9 (4.18)89.5 (4.15)74.9 (4.12)4.42

This shows that softer targets lower Gen-PPL at nearby entropies but reduce maximum entropy. Independent and OT coupling are similar, and the k=2k=2 arm reaches higher maximum entropy.

[W4] The refinement sampler closely follows FMLM+ and MaskGIT without attribution where it is introduced. DBTM + ril (attn) on OWT degrades from 109.6 at 2 NFE to 121.0 at 16 NFE, contradicting “quality improves with budget”.

Eqs. 19-23 now credit FMLM+ [2] for the partial-context interpolant and threshold-plus-floor commit, and MaskGIT [4] for confidence-based commit and remask. DBTM adds a time-free map trained on the sampler’s own states.

At a fixed threshold, entropy drifts as the budget grows: DBTM + ril (attn) moves from 4.94 to 5.50, past the data entropy of 5.441 (Table 7). This makes raw Gen-PPL non-monotone. Comparing at entropy 5.70 instead gives Gen-PPL 227.4, 180.6 and 135.6 at 4, 8 and 16 NFE (Table 14). Sec. 6.2 now describes the improvement at these evaluated matched-entropy points.

Please see Part 1 for [Q1].

[Q2] Is the partially trained map a truncated flow? Can you explain the degradation when iterating the map (Table 10)?

Please see [W1b] for the ideal flow argument. In Table 10, outputs approach a poor, nearly stable state since soft predictions differ from training inputs. Re-encoding improves quality (Table R4), and training on self-generated inputs with the repair loss makes plain iteration improve with kk (4w3P [W3], Table R5).


If we have addressed your concerns, we hope you consider raising your score in stronger support of acceptance, and if any remain, we are happy to continue the discussion!

Thank you,

The Authors

AuthorsOpenReview ↗

Follow up to Reviewer Ngbi

Dear Reviewer Ngbi,

Thank you for your constructive feedback which have strengthened our paper!

Since you did not note any remaining questions, we wanted to check whether anything is still open on your side. If there is a specific experiment or clarification that would move the paper toward a clearer accept, please let us know, and we would be happy to run it while the discussion period is open.

Thank you again for taking the time to review our paper!

The Authors

Reviewer_6um3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Discrete Beckmann Transport Models (DBTM), a framework for one-step and few-step discrete sequence generation. The method constructs a time-independent transport field and learns a map through a conservation equation, avoiding teacher distillation and explicit time conditioning. The authors further introduce a phase-transition-based anchoring objective and a refinement-in-loop training procedure to improve generation quality.

Critical strengths
  1. The paper provides a principled formulation of one-step generation through autonomous transport and a conservation equation. Avoiding teacher distillation and time conditioning is appealing, and the phase-transition analysis offers useful motivation for the anchoring objective.

  2. DBTM achieves competitive generative perplexity at low NFE budgets on LM1B and OpenWebText. The experiments also demonstrate the effectiveness of refinement-in-loop training.

Critical weaknesses
  1. The theoretical analysis establishes convergence properties for the ideal autonomous flow and its transport map. However, the learned neural map is only an approximation, and the paper does not establish whether minimizing the proposed objectives preserves these guarantees. In particular, the claim that a partially trained map corresponds to a finite-time autonomous flow appears insufficiently justified. Additional analysis of approximation error, convergence, or fixed-point behavior would strengthen the theoretical contribution.

  2. the GSM8K results remain substantially below autoregressive baselines. DBTM achieves only 0.8% accuracy at one NFE and 16.8% at 32 NFEs, compared with 63.3% for the autoregressive greedy baseline at 512 NFEs. The few-step improvements over FMLM+ on GSM8K are also relatively small. This raises questions about whether the proposed framework can effectively capture the sequential dependencies required for more complex language reasoning.

Questions for authors
  1. Can the authors provide theoretical or empirical evidence that the learned transport map approximates a finite-time autonomous flow? How does the fixed-point residual correlate with generation quality and convergence?

  2. What primarily limits DBTM's GSM8K performance? Is the bottleneck the transport objective, the refinement strategy, or the inability to capture sequential dependencies?

3 comments
AuthorsOpenReview ↗

Author Response To Reviewer 6um3 (1/2)

Dear Reviewer 6um3,

Thank you for your thoughtful and encouraging review. We are glad you found the autonomous-transport and conservation-equation formulation principled, the absence of an external pretrained teacher and time conditioning appealing, and the phase-transition analysis useful in motivating the anchor. We also appreciate your recognition of the competitive low-NFE results and the effectiveness of refinement-in-loop training.

We hope to clear up your remaining concerns with the additional experiments and responses below:

[W1] The guarantees are for the ideal flow, but it is not shown that minimizing the objectives preserves them. The claim that a partially trained map is a finite-time autonomous flow is insufficiently justified.

Sec. 4.1 now separates ideal regression from neural approximation. Exact regression onto a frozen previous iterate gives T(n+1)=T(n)+η b⋅∇T(n)T^{(n+1)}=T^{(n)}+\eta\,b\cdot\nabla T^{(n)}. Under suitable regularity, the continuous-update limit with T0=idT_0=\mathrm{id} is Ts=XsT_s=X_s. If an unconstrained iterate approximates XhX_h at a common time hh, repeated applications then approximate XkhX_{kh} by the semigroup property, provided the errors remain controlled along the iterates.

We tested this interpretation on small problems where the reference flow can be computed exactly, including one position with three possible tokens, and two correlated positions with four tokens each. We numerically compute the autonomous field bb, trace its trajectories, and compare the learned maps with these reference paths and endpoints (Tables R2 and R3).

Starting from the identity map, which initially leaves every input unchanged, we show that successive training checkpoints move predictions farther along the reference trajectories. The median nearest trajectory time increases from 0.20 to 0.95. We note that different inputs advance at different rates, so these results support movement along the flow without identifying each checkpoint with one common flow time.

The practical softmax model predicts token probabilities rather than points along these ambient-space trajectories. We therefore evaluate whether it selects the correct endpoint. With the anchor loss, it agrees with the reference endpoints on approximately 97% and 89% of test points (Table R3).

The appendix provides a complementary error bound proving that the learned prediction is close to the correct endpoint when it is accurate near the end of the reference trajectory and changes little along the way. The bound makes these sufficient conditions explicit, and the toy experiments provide empirical evidence of endpoint recovery.

[W2] GSM8K results remain far below AR (16.8% at 32 NFE vs. 63.3% greedy AR at 512 NFE), and gains over FMLM+ are small.

We agree, but we emphasize that the gap to AR models is large across all evaluated continuous and masked diffusion baselines, including those using 1024 NFE. The submission reports 16.8% for DBTM + ril at 32 NFE, compared with 16.6% for FMLM+ at the same budget and 63.3% for greedy AR at 512 NFE. At 1024 NFE, the masked diffusion baselines MDLM and Duo reach 18.0% and 17.2%, respectively. The continuous baselines CANDI, FLM and S-FLM reach 0.2%, 0.3% and 18.0%. Thus, DBTM + ril approaches the strongest reported diffusion-baseline accuracy with 32× fewer function evaluations. The large remaining gap to AR is shared across these methods, motivating our bottleneck analysis in [Q2] and the post-training experiments below.

In our revision, we show that post-training can significantly improve DBTM’s performance on GSM8K with additional verified supervision. Since the map often produces a correct program that a single sample misses ([Q2]), we used verifier-guided post-training on the same 250K-update checkpoint, keeping the DBTM objective. For each training problem, we draw 8 programs from the sampler and execute them. Where a failed program differs from the closest correct program for the same problem, we train the map toward the correct tokens.

On the full 1,319-problem test set at 32 NFE with one sample per problem, accuracy rises from 16.6% to 23.1% after 2K updates, reaches 29.6% by 10K and 34.4% by 20K (Table R15), more than doubling base accuracy. These gains provide evidence of DBTM’s capacity to benefit from verifier-guided post-training. This narrows the gap with AR baselines and motivates future work optimizing post-training algorithms designed for DBTM.

Table R15: Verifier-guided post-training on GSM8K. Accuracy (%) on the full 1,319-problem test set at 32 NFE, using one sample per problem. Values are means over three sampling seeds of one post-training run. See appendix Table R15.

Updates01K2K4K6K8K10K20K
Verifier-guided16.621.523.127.528.228.929.634.4
AuthorsOpenReview ↗

Author Response To Reviewer 6um3 (2/2)

[Q1] Is there theoretical or empirical evidence that the learned map approximates a finite-time autonomous flow? How does the fixed-point residual correlate with quality and convergence?

Please see [W1] for the flow interpretation. On LM1B, we tracked three diagnostics alongside 1- and 8-NFE Gen-PPL and entropy (Table R6): the samplewise directional-derivative diagnostic E∥I˙t⋅∇Tθ(It)∥2\mathbb{E}\Vert\dot I_t\cdot\nabla T_\theta(I_t)\Vert^2 on held-out data, the boundary residual ∥∣Tθ(x1)−x1∥2\Vert|T_\theta(\boldsymbol{x}_1)-\boldsymbol{x}_1\Vert^2, and a perturbation stability diagnostic, the fraction of tokens that change when half of a finished sample is renoised and remapped.

  1. The sampler stability diagnostic correlates with quality in this run. It falls steadily with training (Spearman ρ=−0.98\rho=-0.98 with step) and correlates with 8-NFE quality: ρ=+0.94\rho=+0.94 with Gen-PPL and −0.96-0.96 with entropy, so a lower residual comes with lower perplexity and higher entropy. Over training the 8-NFE samples improves on both axes, from 88.2 at entropy 3.73 to 69.9 at 4.02.
  2. The boundary residual is effectively zero throughout, showing that the map preserves clean data sequences at every evaluated checkpoint.
  3. The samplewise directional-derivative diagnostic increases during training (ρ=+0.78\rho=+0.78 with step), even as 8-NFE sample quality improves. This is not the characteristic residual E∣b⋅∇Tθ∣2\mathbb{E}\vert b\cdot\nabla T_\theta\vert^2 in the endpoint bound: because b(x)=E[I˙t∣It=x]b(x)=\mathbb{E}[\dot I_t\mid I_t=x], squaring before conditional averaging adds a conditional-variance contribution, so its increase does not by itself mean an increase in the residual of the true residual we aim to minimize.

At 1 NFE, perplexity and entropy both rise during training, reflecting a quality–diversity tradeoff at this fixed sampler setting. The clearer improvement is at 8 NFE, where perplexity falls while entropy increases.

Table R6: Diagnostics and sample quality over training on LM1B. Diagnostics are measured on 512 held-out sequences, and Gen-PPL is evaluated on 1024 samples at commit threshold 0.9. All 14 evaluated checkpoints enter the reported correlations.

StepSamplewise derivativeBoundaryPerturbation token-change rateGen-PPL, NFE 1 (ent.)Gen-PPL, NFE 8 (ent.)
25K39.31.5e-120.870169.6 (3.78)88.2 (3.73)
100K46.85.2e-140.860208.5 (3.93)89.2 (3.86)
200K56.24.8e-120.839222.1 (4.00)84.3 (3.94)
400K56.35.0e-140.828224.5 (4.03)78.9 (3.99)
600K50.21.1e-120.818243.1 (4.09)76.9 (4.01)
800K53.52.5e-090.807255.1 (4.13)70.4 (4.01)
975K56.41.8e-130.805248.8 (4.12)69.9 (4.02)

[Q2] What primarily limits GSM8K performance: the transport objective, the refinement strategy, or sequential dependencies?

Three diagnostics hold the checkpoint fixed within each comparison, although settings differ across diagnostics.

  1. Error types. Executability rises from 6.4% at 4 NFE to 75.4% at 64 NFE, while accuracy is flat past 32 NFE. Accuracy given execution falls from 77.6% to 19.6%, indicating that extra rounds mainly improve executability.
  2. Commit order. Forcing commits to move left to right at inference collapses accuracy (0.30% against 11.07% for confidence order, NFE 32), indicating that the order cannot simply be defined at inference on a model that never trained on such contexts. However, training DBTM with prefix context is an interesting direction for future work.
  3. Multiple candidates. We redo a four-draw evaluation of the 250K checkpoint, which gives 27.8% Pass@4 versus 16.3% Pass@1 at 32 NFE. This shows that correct candidates are available, motivating ranking or verifier-guided training.

These diagnostics suggest that additional refinement improves executability more than answer correctness, while multiple rollouts reveal correct solutions that single-sample generation misses. Motivated by these findings, verifier-guided post-training raises single-sample accuracy from 16.6% to 34.4% at 32 NFE (Table R15). This demonstrates that DBTM can learn substantially from verified self-generated programs.


If we have addressed your concerns, we hope you consider raising your score in stronger support of acceptance, and if any remain, we are happy to continue the discussion!

Thank you,

The Authors

AuthorsOpenReview ↗

Follow up to Reviewer 6um3

Dear Reviewer 6um3,

Thank you for your constructive feedback which have strengthened our paper!

Since you did not note any remaining questions, we wanted to check whether anything is still open on your side. If there is a specific experiment or clarification that would move the paper toward a clearer accept, please let us know, and we would be happy to run it while the discussion period is open.

Thank you again for taking the time to review our paper!

The Authors

Reviewer_4w3POpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes Discrete Beckmann Transport Models (DBTM), which operate in an autonomous (time-independent) ambient space to theoretically enable a one-step flow map to a point on the simplex. The paper details how this transport map can be trained directly from data without distillation from a teacher model. Experiments on LM1B and OWT, as well as Sudoku and GSM8K, claim that DBTMs with refinement-in-the-loop (+ril) yield improvements in unconditional language modeling performance on the former and accuracy on the latter.

Critical strengths

This paper presents interesting and non-trivial theory to enable the learning of one-step flow maps to the data manifold lying on the simplex. As such, the theory adds a helpful angle to the literature on flow and flow map language modeling. The aspect of distillation-free training and of a time-free transport map are appealing, and bring the training and modeling more in line with current trends in the mainstream dLLM literature.

Critical weaknesses

The experimental results do not validate the theoretical benefits of this approach, while relegating some critical negative results to the appendix.

  • DBTM without refinement-in-the-loop ("+ril") performs worse than FMLM at low-NFE regimes (between 1 and 8), and at matched entropy on LM1B DBTM consistently loses out to FMLM in Table 13. No refinement-in-the-loop machinery is applied to FMLM for fair comparison.

  • The OWT gains of plain DBTM over FMLM of Table 1 are confounded by a different backbone architecture from the standard DiT — from Table 7 in the appendix it can be seen that with the same DiT architecture, DBTM (attn) on OWT consistently loses out to FMLM.

  • It is unclear to what extent the model reflects the theory. In Table 10, iterating the transport map Tθ∘kT_{\theta}^{\circ k} 8 times blows up generative perplexity from 178.15 to 501.24. Repeated application of learned map therefore seems to worsen the sample quality rather than improve it (even if in practice the partially-trained map is not idempotent). Since this appears to be the only ablation where the authors iterate the learned map, it is concerning that PPL appears to diverge rather than to converge.

Questions for authors

Could the authors report a compute-matched comparison in terms of FLOPs with FMLM to determine the computational cost of the proposed distillation-free training procedure?

2 comments
AuthorsOpenReview ↗

Author Response To Reviewer 4w3P (1/2)

Dear Reviewer 4w3P,

Thank you for your thoughtful and detailed review. We are happy to hear you found the theory interesting and non-trivial, appreciated its contribution to flow-map language modeling, and found direct training without an external pretrained teacher and the time-free transport map appealing.

We hope to clear up your concerns with the additional experiments and responses below:

[W1] DBTM without +ril performs worse than FMLM at 1 to 8 NFE, and at matched entropy on LM1B it consistently loses to FMLM (Table 13). No refinement-in-the-loop machinery is applied to FMLM for fair comparison.

The main text now includes plain DBTM’s matched-entropy comparison with FMLM from Table 13. We also compare DBTM + ril for two reasons:

  1. Both pipelines include distillation. The self-distillation component of DBTM + ril is the counterpart of FLM → FMLM distillation [1] since both compress many-step generation into fewer evaluations. FMLM already includes this stage, whereas plain DBTM does not. This motivates comparing the full DBTM + ril and FLM → FMLM pipelines, counting all training stages.
  2. FMLM+ [2] already adds refinement to flow maps and is included in Tables 7 and 14. It uses per-token time conditioning, whereas DBTM uses a time-free map on partial-context interpolants. We show that DBTM+ril outperforms FMLM+ across tasks.

FMLM has lower Gen-PPL at the reported matched entropies at 1-2 NFE, while DBTM + ril is lower at 4-16 NFE and covers a broader entropy range (Table R1). We use entropy 4.25 except at 1 NFE, where FMLM only reaches 4.18. The released FMLM checkpoint used more training compute. The larger Gen-PPL reduction from 2 to 16 NFE at entropy 4.25 indicates that DBTM + ril benefits more from additional NFEs compared to more integration steps with FMLM. [Q1] reports costs and [W2] controls the training budget.

Table R1: LM1B Gen-PPL (↓\downarrow) at matched entropy. Frontier ranges and values from Table 13. Bold marks lower Gen-PPL.

NFEFMLM entropy rangeDBTM + ril entropy rangeComparison entropyFMLM PPL ↓DBTM + ril PPL ↓
14.12-4.183.92-4.404.15116.2242.2
24.20-4.323.97-4.254.25122.9128.5
44.21-4.323.93--4.404.25109.0103.0
84.19-4.303.87-4.454.2598.885.4
164.22-4.313.83-4.474.2589.675.8

[W2] The OWT gains over FMLM in Table 1 are confounded by a different backbone.

The original Table 1 emphasized the stronger linear-backbone DBTM results (at matched parameter count), while Table 14 included DiT. Since FMLM was trained extensively (1M+500K steps), we didn't originally train FMLM with a linear backbone.

In this revision, we retrained FLM+FMLM from released code and DBTM on both backbones at matched compute (Tables R12–R13). Including teacher pretraining and distillation for FMLM, the budgets are about 56 and 66.5 node-hours on 4×H200 (appendix Table R11). Figure 1 now points to an appendix figure showing both. Against released FMLM on DiT, plain DBTM has higher Gen-PPL (184.8 vs. 121.6 at entropy 5.30, 4 NFE). However, DBTM + ril reaches data entropy 5.441 and beyond, demonstrating broader diversity coverage.

We hold the pretrained teacher fixed within each backbone and extend student distillation to evaluate two total training budgets, counting both stages. This teacher-student allocation is not established as optimal, so our experiment measures performance under these controlled allocations rather than the best achievable performance at each budget.

Tables R12–R13 include all the frontier metrics across various sampler settings. The results show:

  1. Linear backbone. DBTM + ril has the lower Gen-PPL at every matched entropy except one (C1, 2 NFE, entropy 5.70: 500.4 vs. 443.9 on the DBTM + ril frontier). At C2, it reaches the data entropy at every NFE ≥ 2 (184.4, 94.7, 66.5 and 55.2 at 2, 4, 8 and 16 NFE). FMLM reaches it at 8 and 16 NFE (294.8 and 227.2).
  2. DiT backbone. FMLM is lower at 1 NFE and C1 (324.4 vs. 391.0 at entropy 5.40). Wherever else both frontiers reach the same entropy, DBTM + ril is lower, e.g. 128.3 vs. 276.8 at entropy 5.60, 4 NFE, C2. These Gen-PPL comparisons alone do not establish an overall sample-quality advantage. At C2, the frontiers share no common entropy at 8 NFE and do not overlap at 16 NFE.

Teacher Quality at This Budget (Appendix Table R14)

At 1024 Euler steps, the 145K-step DiT teacher reaches Gen-PPL 68.6 at entropy 5.36, close to the released teacher’s 63.2 at 5.35 (256 samples), suggesting a substantial distillation contribution to the few-step gap. The linear teacher is weaker (134.1 at 5.61), so teacher quality may also contribute.

AuthorsOpenReview ↗

Author Response To Reviewer 4w3P (2/2)

[W3] Iterating the learned map 8 times blows up Gen-PPL from 178.15 to 501.24 (Table 10). PPL appears to diverge rather than converge.

Thank you for bringing this up, and we agree that the paper doesn't sufficiently explain the PPL increase, so we hope to clarify here. We measured both sample quality and token changes over repeated applications. While plain iteration degrades sharply at first, Gen-PPL then plateaus near 500 (Table R4). Token changes fall from 53% between k=1k=1 and 2 to 24% between 2 and 3, and below 1% after k=5k=5. Therefore, over the tested horizon, the outputs settle near a poor fixed point rather than continuing to diverge.

In practice, the learned map is not idempotent when applied to the first proposal (i.e. T(T(x))≠T(x)T(T(x))\neq T(x)). This can be attributed to the fact that the softmax outputs of the model differ substantially from its training inputs at the first map application. Training inputs have per-position coordinate sums spread over roughly ±267\pm267 at t=0t=0 and ±26\pm26 at t=0.9t=0.9 (10th–90th percentiles), while the soft output sums to 1. The boundary residual is at most 2×10−82\times10^{-8} on clean validation sequences, but these predictions are different inputs. Re-encoding them with fresh noise restores the interpolant form and improves Gen-PPL from 178.4 to 92.5 by k=8k=8 at t=0.75t=0.75 (Table R4), with a smaller improvement at t=0.9t=0.9. This fits the commitment-time intuition, although generated context can still introduce exposure bias.

Improving Plain Iteration with a Repair Loss

To prevent convergence to a bad fixed point, we added a repair loss to reduce exposure bias during plain iteration. Base training exposes the map to data-derived interpolants and clean data vertices, so its own imperfect soft predictions can therefore be out of distribution when fed back as inputs. The repair loss trains the map on these self-generated soft outputs: at interpolant positions past tanchort_{\text{anchor}}, it applies the map to its detached prediction and pins the result to the data:

Lrepair=CE(Tθ(sg Tθ(It)),x1)\mathcal{L}_{\text{repair}}=\mathrm{CE}(T_\theta(\mathrm{sg}\,T_{\theta}(I_t)),\boldsymbol{x}_1)

The target is x1\boldsymbol{x}_1, as in the anchor loss. This teaches the map to correct its prediction rather than preserve an incorrect one. After fine-tuning our 200K checkpoint for 50K steps, plain iteration improves at every tested step. At k=16k=16 it reaches 91.9 at entropy 4.19, close to the control’s renoise-and-remap result of 90.7 at 4.20. The repair-trained sampler also stays within 10% of the control’s matched-entropy frontier. Table R5 reports plain iteration, where entropy dips by 0.09 at k=2k=2 before recovering. The revision includes this loss and the results in appendix Table R5.

Table R5: Effect of the repair loss on plain map iteration. Gen-PPL (↓\downarrow) and entropy (in parentheses) of Tθ∘k(x0)T_\theta^{\circ k}(\boldsymbol{x}_0) on LM1B at σ=1.0\sigma=1.0. Both models are evaluated at 250K steps using 1024 samples and one training seed.

kk124816
Without repair231.1 (4.02)411.5 (4.18)308.9 (4.14)296.0 (4.12)298.3 (4.13)
With repair217.0 (3.99)162.3 (3.90)132.2 (3.91)107.9 (4.06)91.9 (4.19)

Rolling the repair loss through four self-applications instead of one does not help further (217.9, 150.9, 124.3, 105.9, 93.5 at k=1k=1 to 16), so we report the single-application form. Sec. 4.1 now explains the ideal regression argument behind the finite-time flow interpretation and distinguishes it from the softmax map used in practice. Please see our reply to Reviewer Ngbi [W1b] and Tables R2 and R3 of the General Response.

[Q1] Could the authors report a compute-matched comparison in FLOPs with FMLM?

Table R10 reports the LM1B training FLOP estimates, including teacher pretraining and student distillation for FMLM. Although a base DBTM step costs 2.7× an FLM step per sample, the reported DBTM + ril pipeline uses approximately 6.8× fewer total training FLOPs, and plain DBTM uses approximately 14× fewer, owing to fewer training steps and smaller batches. To evaluate quality under a controlled training budget, we additionally trained both pipelines from scratch on OWT with the same backbone and approximately matched total training node-hours, counting all stages (Tables R12–R13). DBTM + ril achieves lower Gen-PPL at most evaluated matched-entropy comparisons, and [W2] details the results for each backbone and budget.


If we have addressed your concerns, we hope you will consider raising your score in support of acceptance, and if any remain, we are happy to continue the discussion!

Thank you,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response to Reviewers (1/2)

We sincerely thank all four reviewers for their careful and specific comments. We appreciate that the reviewers acknowledged the principled theoretical framework (4w3P, 6um3, Ngbi, WyKQ) that enables one-step generation, the appealing time-conditioning-free and teacher-free objective (4w3P, 6um3) and simplification from flow-map methods (Ngbi, WyKQ), and strong empirical results on language and reasoning (6um3, Ngbi).

The main concerns were the connection between the learned map and the theory, the fairness of the comparisons, and the contribution of the training objectives. We ran additional experiments and updated the manuscript to address these questions. We summarize the changes below and discuss each concern in the individual replies. Revised text is highlighted in red\color{red}{\text{red}} in the updated manuscript.

Overview of Key Updates

  • Matched backbone and training compute on OWT (Tables R12 and R13). We trained DBTM + ril and the full FLM-teacher/FMLM-distillation pipeline from scratch on both DiT and linear-attention backbones, with the same data, tokenizer, context length and hardware. Budgets include all training stages. The revision labels each backbone and makes the DiT rows the main comparison.
  • Training FLOPs (Table R10). A DBTM step is more expensive, but its reported training pipeline uses fewer total FLOPs. This cost accounting complements the OWT node-hour comparison. The LM1B results themselves are not compute-matched since we use the publicly available checkpoints and reported metrics from the baseline papers.
  • Map iteration (Table R4). Plain iteration settles near a poor fixed point, which motivates the renoise-then-remap scheme used in our initial submission to scale NFEs. In the revision, we also show that re-encoding the output before the next application improves quality, and a new repair loss also improves plain iteration.
  • Loss ablations (Table R7). Removing only the transport loss hurts at each tested NFE in the base recipe. Against pure time-free x1\boldsymbol{x}_1 prediction, the demonstrated transport benefit is at one and two steps.
  • Exact low-dimensional checks (Tables R2 and R3). We clarify when ideal regression follows the autonomous flow and test how closely neural maps follow this theory.
  • Clarification of theoretical claims. We distinguish ideal-flow results from properties of the learned sampler and add local curvature formulas alongside the empirical straightness comparison.

Shared Tables

The tables below are shared across replies. Table R13, Tables R2–R4 and R7, and the DFM comparison are in Part 2. Training and sampling seed counts are specified with the relevant experiments.

Table R12: OWT decode frontier with DiT at matched training compute. Gen-PPL (↓\downarrow) at matched entropy on each method's Pareto frontier over all completed sampler settings. The training budget is C2 = 66.5 node-hours (appendix Table R12). Entropy ranges are shown for each method, and bold marks the lower Gen-PPL where both have a point. Real OWT has entropy 5.441. Linear-attention results follow in Part 2.

NFEMethodEnt. range5.205.305.405.505.605.705.80
1FMLM5.31–5.59––327.9474.8–––
1DBTM + ril5.15–5.66170.0224.4306.9399.7539.2––
2FMLM5.44–5.62–––287.7554.0––
2DBTM + ril5.39–5.83––143.3165.4260.0303.6428.3
4FMLM5.46–5.66–––185.7276.8––
4DBTM + ril5.53–6.10––––128.3155.9208.0
8FMLM5.51–5.69––––167.0––
8DBTM + ril5.61–6.11–––––102.0134.9

At budget C1 (56 node-hours), FMLM is lower at 1 NFE (324.4 vs. 391.0 at entropy 5.40) and DBTM + ril at every other shared entropy (Table R12).

Table R10: Training FLOPs on LM1B. FLOPs for one full training step (forward, backward, optimizer), counted with PyTorch's FLOP counter at the same micro-batch size and reported per sample. Total FLOPs are computed by multiplying by the training steps and global batch size.

ModelFLOPs per sample per stepSteps × batchTotal
FLM teacher [1]9.7e101M × 5125.0e19
FMLM student [1]1.5e11550K × 5124.2e19
FMLM total9.2e19
DBTM2.6e11200K × 1286.7e18
DBTM + ril2.6e11, then 7.9e11(100K + 100K) × 1281.35e19

Citations

[1] Lee et al. “Flow Map Language Models: One-Step Language Modeling via Continuous Denoising.” (2026). (FLM, FMLM)

[2] Agarwal et al. “Posterior Refinement: Fast Language Generation via Any-Order Flow Maps.” (2026). (FMLM+)

[3] Potaptchik et al. “Discrete Flow Maps.” (2026). (DFM)

[4] Chang et al. “MaskGIT: Masked Generative Image Transformer.” CVPR (2022).

AuthorsOpenReview ↗

Global Response to Reviewers (2/2)

We note that the submission table numbers are unchanged in the revised PDF. Tables reproduced below use the same R numbers as the revised manuscript.

Table R13: OWT decode frontier with linear attention at matched training compute. Gen-PPL (↓\downarrow) at matched entropy for budget C2, following the protocol in Part 1 (appendix Table R13).

NFEMethodEnt. range5.105.205.305.405.505.605.705.80
1FMLM5.13–5.36–385.3446.7–––––
1DBTM + ril4.91–5.16286.1–––––––
2FMLM5.55–5.68–––––410.2––
2DBTM + ril5.07–5.7564.387.8119.8165.6239.4358.3477.2–
4FMLM5.64–5.78––––––396.1–
4DBTM + ril5.20–5.81––57.279.6117.6164.4236.8316.2
8FMLM5.13–5.93–237.0258.3284.2301.3304.7308.0404.1
8DBTM + ril5.22–5.88––39.459.879.1136.7165.0285.7
16FMLM5.35–5.91–––223.2233.2243.5273.3349.4
16DBTM + ril4.99–5.7621.324.535.048.884.0101.6136.2–

At budget C1, DBTM + ril is lower at every shared entropy except 2 NFE at 5.70 (500.4 vs. 443.9, Table R13).

Table R4: Effect of re-encoding on map iteration. Gen-PPL (↓\downarrow) and entropy (in parentheses) on LM1B, using the σ=1.2\sigma=1.2 checkpoint of Table 10 and 1024 samples. Re-encoding applies x←(1−t)z′+t onehot(arg⁡max⁡Tθ(x))\boldsymbol{x}\leftarrow(1-t)\boldsymbol{z}'+t\,\mathrm{onehot}(\arg\max T_\theta(\boldsymbol{x})) with fresh z′\boldsymbol{z}' before the next map application. See appendix Table R4.

Iteration rulek=1k=1k=2k=2k=4k=4k=8k=8
Plain Tθ∘kT_\theta^{\circ k}178.4 (3.93)503.5 (4.21)490.3 (4.23)494.1 (4.23)
Re-encode at t=0.5t=0.5178.4 (3.93)146.6 (3.91)144.3 (3.91)145.4 (3.91)
Re-encode at t=0.75t=0.75178.4 (3.93)118.5 (3.90)100.2 (3.88)92.5 (3.87)
Re-encode at t=0.9t=0.9178.4 (3.93)171.5 (3.94)166.2 (3.94)161.3 (3.96)
Renoise and remap (sampler)178.1 (3.94)133.3 (4.04)87.4 (4.02)61.9 (3.98)

Table R7: Loss-term ablation on LM1B. Gen-PPL (↓\downarrow) on each arm's Pareto frontier after 100K steps, using 1024 samples per point. Entropy is in parentheses. The comparison entropy is 4.00, or 4.10 at NFE 2. † marks the nearest point when the frontier does not reach that entropy. ‡ marks the highest-entropy point for pure x1x_1 prediction at NFE 1 and 2. Points below its frontier gap are collapsed, with entropy ≤2.8\le 2.8. At NFE 8 and 16, that arm's displayed points have entropy 3.89 and 4.01. See appendix Table R7.

ArmNFE 1NFE 2NFE 8NFE 16
Full DBTM208.6 (4.00)188.2 (4.10)103.1 (4.00)87.6 (4.00)
(i) Transport off289.6 (4.00)368.1 (4.10)311.4 (4.00)291.6 (4.00)
(ii) Transport off, anchor at all tt1436.0 (4.39)‡868.2 (4.38)‡45.6 (3.89)43.8 (4.01)
No anchor3784.1 (4.66)†2087.6 (4.47)†669.7 (4.05)†650.1 (4.04)†
Anchor at t⋆=0.82t^\star=0.82520.3 (4.36)†272.9 (4.22)†117.6 (4.00)95.5 (3.95)†

Table R2: Partially trained maps and autonomous flow trajectories. Results for one position with three tokens and target weights (0.6,0.3,0.1)(0.6,0.3,0.1), using an identity-initialized map trained with transport and boundary losses. s⋆s^\star is the time on the reference trajectory nearest Tθ(n)(x0)T_\theta^{(n)}(\boldsymbol{x}_0). See appendix Table R2.

Training step1019397915831620,000
Median s⋆s^\star0.200.350.650.750.850.950.95
Median distance / path length0.320.260.250.130.130.100.09
IQR of s⋆s^\star across points0.400.500.550.500.550.500.50

Table R3: Agreement with the transport map and argmax. Results for the paper's softmax head trained with transport, boundary and anchor losses, using 4000 test points per setting. TV is the total-variation distance between each map's output distribution and the data distribution. See appendix Table R3.

SettingT≠T\neq argmaxTV: true TTTV: argmaxTV: trainedAgrees with TTAgrees with argmax
uniform, 1 position, 3 tokens0%0.0070.0070.0320.9610.961
weights (0.6, 0.3, 0.1)31%0.0040.2590.0130.9760.707
2 correlated positions, 4 tokens51%0.0250.4560.0360.8890.471

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.