- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors tackle the problem of developing a one-step flow generative model of text that can be trained without a teacher distillation or bootstrapping objective. Their key insight is to average over all interpolants over time to get a time-independent autonomous flow, and to learn this flow directly from the data by defining it as the solution of a conservation equation. The authors provide early results in unconditional language modeling where their DBTM starts to outperform FMLM at higher NFE, and outperforms FMLM on Sudoku and GSM8K.
Critical strengths
- The time-independent autonomous flow paradigm is conceptually very nice and allows the authors to avoid using either distillation or bootstrapping to train one-step flow LMs. A single stage is much cleaner and means that the teacher is no longer the performance ceiling, nor does it contribute approximation error during student training.
- It's also quite nice to explore how this paradigm can be used for few- rather than one-step generation, and the fixed-point refinement idea makes a lot of sense for this.
Critical weaknesses
- Could the authors provide training FLOP counts for the training of the various baselines and DBTM? It is very well-accepted at this point that the meaningful quantity to control for pretraining language models is compute, not solely parameters. This would make the empirical claims much more sound.
- Overall, the paradigm involves quite a lot of different loss terms, and it is sometimes unclear when the terms are required for having a consistent objective, versus when they happen to empirically help but there is no deep reason they are necessary: e.g., the anchor loss, semi-group loss, and refinement-in-the-loop objective. Could you comment on which of the losses are actually necessary for consistency, and for those loss terms which are not required for consistency, could you provide some more intuition in the paper on why they're empirically useful?
Questions for authors
Please see the weaknesses above for the two main asks (FLOPs accounting, and more discussion on consistency of the objective / when loss terms are necessary versus happen to help empirically).
3 comments
Thanks to the authors for a nice rebuttal. I appreciate the OWT pretraining experiment with controlled train FLOPs, which looks fairly convincing. Thanks also for the disambiguation of the loss terms into those that are needed for consistency or otherwise useful empirically.
I can appreciate that novel generative model work often starts out complicated, with many auxiliary losses, and is slowly simplified over time. I think the notion of training a one-step flow from scratch, without distillation or bootstrapping, is an exciting enough proposition that any successful training of such a model, even if it requires 5+ auxiliary losses, is a worthwhile contribution.
So I am happy to raise my score to clear accept.
Thank you for your support!
We are happy to hear that we have resolved your remaining concerns, and your suggestions have been super helpful in strengthening our paper! We also thank you for recognizing that our paper proposes a novel generative method, and we are also excited about the potential to build upon it.
Sincerely, The Authors
Author Response To Reviewer WyKQ (1/1)
Dear Reviewer WyKQ,
Thank you for your thoughtful and encouraging review. We are happy to hear you found the autonomous-flow paradigm conceptually appealing, valued the possibility of training a one-step map directly from data without an external pretrained teacher, and found the fixed-point refinement idea sensible for few-step generation.
We hope to clear up your remaining concerns with the additional experiments and responses below:
Yes, we have done so! The LM1B training FLOP estimates include teacher pretraining and student distillation for FMLM (See Table R10 in General Response). Although a base DBTM step costs 2.7× an FLM step per sample, the reported DBTM + ril pipeline uses approximately 6.8× fewer total training FLOPs, and plain DBTM uses approximately 14× fewer, owing to fewer training steps and smaller batches. To evaluate quality under a controlled training budget, we additionally trained both pipelines from scratch on OWT with the same backbone and approximately matched total training node-hours, counting all stages (See Tables R12–R13 in General Response). DBTM + ril achieves lower Gen-PPL at most evaluated matched-entropy comparisons and [W2] details the results for each backbone and budget.
Thanks for the great suggestion, we agree this is a useful way to organize the objective! Table R9 now separates all eight terms. The accompanying numbered list in the appendix explains each term and its significance.
Empirical Role of the Anchor
In the exact toy setting, training the softmax map on the transport and boundary losses alone reaches a degenerate solution: the softmax saturates, so the measured residual is near zero, but the basins are wrong, and agreement with the true T is only 37% with weights (0.6, 0.3, 0.1) and 49% with two correlated positions. Adding the anchor at t⋆ from Theorem 4.1 restores it to 98% and 89% (See Table R3 in General Response). At scale on LM1B, the anchor also improves generation: without it, Gen-PPL remains above 650 at every NFE, over a frontier entropy range of 4.04 to 4.69 (See Table R7 in General Response).
Contribution of the Transport Loss
Removing the transport loss raises Gen-PPL at matched entropy at every budget (289.6 vs. 208.6 at NFE 1, and 291.6 vs. 87.6 at NFE 16, both at entropy 4.0). Pure x1 prediction without it does not produce useful one- or two-step samples in the evaluated sweep, yet with the same refinement sampler it is better than DBTM at 16 NFE (43.8 vs. 87.6). See Table R7 in General Response and our reply to Reviewer Ngbi [W3].
Anchor Placement
Theorem 4.1 gives t⋆=0.82 for LM1B while our default is 0.75. Training at exactly 0.82 gives worse quality after 100K steps (117.6 vs. 103.1 at NFE 8 and entropy 4.0), and anchoring well below t⋆ collapses (0.60 in Table 12). The revision says the theorem sets the scale of tanchor and that the best value sits slightly below it.
We now also clarify that a map is uniquely determined within the stated solution class if it remains constant along each flow trajectory and approaches the correct vertex at the trajectory’s endpoint. These conditions characterize the ideal map but do not by themselves guarantee that neural network training recovers it.
Table R9: Roles of the training objectives.
If we have addressed your concerns, we hope you consider raising your score in stronger support of acceptance, and if any remain, we are happy to continue the discussion!
Thank you,
The Authors