- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Gumbel Straight Flow (GSF), a method for distilling autoregressive language models into few-step parallel flow-map generators. The key idea is to construct noise–data pairs using the Gumbel coupling induced by an autoregressive teacher, rather than independently pairing noise and text. The authors theoretically show that this coupling produces non-intersecting straight trajectories under ideal conditions, and combine it with a flow-map consistency objective for efficient generation. Experiments on OpenWebText, LM1B, TinyStories, conditional text completion, and XSum summarization demonstrate improvements over several existing few-step generation and distillation baselines.
Critical strengths
-
Strong theoretical motivation and novelty: The paper provides an interesting connection between autoregressive Gumbel sampling and straight trajectories in continuous flow matching. The non-intersection analysis offers a principled explanation for why the proposed noise–data coupling may make few-step generation easier to learn.
-
Strong few-step results and evaluation: GSF demonstrates substantial improvements over FMLM and Gumbel Distillation, particularly at one sampling step. The paper includes multiple datasets, conditional generation tasks, distributional metrics, LLM-judge evaluations, and ablations, providing meaningful empirical support for the approach.
Critical weaknesses
-
Gap between theoretical straightness and practical generation quality: The paper establishes non-intersecting straight trajectories for the ideal AR-induced coupling, supported by empirical straightness measurements in Figure 5. However, the final one-step flow-map model still produces text with limited coherence, as illustrated in Appendix H. It remains unclear how closely the trained student approximates the ideal flow and what primarily limits its ability to retain the AR teacher's generation quality.
-
Limited breadth of downstream evaluation: While the paper demonstrates strong improvements on generative perplexity, distributional metrics, conditional text completion, and XSum summarization, the downstream evaluation is relatively narrow. In particular, it remains unclear whether the improvements in few-step generation translate to broader language-model capabilities such as question answering, reasoning, or instruction following. Additional task-level evaluation would strengthen the practical significance and generalizability of the proposed approach.
Questions for authors
-
Figure 5 demonstrates improved empirical straightness for GSF-FM, while the final GSF model additionally uses a flow-map consistency objective. Can the authors clarify whether similar trajectory-straightness diagnostics hold for the final trained flow-map model?
-
The paper demonstrates promising results on XSum summarization and conditional text completion. Could the authors provide additional evaluation or discussion of generalization to other conditional generation tasks, such as question answering, to better establish the broader applicability of the method?
-
Appendix D discusses the remaining generation-quality gap relative to the AR teacher, including the challenges of high-dimensional one-hot representations and sensitivity to token prediction errors. Could the authors provide further empirical analysis to identify which factors most strongly contribute to this gap, and whether increasing student capacity helps mitigate it?
3 comments
Official Comment by Authors for Reviewer M6ab (2/3)
Q1 [Straightness of FlowMap]:
(Question 1): Figure 5 demonstrates improved empirical straightness for GSF-FM, while the final GSF model additionally uses a flow-map consistency objective. Can the authors clarify whether similar trajectory-straightness diagnostics hold for the final trained flow-map model?
[Straightness measure] We also measured the trajectory straightness of GSF and FMLM flow maps under the same linear schedule. Both methods show improved straightness over their flow-matching counterparts (at temperature 1, with optimal-temperature results in parentheses), and GSF remains straighter than FMLM. This suggests that the underlying straight-path property carries over to the learned flow maps.
| Method | Straightness ↓ |
|---|---|
| FLM | 4.80 (1.82) |
| GSF-FM | 3.22 (0.70) |
| FMLM | 0.74 |
| GSF-FlowMap | 0.48 |
W2 & Q2 [Limited Downstream task]:
(Weakness 2): Limited breadth of downstream evaluation: While the paper demonstrates strong improvements on generative perplexity, distributional metrics, conditional text completion, and XSum summarization, the downstream evaluation is relatively narrow. In particular, it remains unclear whether the improvements in few-step generation translate to broader language-model capabilities such as question answering, reasoning, or instruction following. Additional task-level evaluation would strengthen the practical significance and generalizability of the proposed approach.
(Question 2): The paper demonstrates promising results on XSum summarization and conditional text completion. Could the authors provide additional evaluation or discussion of generalization to other conditional generation tasks, such as question answering, to better establish the broader applicability of the method?
[Instruction-Following Task] We chose the instruction following as a comprehensive evaluation covering diverse tasks. Following CSD (Kim et al., 2026), we distill a 124M AR student and a 170M GSF student from the same 1.5B AR teacher on Dolly-15K, which includes eight categories: open_qa, general_qa, classification, closed_qa, brainstorming, information_extraction, summarization, and creative_writing. We evaluate both models on the Dolly test split and four separate instruction-following benchmarks (Self-Inst, Vicuna, Sinst, UnNI) to assess generalization beyond the training data.
At 32 steps, GSF achieves an average ROUGE-L of 22.26, outperforming the AR (CSD) student (21.34) and approaching the teacher (23.33), while being 1.22× faster than the AR student (184.0 vs. 223.7 ms) and 36× faster than the teacher (6681 ms). At 16 steps, GSF is 2.43× faster than the AR student (92.0 ms) and still achieves higher ROUGE-L (21.87), and it outperforms MDLM at 16 steps in both latency (92.0 vs. 148.3 ms) and ROUGE-L (21.87 vs. 19.06) with 3.3× fewer parameters. At 8 steps, GSF is 4.86× faster (46.0 ms) with only a small drop in ROUGE-L (20.78). GSF also outperforms the AR student on four of the five benchmarks at 32 steps. These results show that GSF applies to diverse conditional generation tasks beyond summarization and text completion while keeping efficient few-step generation. They also suggest that the teacher–student gap may be smaller when distillation targets downstream tasks, which may require less student capacity than general pre-training.
| Model | Params | Steps | Latency (ms) ↓ | Dolly | Self-Inst | Vicuna | Sinst | UnNI | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Teacher | 1.5B | 256 | 6681 | 26.58 | 14.20 | 16.50 | 27.61 | 31.77 | 23.33 |
| MDLM | 554M | 16 | 148.3 | 19.95 | 11.06 | 16.13 | 23.70 | 24.44 | 19.06 |
| AR (KL) | 124M | 256 | 223.7 | 23.69 | 10.18 | 14.99 | 16.71 | 19.49 | 17.01 |
| AR (CSD) | 124M | 256 | 223.7 | 25.16 | 13.18 | 15.57 | 25.51 | 27.29 | 21.34 |
| GSF (Ours) | 170M | 8 | 46.0 | 24.94 | 12.10 | 17.06 | 23.98 | 25.80 | 20.78 |
| GSF (Ours) | 170M | 16 | 92.0 | 25.69 | 12.54 | 17.28 | 26.13 | 27.70 | 21.87 |
| GSF (Ours) | 170M | 32 | 184.0 | 26.00 | 12.76 | 16.92 | 27.06 | 28.57 | 22.26 |
We have included these results in Table 7 of the revised manuscript, provided further details on instruction following in Appendix C, and reported the wall-clock latency–performance trade-off in Figure 12. Qualitative samples are provided in Appendix J (Table 23).
Official Comment by Authors for Reviewer M6ab (3/3)
Reference (Cited in the Rebuttal)
-
Zhang, C., Hu, X., Liu, B., & Liu, Q. (2026). Gumbel Distillation for Parallel Text Generation. International Conference on Learning Representations (ICLR 2026).
-
Lee, C., Yoo, J., Agarwal, M., Shah, S., Huang, J., Raghunathan, A., ... & Kim, J. (2026). Flow Map Language Models: One-Step Language Modeling via Continuous Denoising. Advances in Neural Information Processing Systems (NeurIPS 2026).
-
Kim, Y., Shin, D., Kang, M., Na, B., & Moon, I.-C. (2026). Distillation of Large Language Models via Concrete Score Matching. International Conference on Learning Representations (ICLR 2026).
Thank you again for your valuable feedback. Please feel free to ask any further questions. Note that we have highlighted all changes in the manuscript in .
Official Comment by Authors for Reviewer M6ab (1/3)
We thank Reviewer M6ab for your careful evaluation and constructive feedback. We particularly appreciate the reviewer's recognition of the following strengths:
We address the reviewer's concerns as follows.
[Theory on practical straightness] Theorem 1 characterizes the optimally trained model, our learning target. For imperfectly trained models, the new Proposition 2 bounds learned trajectory straightness by the FM loss. The non-intersecting property enables a lower FM-loss upper bound, consistent with the reduced empirical training loss (Fig. 5a) and improved empirical straightness (Fig. 5b).
Proposition 2 (Straightness and flow matching loss). Let (x0,x1)∼π for any coupling π with finite second moments, let vθ(⋅,t) be K-Lipschitz for all t∈[0,1] (K>0), and let {xtθ} solve dxtθ=vθ(xtθ,t)dt with x0θ=x0. Then
\le C _K \mathcal{L} _{\mathrm{FM}}(\theta;\pi), \qquad C _K := 2+\frac{1}{2}(e^{2K}-1-2K).$$ **[Finite model capacity]** Since Proposition 2 bounds trajectory straightness by the FM loss, reducing this loss in practice is crucial, with model capacity being a key factor. The student uses 1,024× fewer sequential network evaluations than the AR teacher, limiting its computational expressivity and potentially contributing to the performance gap. **[Analysis on teacher-student capacity gap]** To empirically verify that capacity explains the gap, we fixed the teacher at size S (Gen-PPL=46.5) and varied only the student size (initialized from scratch). As student model capacity (parameters × steps) increased, performance gradually approached that of the teacher. | Gen-PPL ↓ | 1-step | 2-step | 4-step | |-|:-:|:-:|:-:| | S → S (12 layers) | 171.5 | 177.3 | 159.4 | | S → M (24 layers) | 139.6 | 119.3 | 114.8 | | S → L (36 layers) | 114.1 | 109.9 | 103.3 | | S → L (36 layers, more training) | 86.2 | 78.5| 75.7 | We also fixed the student size (AR-init.) and varied the teacher size (S: 46.5, M: 32.05, L: 27.81 Gen-PPL): | Gen-PPL ↓ | 1-step | 2-step | 4-step | |-|:-:|:-:|:-:| | S → S | 114.9 | 105.6 | 99.8 | | M → S | 119.7 | 112.3 | 104.6 | | L → S | 114.7 | 106.7 | 101.2 | At 4 steps, student scaling (S → L) improved Gen-PPL (159.4 → 75.7), while teacher scaling produced almost identical performance. Larger teachers yielded no significant gains, further supporting student capacity as the main bottleneck. **[Clarification on the goal]** The primary goal of distillation is not to match the teacher's absolute performance, but to improve the efficiency–performance frontier, as shown in Figure 1. Figure 1 shows our main empirical contributions: (1) improving distillation over Gumbel-Distill (Zhang et al., 2026) using the AR teacher, and (2) outperforming FMLM (Lee et al., 2026) through AR teacher supervision with the same student parametrization. We also outperform compute-matched AR models. **[Comparison to compute-matched AR and coherence]** We evaluated the wall-clock latency–performance trade-off with KV caching and sufficiently large batch sizes. As shown in Figure 11 of the revised manuscript, GSF achieves a better latency–performance trade-off than AR models, confirming the Pareto frontier advantage illustrated in Figure 1. In particular, compared with the most efficient AR (3.1M, 1024 steps) model in Figure 1, our 2-step and 4-step GSF models achieve higher efficiency while outperforming it in both coherence and overall winning rate according to LLM-judge evaluations: | Method | Params × Step Efficiency | Wall-clock Speed-up | Coherence | Overall | |-|:-:|:-:|:-:|:-:| | GSF-2 | **9.45×** | **2.41×** | **71.3%** | **64.3%** | | GSF-4 | **4.72×** | **1.53×** | **81.0%** | **84.4%** |