- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Seq-Drifting for one-step text generation. Compared with multi-step diffusion language models, it directly maps Gaussian noise to the full token sequence in a single forward pass, which could significantly reduce the inference time incurred by multi-step dLLMs. An essential component of Seq-Drifting is the procedure for constructing the training loss. It combines several techniques, including spherical geometry, context-dependent token support sets from a frozen language model, and attractive and repulsive forces. The paper also provides a relatively comprehensive suite of evaluations, including unconditional/conditional generation, translation and summarization, and reasoning. The results show competitive Gen-PPL and substantial inference speedups.
Critical strengths
-
The numerical results provide convincing evidence of the inference efficiency of Seq-Drifting, achieving competitive Gen-PPL with a single forward pass and significant inference-time speedups over few-step dLLMs. This addresses a crucial bottleneck in dLLM inference.
-
The design of the method is interesting. In particular, using context-dependent support sets to provide adaptive supervision instead of fixed ground-truth tokens is an interesting choice. It is also nice to see that the ablation studies examine the effects of spherical geometry, on-policy support, and intra-sequence repulsion.
Critical weaknesses
-
The proposed method resembles on-policy imitation learning. Therefore, a potential weakness is that its performance may depend heavily on the frozen autoregressive teacher that provides plausible reference tokens. Indeed, this sensitivity is reflected in the results: LM1B Gen-PPL improves from 93.81 (same-corpus teacher) to 36.01 (pretrained GPT-2). What remains unclear is the benefit beyond distillation from a teacher. Would it be possible to test a simpler one-step distillation baseline with the same teacher, student architecture, and training budget, varying only the training objective? This could help isolate the contribution of the proposed drifting objective compared with a standard distillation objective.
-
The primary generation metrics used in the experiments are Gen-PPL and unigram entropy, with Self-BLEU also reported in the appendix. These metrics may not fully capture the quality of generated text, especially how well it matches the ground-truth distribution. For example, in the anti-repetition ablation in Table 2, both Gen-PPL and entropy decrease substantially, indicating that lower Gen-PPL does not necessarily imply better generation quality. It would be helpful to report additional distributional metrics, such as MAUVE [1], to provide more convincing evidence of generation quality and diversity. [1] Pillutla et al., "MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers."
Questions for authors
-
How does Seq-Drifting compare against standard one-step knowledge distillation using the same frozen teacher? What performance improvement is attributable specifically to the proposed drifting objective?
-
Could the authors report MAUVE or another distributional quality metric? How does Seq-Drifting compare with baselines at comparable diversity levels?
-
Appendix A.2 establishes that the drifting field vanishes when the generated distribution matches the data distribution, under certain additional assumptions. However, does the converse hold? In particular, could the field also vanish for an incorrect distribution if generated embeddings coincide with tokens in their respective support sets? Are there additional conditions that guarantee convergence to the target distribution?
3 comments
Thank you to the authors for the responses and additional experiments. I appreciate the efforts to address my concerns, particularly the new knowledge distillation baseline and MAUVE evaluations. The clarification regarding the scope of the fixed-point analysis is also helpful. Overall, I find the responses satisfactory and believe they have strengthened the paper. I will maintain my original score.
We thank the reviewer for the positive follow-up. Your thoughtful questions and suggestions have greatly improved the paper. We sincerely appreciate the time and efforts you put into the review.
We thank the reviewer for the careful review and positive feedback of our inference efficiency, method design, and experiments. We appreciate the time and thought devoted to these constructive suggestions.
W1&Q1: Comparison with a one-step knowledge distillation baseline
Thank you for this insightful question. Seq-Drifting$^{S}$ uses a support model trained from scratch on the same corpus as the baselines. On LM1B, it achieves Gen-PPL 93.81 in one step, compared with 98.76 for four-step FMLM. Thus, this gain does not rely on external pretrained teachers.
To further address your question, we introduce an one-step knowledge distillation baseline, with the same frozen teacher, generator architecture, and number of training steps. We conduct experiments on LM1B using either our GPT-2 trained from scratch on LM1B (S) or pretrained GPT-2 (P).
Unconditional generation (Sequence length=128)
Conditional generation (64-token prefix → 64-token continuation)
We observe that direct knowledge distillation performs poorly. For unconditional generation, this is probably because each position is distilled largely independently from a teacher distribution, so the generator receives little guidance about dependencies among its jointly generated tokens. In contrast, our context-dependent drifting field supplies plausible next-token candidates under the generator's own current context, aligning supervision with the sequence it produces.
For conditional generation, KD may favor a few frequent teacher continuations for a given prefix, leading to repetitive outputs and low entropy. Our model instead constructs supervision under contexts formed by its own generated tokens, allowing different noise inputs to explore other plausible continuations, including those the teacher produces less frequently. These results highlight the value of our training target for coherent and diverse one-step generation.
Following your comment, we add this experiment in Appendix B.11 (Table 11).
W2&Q2: Report MAUVE or another distributional quality metric?
Following your suggestion, we add MAUVE metric below and in Table 6 of the revised paper.
Seq-Drifting achieves the highest one-step MAUVE on both datasets. On LM1B, Seq-Drifting$^{P}$ surpasses all reported two- and four-step baselines, while Seq-Drifting$^{S}$ surpasses all except four-step MDLM + Di4C. On OWT, Seq-Drifting$^{P}$ also outperforms most two- and four-step baselines. We have included these results in the revised paper.
At matched LM1B unigram entropy of 4.21, Seq-Drifting$^{S}$ has lower Gen-PPL than four-step FMLM, 93.81 versus 98.76 (Figure 3), and lower Self-BLEU, 0.066 versus 0.204 (Table 5).
Q3: Scope of the fixed-point analysis
Thank you for this thoughtful question. Appendix A.2 establishes that matching distributions yield a vanishing field under the stated support, representation, and zero-temperature assumptions. Theoretically, the converse implication does not generally hold. In other words, distribution matching is a sufficient, but not necessary, condition for a vanishing field: a vanishing field does not imply distribution matching. We agree that the situation you describe can occur. Following the fixed-point perspective of drifting models (Deng et al., 2026), our analysis aims to verify that the field introduces no further training perturbation when the distributions match. We add clarification of this purpose and the sufficient-condition interpretation in Appendix A.2.