acceptodds
Under review as a conference paper at ICLR 2027

One-Step Text Generation by Seq-Drifting

Abstract

Diffusion language models generate tokens in parallel, but still require multiple sampling steps to produce high-quality text. In this paper, we investigate whether a model can learn to generate an entire sequence in a single forward pass. We propose Seq-Drifting, a framework that applies drifting to sequences of continuous token representations and directly learns a map from Gaussian noise to text. Unlike existing DLMs that derive the supervision at each position directly from the observed token in a training sequence, Seq-Drifting builds a drift field whose position-wise targets adapt to the current generated context. At each position, attraction guides each output toward a set of contextually plausible token representations, while repulsion discourages collapse across samples and repetition within sequences. The generator learns from this drift field during training and produces all tokens jointly in one pass at inference. Empirically, Seq-Drifting achieves a generative perplexity of 36.01 on LM1B in one step and performs competitively in both unconditional and conditional generation. Experiments on translation, summarization, and reasoning further demonstrate the applicability of the framework beyond open-ended generation. These results suggest that drifting over token sequences offers a promising path toward effective one-step text generation.

Primary area:
generative models
Keywords:
Text Generation, Diffusion Language Model, Drifting

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_LYtEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Seq-Drifting for one-step text generation. Compared with multi-step diffusion language models, it directly maps Gaussian noise to the full token sequence in a single forward pass, which could significantly reduce the inference time incurred by multi-step dLLMs. An essential component of Seq-Drifting is the procedure for constructing the training loss. It combines several techniques, including spherical geometry, context-dependent token support sets from a frozen language model, and attractive and repulsive forces. The paper also provides a relatively comprehensive suite of evaluations, including unconditional/conditional generation, translation and summarization, and reasoning. The results show competitive Gen-PPL and substantial inference speedups.

Critical strengths
  1. The numerical results provide convincing evidence of the inference efficiency of Seq-Drifting, achieving competitive Gen-PPL with a single forward pass and significant inference-time speedups over few-step dLLMs. This addresses a crucial bottleneck in dLLM inference.

  2. The design of the method is interesting. In particular, using context-dependent support sets to provide adaptive supervision instead of fixed ground-truth tokens is an interesting choice. It is also nice to see that the ablation studies examine the effects of spherical geometry, on-policy support, and intra-sequence repulsion.

Critical weaknesses
  1. The proposed method resembles on-policy imitation learning. Therefore, a potential weakness is that its performance may depend heavily on the frozen autoregressive teacher that provides plausible reference tokens. Indeed, this sensitivity is reflected in the results: LM1B Gen-PPL improves from 93.81 (same-corpus teacher) to 36.01 (pretrained GPT-2). What remains unclear is the benefit beyond distillation from a teacher. Would it be possible to test a simpler one-step distillation baseline with the same teacher, student architecture, and training budget, varying only the training objective? This could help isolate the contribution of the proposed drifting objective compared with a standard distillation objective.

  2. The primary generation metrics used in the experiments are Gen-PPL and unigram entropy, with Self-BLEU also reported in the appendix. These metrics may not fully capture the quality of generated text, especially how well it matches the ground-truth distribution. For example, in the anti-repetition ablation in Table 2, both Gen-PPL and entropy decrease substantially, indicating that lower Gen-PPL does not necessarily imply better generation quality. It would be helpful to report additional distributional metrics, such as MAUVE [1], to provide more convincing evidence of generation quality and diversity. [1] Pillutla et al., "MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers."

Questions for authors
  1. How does Seq-Drifting compare against standard one-step knowledge distillation using the same frozen teacher? What performance improvement is attributable specifically to the proposed drifting objective?

  2. Could the authors report MAUVE or another distributional quality metric? How does Seq-Drifting compare with baselines at comparable diversity levels?

  3. Appendix A.2 establishes that the drifting field vanishes when the generated distribution matches the data distribution, under certain additional assumptions. However, does the converse hold? In particular, could the field also vanish for an incorrect distribution if generated embeddings coincide with tokens in their respective support sets? Are there additional conditions that guarantee convergence to the target distribution?

3 comments

We thank the reviewer for the careful review and positive feedback of our inference efficiency, method design, and experiments. We appreciate the time and thought devoted to these constructive suggestions.

W1&Q1: Comparison with a one-step knowledge distillation baseline

Thank you for this insightful question. Seq-Drifting$^{S}$ uses a support model trained from scratch on the same corpus as the baselines. On LM1B, it achieves Gen-PPL 93.81 in one step, compared with 98.76 for four-step FMLM. Thus, this gain does not rely on external pretrained teachers.

To further address your question, we introduce an one-step knowledge distillation baseline, with the same frozen teacher, generator architecture, and number of training steps. We conduct experiments on LM1B using either our GPT-2 trained from scratch on LM1B (S) or pretrained GPT-2 (P).

Unconditional generation (Sequence length=128)

MethodTeacherGen-PPL ↓Entropy
Seq-Drifting$^{S}$Same-corpus GPT-293.814.21
One-step KD$^{S}$Same-corpus GPT-24748.124.46
Seq-Drifting$^{P}$Pretrained GPT-236.014.42
One-step KD$^{P}$Pretrained GPT-29927.854.55

Conditional generation (64-token prefix → 64-token continuation)

MethodTeacherGen-PPL ↓Entropy
Seq-Drifting$^{S}$Same-corpus GPT-279.133.74
One-step KD$^{S}$Same-corpus GPT-2136.312.59
Seq-Drifting$^{P}$Pretrained GPT-276.164.40
One-step KD$^{P}$Pretrained GPT-2217.112.80

We observe that direct knowledge distillation performs poorly. For unconditional generation, this is probably because each position is distilled largely independently from a teacher distribution, so the generator receives little guidance about dependencies among its jointly generated tokens. In contrast, our context-dependent drifting field supplies plausible next-token candidates under the generator's own current context, aligning supervision with the sequence it produces.

For conditional generation, KD may favor a few frequent teacher continuations for a given prefix, leading to repetitive outputs and low entropy. Our model instead constructs supervision under contexts formed by its own generated tokens, allowing different noise inputs to explore other plausible continuations, including those the teacher produces less frequently. These results highlight the value of our training target for coherent and diverse one-step generation.

Following your comment, we add this experiment in Appendix B.11 (Table 11).

W2&Q2: Report MAUVE or another distributional quality metric?

Following your suggestion, we add MAUVE metric below and in Table 6 of the revised paper.

MethodStepsLM1B ↑OWT ↑
MDLM10.490.72
20.730.71
45.821.28
MDLM+SDTT10.480.54
20.780.54
43.104.93
MDLM+Di4C10.480.41
20.950.50
425.016.40
Duo10.550.55
20.451.51
44.9162.26
Duo+DCD10.431.33
21.094.30
41.6878.71
Duo+Di4C10.410.41
20.480.53
40.7318.62
FLM10.410.91
20.901.03
41.251.67
FMLM14.423.91
22.2965.33
415.9877.69
Seq-Drifting$^{S}$119.4115.04
Seq-Drifting$^{P}$134.4117.85

Seq-Drifting achieves the highest one-step MAUVE on both datasets. On LM1B, Seq-Drifting$^{P}$ surpasses all reported two- and four-step baselines, while Seq-Drifting$^{S}$ surpasses all except four-step MDLM + Di4C. On OWT, Seq-Drifting$^{P}$ also outperforms most two- and four-step baselines. We have included these results in the revised paper.

At matched LM1B unigram entropy of 4.21, Seq-Drifting$^{S}$ has lower Gen-PPL than four-step FMLM, 93.81 versus 98.76 (Figure 3), and lower Self-BLEU, 0.066 versus 0.204 (Table 5).

Q3: Scope of the fixed-point analysis

Thank you for this thoughtful question. Appendix A.2 establishes that matching distributions yield a vanishing field under the stated support, representation, and zero-temperature assumptions. Theoretically, the converse implication does not generally hold. In other words, distribution matching is a sufficient, but not necessary, condition for a vanishing field: a vanishing field does not imply distribution matching. We agree that the situation you describe can occur. Following the fixed-point perspective of drifting models (Deng et al., 2026), our analysis aims to verify that the field introduces no further training perturbation when the distributions match. We add clarification of this purpose and the sufficient-condition interpretation in Appendix A.2.

Thank you to the authors for the responses and additional experiments. I appreciate the efforts to address my concerns, particularly the new knowledge distillation baseline and MAUVE evaluations. The clarification regarding the scope of the fixed-point analysis is also helpful. Overall, I find the responses satisfactory and believe they have strengthened the paper. I will maintain my original score.

We thank the reviewer for the positive follow-up. Your thoughtful questions and suggestions have greatly improved the paper. We sincerely appreciate the time and efforts you put into the review.

Reviewer_deUHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Seq-Drifting, a framework for generating an entire text sequence in a single forward pass. It addresses the inference cost of diffusion language models, which generate tokens in parallel but typically require multiple sampling steps to achieve good generation quality.

The key idea is to train a generator that maps Gaussian noise directly to a sequence of continuous token representations. Rather than supervising each position solely with the observed token in a training example, Seq-Drifting constructs a context-dependent drift field using a set of plausible token candidates. The resulting objective combines attraction toward plausible token representations with repulsion across samples and across positions within a sequence. The paper evaluates Seq-Drifting on unconditional and conditional text generation, as well as downstream tasks including machine translation, summarization, and reasoning.

Critical strengths
  1. The paper addresses an important challenge in diffusion-based text generation. It reduces inference latency while retaining competitive generation quality. Generating a complete sequence in one forward pass is an appealing alternative to iterative sampling.

  2. Seq-Drifting combines context-dependent attraction and two forms of repulsion within a unified drift field over continuous token representations. This provides a clear methodological framework, and the individual components allow for targeted ablation and hyperparameter analyses.

  3. The paper evaluates the approach across unconditional and conditional generation, multiple downstream tasks, and several baselines.

Critical weaknesses
  1. The results vary across datasets. As shown in Figure 3, Seq-Drifting's results on OpenWebText are less attractive than those on LM1B. The two evaluations differ in sequence length (1024 tokens for OpenWebText versus 128 tokens for LM1B), making it difficult to determine whether the performance gap arises from the dataset, the longer generation horizon, or others.

  2. Figure 3 shows a substantial performance difference between the SS and PP settings, highlighting the importance of support-model choice. Although the comparison between Seq-Drifting$^S$ and FMLM provides useful evidence for the proposed method, it remains important to clarify how comparable the methods are in terms of training data, model capacity, supervision, and training compute. The support-model scaling results are also useful, but further analysis of how support-model size/performance affects generation quality.

  3. The proposed method uses cross-sample and intra-sequence repulsion, but their individual contributions and effects on generation quality are not fully discussed.

Questions for authors

Please see the weaknesses.

4 comments

We thank the reviewer for the thoughtful review and positive feedback on the importance of our one-step generation, the unified drift-field design, and our experimental coverage. We appreciate the time and thought devoted to these constructive suggestions.

W1: Performance gap arises from dataset, sequence length, or others?

We speculate that sequence length is a more important factor. Our generator learns from support-model fields on its current contexts, rather than directly fitting fixed dataset tokens, so the dataset itself is not the core reason. Generating 1024 tokens in a single step is more challenging than generating 128 tokens, while multi-step methods can repeatedly refine their outputs. However, even in this setting, our one-step model achieves comparable or better Gen-PPL than many few-step baselines, highlighting the potential of our approach for efficient long-sequence generation.

To address this concern, we additionally conduct an OWT-128 experiment below and in Table 7. The results show that shorter sequences yield lower Gen-PPL for both variants, supporting our explanation.

DatasetTSeq-Drifting$^{S}$ Gen-PPL ↓EntropySeq-Drifting$^{P}$ Gen-PPL ↓Entropy
LM1B12893.814.2136.014.42
OWT128109.834.0549.134.66
OWT1024183.984.95128.335.08

W2: Training data, model capacity, supervision, and training compute compared with FMLM. Effect of support-model size/performance

  • Training data. Seq-Drifting$^{S}$ trains its GPT-2 support model from scratch on exactly the same corpus as FMLM, then freezes it to supervise the generator. Thus, Seq-Drifting$^{S}$ and FMLM use the same training corpus.

  • Model capacity. The number of parameters are listed as follows, which indicates that model capacity is not the source of our advantage.

ModelParameters (M)
Ours187
FMLM179
  • Supervision. This difference is part of our contribution. We construct a context-dependent drift field combining attraction to plausible tokens with repulsion against collapse and repetition. FMLM instead distills a continuous denoising flow into a map between timepoints. The methods share the training corpus but learn through different objectives.

  • Training budget. We train the generator for 200k steps for unconditional generation and 300k steps for conditional generation, including warm-up. Adding 100k support-model training steps, Seq-Drifting requires 300k and 400k total steps, respectively. FMLM needs >1M total steps. Under the same setting (4xH200 GPUs, global batch size 2048, sequence length 128), the average training time per step is 0.643 s for FMLM, and 0.661 s for Seq-Drifting. Our training budgets are therefore only approximately 27.3% and 36.4% of FMLM's, respectively.

How support-model size/performance affects generation quality. Better support models provide more informative candidate sets for the drifting field. To quantify this, conditioned on the source and reference prefix, we measure reference-token coverage (RC), mean reference-token probability (RP), and mean non-reference candidate count per position (NC).

TaskSupport modelRC ↑RP ↑NC ↓
WMTGPT-2 Small87.29%51.68%5.05
WMTGPT-2 Large91.30%56.05%4.42
XSumGPT-2, Small84.82%42.51%6.78
XSumGPT-2, Large85.55%44.61%6.39

We also provide two cases to illustrate this.

Case 1 · WMT: lexical grounding

Source: “Rat nimmt Gleisanlagen ins Visier”

Prefix: “Council sets its sights on”

TokenGPT-2 SmallGPT-2 LargeSupport change
rail (reference)0.14%18.66%Added
visa19.84%<1%Removed

Case 2 · XSum: entity selection

Source cue: “Forest Green are currently second in the National League table...”

Prefix: “National League side”

TokenGPT-2 SmallGPT-2 LargeSupport change
Forest (reference)67.74%93.35%Retained
Bristol2.11%0.65%Removed
Blackburn1.33%0.19%Removed

Together, these statistics and cases show that better support models improve candidate selection and provide more reliable guidance for the drifting field. We add these analyses in Figure 6 and Section 5.6.

W3: Ablation on cross-sample repulsion and intra-sequence repulsion

Table 2 shows that removing intra-sequence repulsion ("w/o intra") leads to model collapse. We add ablation on cross-sample repulsion as below. Without cross-sample repulsion, different noise inputs produce the same sequence (Self-BLEU=1). Thus, intra-sequence and cross-sample repulsion prevent collapse within sequences and across noise inputs, respectively.

VariantGen-PPL ↓EntropySelf-BLEU ↓Gen-PPL ↓EntropySelf-BLEU ↓
Seq-Drifting36.014.420.04276.164.400.065
W/o intra-sequence repulsion (Table 2)10.011.820.04612.181.310.025
W/o cross-sample repulsion31.453.171.000204.63.831.000

Dear Reviewer deUH,

Thank you again for your thoughtful review and encouraging feedback. As the discussion period draws to a close, we would like to express our sincere appreciation for your time and consideration. Should any further clarification be helpful, we would be very happy to provide it. Thank you again for your time and expertise.

Thanks for the clarification and the new results. In my initial review, this paper proposed a nice approach to address an important challenge of diffusion language models. The response further addressed some of my concerns. I maintain my positive rating of 3 for weak acceptance.

We thank the reviewer for the positive follow-up. Your thoughtful questions and suggestions have greatly improved the paper. We sincerely appreciate the time and efforts you put into the review.

Reviewer_ukL3OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Seq-Drifting, a method for one-step text generation via drifting over continious tokens. The method constructs the attraction to positives via sampling candidates from a pretrained autoregressive model. Token embeddings are projected to unit sphere thus allowing to overcome the difficulties of adopting drifting to discrete sequences. The authors ablate the architectural choices on a 128 token lengths sequences. They present their main results on common language modeling benchmarks such as OpenWebText (1024 tokens) and LM1B (128 tokens). They additionally include experiments on various conditional tasks, such as summarization, translation and reasoning.

Critical strengths

To my knowledge, it is the first adaptation of drifting models to language. I find it to be a promising and exciting new direction of research for one-step generators.

I find the techniques that the authors implemented to alleviate the challenges of adaptation drifting to language to be natural and well motivated. Both the sphere geometry and context-dependent support tokens in particular are rather elegant solutions.

The method's performance is remarkable. Both conditional and unconditional one-step generations either perform on-par or outperform strong contemporary baselines. Sequence-to-sequence tasks in some settings even outperform many-steps baselines.

Experiments are extensive and cover a wide fraction of different tasks, allowing for adequate comparison with the baselines.

Critical weaknesses

Despite the aforementioned strengths, there are concerns to be adressed. Most of them touch the conditional tasks performance and the details of evaluated metrics.

GPT-2 was chosen as a support model −- which is an understandable choice. However, since Seq-Drifting is essentially a distillation method, using, for instance, MDLM, Duo or FLM as the support model would provide clearer insight on Seq-Drifting performace comparing to other methods.

Another major concern is the conditional open-ended generation metrics. Generational perplexity of the conditional task (Table 1) is significantly higher than on the unconditional. I could not find the particular evaluation protocol, so I will assume that for conditional generation GenPPL is computed over only generated tokens with conditioning on the prefix. If my understanding is correct, this seems counterintuitive, especially if we take conditonal tasks performance into account. It would be great if the authors clarify the setup of these metrics evaluation. It would be even more helpful to add distributional metrics such as MAUVE. Otherwise, this behaviour would raise questions.

Considering all of the discussed strengths and weaknesses, I would rate the paper as weak accept.

Questions for authors
  1. Adding common pre-trained model as a support model (such as MDLM or Duo) would strengthen the contribution
  2. Adding more extensive and clarified conditional open-ended generation metrics and specifications
  3. The comparison with multi-step language diffusion method lacks LDLM, a very strong baseline.
  4. The motivation of using cross-attention for conditioning instead of self-attention in-context conditioning is not clear to me. Have you experimented with it or am I missing some details of the method that do not allow for it?
4 comments

We thank the reviewer for the careful review and encouraging feedback on the novelty, our design choices, and the extensive evaluation. We appreciate the time and thought devoted to these constructive suggestions.

W1&Q1: Use diffusion language models (DLMs) as support model

Thank you for this insightful suggestion. Both Seq-Drifting and conventional DLM distillation use a two-stage pipeline: we first train a support model and then a one-step generator, while conventional approaches first train a multi-step DLM and then distill it into a few-step DLM.

We have carefully considered using pretrained DLMs as support models. However, our support construction needs plausible next-token candidates conditioned on the current generated prefix at each position. AR models naturally support this next-token prediction setting, whereas standard DLMs are not directly designed for it. Using DLMs as support models may therefore require additional adaptation.

We agree that other pretrained models are worth exploring, and our framework can readily use other pretrained AR support models. For a fair comparison, in our experiments, Seq-Drifting$^{S}$ trains its support model from scratch on the same corpus used by the baselines. Under this same-corpus setting, our full pipeline achieves LM1B Gen-PPL of 93.81 in one step, compared with 98.76 for four-step FMLM, with an 11.2× wall-clock speedup (Figure 3). Thus, our improvement does not rely on external pretraining.

Following your comments, We add clarification of the support-model requirements in Section 4.2.

W2&Q2: Clarify conditional metric and add MAUVE metric

Yes, your understanding is correct. For each method, given an input prefix x=(x1,…,xt)x=(x_1,\ldots,x_t), the evaluated model generates a continuation y=(yt+1,…,yT)y=(y_{t+1},\ldots,y_T). GPT-2 Large then assigns each generated token a probability peval(yi∣x,yt+1:i−1)p_{\mathrm{eval}}(y_i\mid x,y_{t+1:i-1}). Gen-PPL is the exponential of the average negative log probability over all generated tokens in the evaluation set. Prefix tokens provide context but are not scored.

We speculate that the higher Gen-PPL in conditional generation arises because unconditional generation allows the model to generate its own topics and contexts, whereas conditional generation must produce continuations consistent with externally supplied prefixes. Even a locally fluent continuation may receive a lower probability from the evaluator if it is inconsistent with the prefix's topic, entities, or discourse.

Following your suggestion, we add MAUVE metric below and in Table 6 of the revised paper.

MethodStepsLM1B ↑OWT ↑
MDLM10.490.72
20.730.71
45.821.28
MDLM+SDTT10.480.54
20.780.54
43.104.93
MDLM+Di4C10.480.41
20.950.50
425.016.40
Duo10.550.55
20.451.51
44.9162.26
Duo+DCD10.431.33
21.094.30
41.6878.71
Duo+Di4C10.410.41
20.480.53
40.7318.62
FLM10.410.91
20.901.03
41.251.67
FMLM14.423.91
22.2965.33
415.9877.69
Seq-Drifting$^{S}$119.4115.04
Seq-Drifting$^{P}$134.4117.85

Seq-Drifting achieves the highest one-step MAUVE on both datasets. On LM1B, Seq-Drifting$^{P}$ surpasses all two- and four-step baselines, while Seq-Drifting$^{S}$ surpasses all except four-step MDLM+Di4C. On OWT, Seq-Drifting$^{P}$ also outperforms most two- and four-step baselines.

Q3: Comparison with LDLM

Thank you for identifying this valuable baseline. We add this paper and discuss it in the related work. We also evaluate its public checkpoints at 1/8/1024 steps as follows and in Table 8. We observe that Seq-Drifting$^{S}$ achieves lower Gen-PPL and Self-BLEU than eight-step LDLM on both datasets, and Seq-Drifting$^{P}$ outperforms 1024-step LDLM in LM1B Gen-PPL. These results also suggest that there remains room to further improve distribution matching in our one-step method to reach the level of multi-step DLMs.

DatasetMethodStepsGen-PPL ↓EntropySelf-BLEU ↓MAUVE ↑
LM1BLDLM115.061.410.9330.41
LDLM8137.132.240.4810.58
LDLM102464.674.360.15198.48
Seq-Drifting$^{S}$193.814.210.06619.41
Seq-Drifting$^{P}$136.014.420.04234.41
OWTLDLM18.700.850.7691.09
LDLM81315.494.980.2211.61
LDLM102425.955.350.22399.00
Seq-Drifting$^{S}$1183.984.950.14615.04
Seq-Drifting$^{P}$1128.335.080.20617.85

Q4: Conditioning architecture

Thank you for highlighting this ambiguity. Our generator uses both mechanisms: we first apply a single cross-attention layer before the transformer stack, followed by bidirectional self-attention over concatenated prefix and noise-derived output representations. This design is a simple implementation choice rather than a requirement of the drifting formulation. We have revised the wording in Section 4.5 and provided a clearer architectural description in Appendix A.1.

Dear Reviewer ukL3,

Thank you again for your thoughtful review and encouraging feedback. As the discussion period draws to a close, we would like to express our sincere appreciation for your time and consideration. Should any further clarification be helpful, we would be very happy to provide it. Thank you again for your time and expertise.

Reviewer_ukL3OpenReview ↗

Rebuttal Responce

Dear authors,

I thank you for your detailed rebuttal and apologize for the delayed answer. My concerns were quite extensively covered. I believe that the proposed method has great potential in the field. Despite current unperperformance in conditional open-ended generation, it still outperforms most baselines in the one-step regime.

Considering all of the aforementioned discussion, I raise my score.

Thank you for your encouraging feedback and for raising the score. We're glad that our responses have addressed your concerns. Your thoughtful questions and suggestions have greatly improved the paper. We sincerely appreciate the time and efforts you put into the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.