acceptodds
Under review as a conference paper at ICLR 2027

Continuous Space Drifting for Discrete Diffusion Language Models

Abstract

Masked diffusion language models have become capable generators, motivating objectives that complement token reconstruction with sequence-level feedback. We present LD2LM (Large Drifting Diffusion Language Model), which adapts continuous feature-space supervision to an 8B discrete block-diffusion model. Building on TokenDrift's differentiable soft-token interface, LD2LM retains denoising cross-entropy (CE) and adds multiscale feature views across encoder depths and sequence regions. We also study how this supervision interacts with token- and feature-level KL constraints. Although TokenDrift reports benefits from feature-space Drifting, directly transferring its Drift-only recipe to DreamReasoner8B performs substantially worse than CE. We hypothesize that Drift-only feedback may not transfer directly to scaled discrete block-diffusion models. Drawing on scaling experience with richer representations, LD2LM combines continuous multiscale feature feedback with token-level CE. In a matched training-stage comparison, the Full schedule raises HumanEval from 74.39% to 75.00%, MBPP from 60.40% to 61.20%, and LiveCodeBench pass@1 from 44.84% to 49.26%; tokens per forward pass increase from 1.5490 to 1.5679, with the inference sampler unchanged. In a separate token-KL comparison, adding Drifting improves HumanEval from 73.78% to 75.61% and LiveCodeBench from 49.56% to 51.33%, while tying MBPP at 61.40%. These results show the promise of continuous multiscale feature feedback for adapting Drifting to scaled discrete block-diffusion models.

Primary area:
generative models
Keywords:
discrete diffusion language models, block diffusion language models, continuous representation learning, feature-space supervision, soft tokens, knowledge distillation, parallel decoding, language model training

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_BsZWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LD2LM that adapts drifting techniques to DreamReasoner8B. The method is directly built upon TokenDrift, while retaining the token-level cross-entropy loss and also combining with the multiscale feature drifting loss. Experiments are performed on top of DreamReasoner8B on code generation and math reasoning. Results show that drift-only training degrades performance while the CE loss and multiscale drifting loss preserves pass@1 and improves TPF.

Critical strengths
  1. The paper adapts drifting loss to 8B-scale diffusion language model and evaluated on conditional generation tasks like coding and math reasoning.

  2. The method is relatively simple and easy to implement.

Critical weaknesses
  1. The experiment setting is troublesome and the results are not convincing. Table 1 caption explicitly lists "Math uses strict Math-Verify 0.8.0 and historical, non-matched runs" which is confusing. Most of the experiment results only show marginal improvement or even degradation, e.g., the method actually hurts LiveCodeBench performance compared to CE and even underperforms the starting checkpoint.

  2. The novelty is limited and the method seems to be incremental to TokenDrift. The soft-token interface, FIFO queues, and the construction of the drifting field are already present in existing works. TokenDrift itself has also already explored a combination with CE loss. The experiment results also show that CE + feature KL often outperforms LD2LM and even matches LD2LM + tokenKL mostly.

Questions for authors
  1. Why does CE continuation baseline has a score of 51.03 on LiveCodeBench in Table 1 but only 44.84 in Figure 3 and Table 2?

  2. Why does LD2LM even hurt the performance on LiveCodeBench in Table 1 compared with the starting checkpoint?

  3. In Table 2, CE + feature KL outperforms LD2LM on LiveCodeBench and ties on HE and MBPP. If this is already simple yet effective enough, why adding the FIFO queues and sophisticated designs in the drifting field?

  4. The writings in the manuscript appear to be confusing and unnatural. A few examples below.

line 128: "provides the denoising foundation": what does denoising foundation mean?

line 138: "The distinction concerns the supervision space, rather than whether the model has access to sequence context": what does this mean?

line 207 and line 203 repeat the same sentence: "while the current hirh_{ir} remains differentiable"

line 300 unnatural narrative: "This is a history-length estimate, not a measured queue age. Queue capacity is a tested setting..."

The paper may require substantial rewriting.

Reviewer_XNB7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper adds TokenDrift-style feature-space "drifting" to cross-entropy training of DreamReasoner8B. Drifting attracts predictions toward clean features and repels them from generated features, using a frozen copy of the model as the feature encoder. They train add a low-rank LoRA to adapt the 8B model and report results on coding and math benchmarks.

Critical strengths
  • The research topic is important and improving the performance of discrete diffusion models is an interesting research direction. Furthermore, giving more training signal to the model (possibly introducing some form of continuous signal) is also interesting and under explored.
Critical weaknesses
  • The paper is very hard to understand. Information is scattered around multiple sections and the appendix. Lastly, Figure 2 is also confusing and hard to parse. It makes understanding the specific details and which parts of the proposed method actually improve performance.

  • Training details are confusing. What does "the full code set has 311,050 sequences" and "one drop-last pass takes 38,881 optimizer steps at global batch size eight" mean? Do they refer to the number of training sequences and how many steps one epoch takes? What is the sequence size used? Which LR scheduler was used? Was the global batch size 8? Having batch size 1 per device when using LoRA does not add up on H100s and that model size. Overall it is very hard to reproduce the experiments given the lack of important details.

  • Results need standardization. Currently, depending on the dataset, results are shown in different ways and with different amounts of information. For example, Figure 4 can just be a Pareto frontier that shows how different thresholds interact with the speed and performance of the method. It is hard to parse the figure as it currently is. Similarly, the authors can standardize Figure 5 with the same visualization of Pareto fronts for all datasets and different queue sizes. Currently it shows different information depending on the dataset. For LiveCodeBench it shows performance related to threshold and queue size, for the math datasets it omits the threshold. Additionally, what is the reason for queue size impacting so much math datasets? It has basically no effect on the code benchmark. Lastly, why are results different across different tables? For example, the control "CE continuation" and LD2LM have different results in Table 1 and Table 2.

  • Important experiments are also missing. Does the approach work for other models? How does the LoRA rank impact performance and how does fine-tuning the whole model perform?

Questions for authors

See weaknesses.

Reviewer_43UHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors consider applying drifting model loss for an 8B discrete diffusion language model closely following the TokenDrift approach. Specifically, they map token predictions to soft embeddings and use a frozen feature encoder to produce feature maps used to calculate drifting loss. The proposed method called LD2LM combines this drifting loss in multiple-level feature space with the standard denoising cross-entropy loss. The method is tested on code generation and mathematical tasks.

Critical strengths

None

Critical weaknesses

Limited novelty. The proposed method is almost the original TokenDrift [1]. The main modification is replacing the original feature space representation with multiple features of different encoder layers and sequence segments. However, there is no matched comparison using the original TokenDrift feature representation approach while keeping crossentropy in both methods. Furthermore, using drift loss together with cross-entropy loss is not novel and was studied in TokenDrift [1, Section 4.3].

Poor experimental results and inconsistent comparisons. According to Table 1, on all code setups (HE, MBPP, LCB) and math setups except AIME, the proposed approach either loses or be better only by 0-1.3% than CE continuation, which might not be a statistically significant improvement given the size of benchmarks. On AIME24 and AIME25, results are better by 3-7%, but AIME benchmarks consist only of 30 tasks, so the improvement is only in 1-2 solved tasks, which also might be statistically not significant.

The authors also study the addition of "token KL" loss, which should be a regularizer to the original starting model. However, the authors do not provide enough details to how this loss works and how it is added to the drifting and crossentropy losses. Furthermore, Figure 4 compares CE with CE + Drifting + token KL, so the improvements cannot be attributed to drifting alone.

[1] Oba, D., et al. Drifting Objectives for Refining Discrete Diffusion Language Models.

Questions for authors

Why does the CE-only baseline achieve 51.03% on LiveCodeBench in Table 1 but 44.84% in Figure 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.