acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Inefficient Training of Diffusion Language Models

Abstract

Diffusion language models (DLMs) need to be trained far longer than autoregressive models (ARMs) to reach the same held-out loss, despite their training objectives sharing a Bayes-optimal value. Proposed explanations include gradient noise, competition among corruptions and hard prediction problems; we test all three. We diagnose this training efficiency gap with a noisy quadratic model of training dynamics and find that it is largely explained not by gradient noise but by the features the DLM has learned, which permit far less loss reduction than an ARM's features taken at the same loss. This deficiency does not arise from averaging over corruptions, whose gradients point in the same direction, but from corruptions at high rates, which carry most of the DLM's loss in errors that its features can do little to correct. To distinguish limited expressivity from a failure to learn good features, we design Surface Sudoku, a Sudoku variant whose digits are latent variables. DLM training stalls at a loss far above an ARM's, while an auxiliary objective that supervises the latent digits helps close the efficiency gap. Our findings challenge the prevailing beliefs about the cause of DLM training inefficiency and suggest that closing the gap requires learning better features, through methods such as auxiliary objectives or adapting ARMs.

Primary area:
generative models
Keywords:
Diffusion Language Models, Training Dynamics, Noisy Quadratic Model, Feature Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_23xuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates why diffusion language models (DLMs) require substantially more training compute than autoregressive models (ARMs) to reach comparable held-out loss, despite sharing the same Bayes-optimal objective value. The authors introduce a noisy quadratic approximation based on the local Gauss–Newton geometry to decompose training progress into attainable loss reduction, optimizer progress, and gradient-noise cost. Their analysis indicates that gradient noise accounts for only part of the efficiency gap; the dominant contribution is a smaller attainable decrease under fixed tangent features. They further show that gradients across different corruption rates are strongly aligned, ruling out simple competition among denoising tasks as the primary cause. The deficit instead concentrates at high corruption rates, where much of the loss resides in errors that the current features cannot effectively reduce. Finally, the authors introduce Surface Sudoku, a controlled synthetic task with latent com- positional structure. Auxiliary digit supervision, structured cell masking (with appropriate initialization), and initialization from pretrained models substantially improve diffusion training, in some cases matching ARM performance. The authors interpret these results as evidence that inefficient feature learning is a central bottleneck.

Critical strengths

--The noisy quadratic framework provides a clean decomposition of the training-efficiency gap.

--The result that corruption gradients remain strongly aligned across rates, particularly in the curvature band containing the attainable-decrease gap, challenges the hypothesis that diffusion training is inefficient because different denoising tasks compete.

--Surface Sudoku is a well-motivated synthetic benchmark that makes the missing representation explicit.

Critical weaknesses

--Can the authors elaborate more on “feature learning deficiency”?

Conceptual Framing: Feature Learning vs. Kernel–Task Alignment My main concern is the interpretation of the results as demonstrating a general “feature learning deficiency.” The central diagnostic measures differences in local tangent geometry and the attainable decrease supported by the checkpoint’s features. This establishes that the DLM has less favorable local gradient–curvature geometry than the ARM at matched loss, but does not by itself identify the mechanism responsible for producing these features. An alternative and potentially more specific explanation is that standard diffusion corruption induces a training geometry poorly aligned with the latent structure of the data. Surface Sudoku provides suggestive evidence consistent with this interpretation. Under character- level masking, the DLM plateaus substantially above the ARM, whereas masking complete three-character cells, together with an appropriate initialization, allows diffusion to match ARM performance. This suggests that training efficiency depends strongly on whether the corruption process respects the compositional units of the task. Auxiliary supervision and ARM initialization can also be interpreted as interventions that guide optimization toward more favorable tangent-kernel geometries. However, the paper does not directly measure how these geometries evolve under the different interventions. In this sense, deficient tangent features may describe the observed optimization bottleneck, while corruption–task mismatch may help explain why such features emerge in the first place. These interpretations are not mutually exclusive, but they operate at different explanatory levels. I encourage the authors to distinguish more clearly between the observed limitations of the local tangent geometry and the broader claim that DLMs generally fail to learn useful representations. Investigating corruption-induced kernel–task mismatch as a potential underlying mechanism would strengthen the paper’s conceptual contribution.

--Auxiliary supervision and ARM initialization can also be interpreted as interventions for optimization. Can the authors elaborate on how these geometries evolve under the different interventions?

-(Kernel–task) Could the observed efficiency gap be understood as a mismatch between corruption-induced tangent-kernel geometry and task structure?

Questions for authors

--How sensitive are the attainable-decrease ratios to the dimension and construction of the optimization subspace?

3 comments

We thank the reviewer for their thorough and thoughtful review, and we are glad they appreciated our experimental analysis and findings.

Can the authors elaborate more on "feature learning deficiency"? [...] My main concern is the interpretation of the results as demonstrating a general "feature learning deficiency." [...] This establishes that the DLM has less favorable local gradient–curvature geometry than the ARM at matched loss, but does not by itself identify the mechanism responsible for producing these features.

The intent of our work was not to identify the mechanism behind good vs. bad feature learning, but to trace the gap in optimization and compute inefficiency of DLM training relative to ARM training. We happened to find that feature learning was the determining factor. For greater clarity, we have expanded the explanation for what is meant by "feature learning deficiency" in the Plateau paragraph in Section 5.2, and discuss how our results support it as an alternative explanation to the hypothesis from prior work that hard prediction problems underlie DLM training inefficiency.

Essentially, we intend "feature learning deficiency" as a generic term for a failure to learn features which enable the model to reach a low loss, in contrast to the hard prediction problems hypothesis which would suggest that DLMs struggle to achieve the same loss as ARMs due to limitations in the expressivity of the model architecture/size. As noted in our conclusions, one promising avenue toward a theoretical, mechanistic understanding of feature learning deficiencies are the training dynamics of deep linear networks, which can model network abilities to escape saddle points and thereby escape loss plateaus; as saddle point escape is fundamentally non-convex optimization, it by definition implies that the features, the model Jacobians, must change. This analysis lies outside the scope of our work, whose focus is to properly assess and challenge the common explanations for the compute inefficiency of DLM training.

An alternative and potentially more specific explanation is that standard diffusion corruption induces a training geometry poorly aligned with the latent structure of the data. [...] Under character-level masking, the DLM plateaus substantially above the ARM, whereas masking complete three-character cells, together with an appropriate initialization, allows diffusion to match ARM performance.

We agree with the reviewer that alignment between the corruption and latent structure of the data could be a key mechanism behind the effectiveness of cell-masked training. However, we would like to emphasize that the role of "appropriate initialization" should not be understated, and that the corruption process alone does not determine whether good features are learned. Our Surface Sudoku results in Figure 6 show that changing the corruption on its own is not enough: whether the DLM trained with cell masking (which is presumably better aligned with the latent structure of the data) succeeds depends entirely on its weight initialization. This suggests a much more involved mechanistic understanding for when training does and does not succeed is necessary, and would be valuable future work.

Investigating corruption-induced kernel–task mismatch as a potential underlying mechanism would strengthen the paper's conceptual contribution. [...] Could the observed efficiency gap be understood as a mismatch between corruption-induced tangent-kernel geometry and task structure?

We agree with the reviewer that mechanisms explaining why good features are not learned, such as a kernel–task mismatch, are not mutually exclusive with our very general claim of a feature learning deficiency. However, as mentioned earlier, the importance of weight initialization implies that analyzing a corruption process's tangent-kernel alignment with the loss at initialization would not reliably distinguish good corruption processes from bad ones. Nevertheless, we do believe that designing corruption processes aligned with the latent structure of the data could be an effective way to induce good feature learning and close the efficiency gap between ARMs and DLMs, and our Surface Sudoku findings are intended to encourage such efforts.

We thank the reviewer again for their feedback, which has helped improve the clarity of our work. Please let us know if you have any further questions!

We are still working on new experiments on the sensitivity of our results to basis construction, and are conducting Section 3 style analysis on the Surface Sudoku experiments and interventions. We aim to update our submission with these results before the end of the discussion period.

We have included additional experimental results in the revised submission.

Auxiliary supervision and ARM initialization can also be interpreted as interventions for optimization. Can the authors elaborate on how these geometries evolve under the different interventions?

We have added Appendix I, which investigates how interventions such as adding a latent digit prediction auxiliary loss affect attainable decrease and feature change of character-masked models (Figure 21a-c) as well as cell-masked and AR models (Figure 21d-f). We find that attainable decrease and feature change are associated with loss decrease and plateau escape: training runs that are stuck on the plateau have a smaller feature change and attainable decrease than runs that reach lower losses. Echoing our earlier point, we find that the attainable decrease before plateau escape does not reliably distinguish the checkpoints that go on to escape the plateau (Figure 21e), and that escape is instead associated with rapid feature change (Figure 21f). The results here differ slightly from those presented in Section 5, which were initially obtained with AdamW on a slightly different architecture, whereas Appendix I uses the same core architecture and the ADOPT optimizer as our TinyGSM analysis. We will update the paper with the full results accordingly.

How sensitive are the attainable-decrease ratios to the dimension and construction of the optimization subspace?

We have updated Appendix F.4 to investigate the sensitivity of the measured ratios to the construction of the path span, in particular the dimension of the span and the number of updates taken to form the path from which it is built. As seen in Figures 18b and 18c, the measured ratios are largely insensitive to these changes. Figure 18c also shows that they agree with the ratios of the tangent model trained in all parameters, which involves neither the quadratic approximation nor a subspace approximation.

We hope that we have addressed all of your concerns, please let us know if there are any further questions.

We thank the authors for their responses; I will keep our score.

Reviewer_eiuwOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

It is well-established the training diffusion LMs is much less sample-efficient than training autoregressive LMs. However, the exact reason for this has not been known. This paper looks closer at this problem and investigates it with some cleverly designed experiments.

Critical strengths
  • I believe this is an unusually strong paper, it poses a very important research question, and then answers it with a series of carefully designed experiments. Even the ideas behind some of these experiments seemed very clever to me. But more importantly, the experiments provide intriguing insights into the differences between autoregressive and diffusion LMs.
Critical weaknesses
  • I don't think that training on TinyGSM is appropriate for testing whether different corruptions demand different weight updates. As explained in the paper itself, TinyGSM is chosen because the prompt fully determines all items in the asnwer -- so there shouldn't be any conflicts by design. This might be very different from language modeling on regular text.
Questions for authors

How difficult would it be to verify some of the claims in the paper on more realistic language modeling, even if very approximately?

2 comments

We sincerely thank the reviewer for their positive review of our work!

I don't think that training on TinyGSM is appropriate for testing whether different corruptions demand different weight updates. As explained in the paper itself, TinyGSM is chosen because the prompt fully determines all items in the answer -- so there shouldn't be any conflicts by design. This might be very different from language modeling on regular text. How difficult would it be to verify some of the claims in the paper on more realistic language modeling, even if very approximately?

We agree that this is an important test, and we thank the reviewer for raising it. As noted in the Section 2 background, we chose TinyGSM for the clarity of results afforded by fully specified answers, where the irreducible loss is essentially zero. Following your suggestion, we have added Appendix F.3 to replicate our TinyGSM analysis on OpenWebText, a standard natural language dataset for DLM training and loss evaluation.

  • At the checkpoints we analyzed, the noise-free efficiency gap is smaller than on TinyGSM but still present (Figures 14d and 18).
  • As on TinyGSM, most of AR's extra attainable decrease lies at low curvature (Figure 15), and within this subspace, different corruptions still demand essentially the same weight updates (Figures 16a and 16b).
  • Our concern about irreducible loss and motivation for analyzing TinyGSM becomes evident in Figures 16c and 16d: at high corruption rates, the loss converges to the unigram entropy, which is the Bayes optimal loss at full corruption, and expectedly, the decrease allowed by the features falls almost to zero. However, this leaves us unable to disambiguate the loss that is inherently irreducible, from the loss that is irreducible only under the learned features and which could be reduced provided better features were learned.

Please let us know if there are any further questions we can answer or concerns we can address.

Thank you very much for doing this additional experiment! It's nice to see that it makes your case even stronger. Good luck with the reviewing process :)

Reviewer_oFxxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study the DLM-ARM training efficiency gap. They propose a task, surface Sudoku, that provides an additional training signal to identify the provenance of the performance gap. The results suggest that DLM's learned features allow for much less loss reduction than the ARM's, instead of limited model expressivity, and that auxiliary objectives, or adaptation from a trained ARM can help close the gap.

Critical strengths
  • The paper studies an important question on the ARM-DLM gap.
  • Clean experimental setting and contribution of Surface Sudoku.
Critical weaknesses
  • On the presentation, the key contributions should be simplified and made clearer.
  • The experiments are only done at one small scale. The paper attributes the gap to feature learning rather than expressivity. On Section 5, expressivity is resolved on Surface Sudoku by using the auxiliary loss on a different setup, but on TinyGSM nothing separates the two. How much does model scale interact with these findings, is there any observation that moves with model scale? Repeating the Section 3 measurements (Δ_AR/Δ_DLM at matched loss) at a smaller and/or larger model would show whether the ratio is stable, or whether it shrinks with size, helping to quantify how much of the gap is feature learning or expressivity.
  • Figure 3a and b are relative to one checkpoint, whose features are fixed. Can you show the Figure 3a decomposition at several points along training, and plot Δ and the feature-change component against the training update?
  • Moreover, the authors acknowledge that optimization could have a major role in the gap, and yet Sections 3-4 use ADOPT and Section 5 uses AdamW. Could the authors report an AdamW check on TinyGSM? What is the reason behind this choice?
Questions for authors

Above

2 comments

We thank the reviewer for their time and constructive feedback, and we are glad they appreciated our experimental design!

On Section 5, expressivity is resolved on Surface Sudoku by using the auxiliary loss on a different setup, but on TinyGSM nothing separates the two. How much does model scale interact with these findings, is there any observation that moves with model scale? [...]

We have added a new Appendix F, which replicates our TinyGSM analysis across model scale (F.1) and training (F.2). Appendix F.1 analyzes a model nearly five times larger, at two checkpoints from a much longer training run.

  • At larger scale, the efficiency gap between DLMs and ARMs remains after removing gradient noise costs. Using a four times larger batch yields a ratio of ARM to DLM loss reduction for the network similar to the one we report in the main text, and we observe similar ratios for the noise-free quadratic decreases (Figures 9 and 10). Appendix F.2 compares the two model sizes at similar losses, and the larger model's gap in attainable decrease is about the same as the smaller model's.

  • Noise comprises a larger fraction of optimization gains. With the larger model's checkpoints, the noise cost at the standard batch size exceeds diffusion's noise-free decrease.Relative to the noise-free decrease, the noise cost is even greater later in training, at a lower loss, for all objectives (Figure 13c). This aligns with general views on batch size and learning rate scheduling [1], whereas for simplicity, our training adopted a warmup constant schedule. The effect noise has on optimization efficiency of DLMs compared to ARMs is still largely mediated by diffusion having a much smaller noise-free decrease rather than DLMs simply having more noise, which we expand on in Appendix G, particularly Eq. 22 and Figure 18.

[1] Meterez et al. “Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling” arXiv: 2510.14717

Figure 3a and b are relative to one checkpoint, whose features are fixed. Can you show the Figure 3a decomposition at several points along training, and plot Δ and the feature-change component against the training update?

We have included this plot and its analysis in the newly added Appendix F.2 (Figure 13), which repeats the decomposition at checkpoints throughout training.

  • Attainable decrease (Figure 13a): We plot the amount of attainable decrease under the fixed feature quadratic approximation. We see that at matched loss checkpoints, ARM features consistently allow the same or more attainable decrease than DLM features.

  • Feature change (Figure 13b): We plot the difference between loss improvements under real network training dynamics (where features can change), and loss improvements under the fixed feature tangent model undergoing the same number of updates. We find that real network dynamics for ARMs, which involve feature change, experience much greater loss reduction beyond what is achieved by the fixed features model. In contrast, DLMs achieve very similar loss reduction regardless of whether features are held fixed or not.

[...] Sections 3-4 use ADOPT and Section 5 uses AdamW. Could the authors report an AdamW check on TinyGSM? What is the reason behind this choice?

We have now added Appendix H, which compares ADOPT and AdamW.

-Why ADOPT. AdamW includes the current batch's gradient statistics in its preconditioner, which makes the weight update a non-linear function of the batch gradient. ADOPT updates the preconditioner with the batch's gradient statistics only after the weight update, so the update is a linear function of the batch gradient. While our quadratic analysis freezes the preconditioner, at which point the weight updates of AdamW and ADOPT become identical, we chose ADOPT (with a large β₂ = .999) to ensure that this frozen-preconditioner model is an accurate approximation to the real network's training dynamics.

-AdamW check on TinyGSM. Naturally, ADOPT with a large β₂ might poorly represent realistic network training if it underperforms AdamW with the common choice of β₂ = .95. To dispel such concerns, Figure 19 compares the held-out loss of ADOPT and AdamW for all three objectives, training the larger model of Appendix F.1 on TinyGSM. ADOPT tends to perform worse than AdamW, but not significantly so, especially later in training.

We hope we have addressed your concerns regarding the soundness, replicability and generality of our findings. We have also made an effort to improve the presentation and clarity of writing in the paper including the key contributions. Please let us know if there are any remaining concerns we can address.

Dear Reviewer,

The discussion deadline is approaching. Please review the authors’ responses and clearly indicate whether they adequately address your concerns. Kindly update your score if appropriate.

Best regards,

AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author-Reviewer Discussion

We thank all reviewers for their constructive feedback which has aided us in substantially strengthening the paper. Besides revising the main text to improve clarity, we have added four new appendix sections (F–I) establishing the generality of our findings and addressing reviewer concerns.

We are glad that reviewers found our research question important (oFxx, eiuw) and expressed a lot of appreciation for our experimental design and analysis (oFxx, eiuw, 23xu). We are also very grateful for Reviewer eiuw's strong assessment and support of our work.

Nevertheless, reviewers initially expressed some concerns, requesting:

  • Generality of findings across model scales and training checkpoints (oFxx)
  • Validation on realistic natural language beyond TinyGSM (eiuw)
  • Using ADOPT vs. AdamW across sections without an AdamW check on TinyGSM (oFxx)
  • How fixed feature and quadratic metrics behave for Surface Sudoku experiments and interventions
  • How sensitive reported ratios are to path-span construction (23xu)
  • Presentation clarity (oFxx, 23xu)

We addressed all of these concerns by:

  • Replicating our TinyGSM analysis at a ~5x larger model scale (Appendix F.1), across training checkpoints (Appendix F.1 and F.2), and across datasets by confirming that our core findings hold on OpenWebText (Appendix F.3) as well.
  • Analyzing attainable decrease and feature change for Surface Sudoku interventions (Appendix I) and verifying that our ratios are robust to path span basis dimensionality and construction (i.e. how many quadratic model update steps are used for its construction) (Appendix F.4)
  • Explaining our theoretical motivations for using ADOPT optimizer instead of the more commonly used AdamW, and adding empirical confirmation that ADOPT still matches AdamW performance (Appendix H)
  • Clarifying the framing of "feature learning deficiency" relative to kernel-task alignment and weight initialization (Section 5.2), and revising the main text for clarity

All three reviewers recommended acceptance in their initial reviews (3, 4, 3). After posting our responses, Reviewer eiuw (4) followed up to confirm that the new OpenWebText experiment made our case even stronger. Reviewer 23xu (3) appreciated our response, raising no follow-up questions and maintained their score. While Reviewer oFxx (3) did not respond following our response, we believe our new experiments and revisions thoroughly address all of the concerns and weaknesses they raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.