- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates why diffusion language models (DLMs) require substantially more training compute than autoregressive models (ARMs) to reach comparable held-out loss, despite sharing the same Bayes-optimal objective value. The authors introduce a noisy quadratic approximation based on the local Gauss–Newton geometry to decompose training progress into attainable loss reduction, optimizer progress, and gradient-noise cost. Their analysis indicates that gradient noise accounts for only part of the efficiency gap; the dominant contribution is a smaller attainable decrease under fixed tangent features. They further show that gradients across different corruption rates are strongly aligned, ruling out simple competition among denoising tasks as the primary cause. The deficit instead concentrates at high corruption rates, where much of the loss resides in errors that the current features cannot effectively reduce. Finally, the authors introduce Surface Sudoku, a controlled synthetic task with latent com- positional structure. Auxiliary digit supervision, structured cell masking (with appropriate initialization), and initialization from pretrained models substantially improve diffusion training, in some cases matching ARM performance. The authors interpret these results as evidence that inefficient feature learning is a central bottleneck.
Critical strengths
--The noisy quadratic framework provides a clean decomposition of the training-efficiency gap.
--The result that corruption gradients remain strongly aligned across rates, particularly in the curvature band containing the attainable-decrease gap, challenges the hypothesis that diffusion training is inefficient because different denoising tasks compete.
--Surface Sudoku is a well-motivated synthetic benchmark that makes the missing representation explicit.
Critical weaknesses
--Can the authors elaborate more on “feature learning deficiency”?
Conceptual Framing: Feature Learning vs. Kernel–Task Alignment My main concern is the interpretation of the results as demonstrating a general “feature learning deficiency.” The central diagnostic measures differences in local tangent geometry and the attainable decrease supported by the checkpoint’s features. This establishes that the DLM has less favorable local gradient–curvature geometry than the ARM at matched loss, but does not by itself identify the mechanism responsible for producing these features. An alternative and potentially more specific explanation is that standard diffusion corruption induces a training geometry poorly aligned with the latent structure of the data. Surface Sudoku provides suggestive evidence consistent with this interpretation. Under character- level masking, the DLM plateaus substantially above the ARM, whereas masking complete three-character cells, together with an appropriate initialization, allows diffusion to match ARM performance. This suggests that training efficiency depends strongly on whether the corruption process respects the compositional units of the task. Auxiliary supervision and ARM initialization can also be interpreted as interventions that guide optimization toward more favorable tangent-kernel geometries. However, the paper does not directly measure how these geometries evolve under the different interventions. In this sense, deficient tangent features may describe the observed optimization bottleneck, while corruption–task mismatch may help explain why such features emerge in the first place. These interpretations are not mutually exclusive, but they operate at different explanatory levels. I encourage the authors to distinguish more clearly between the observed limitations of the local tangent geometry and the broader claim that DLMs generally fail to learn useful representations. Investigating corruption-induced kernel–task mismatch as a potential underlying mechanism would strengthen the paper’s conceptual contribution.
--Auxiliary supervision and ARM initialization can also be interpreted as interventions for optimization. Can the authors elaborate on how these geometries evolve under the different interventions?
-(Kernel–task) Could the observed efficiency gap be understood as a mismatch between corruption-induced tangent-kernel geometry and task structure?
Questions for authors
--How sensitive are the attainable-decrease ratios to the dimension and construction of the optimization subspace?
3 comments
We have included additional experimental results in the revised submission.
Auxiliary supervision and ARM initialization can also be interpreted as interventions for optimization. Can the authors elaborate on how these geometries evolve under the different interventions?
We have added Appendix I, which investigates how interventions such as adding a latent digit prediction auxiliary loss affect attainable decrease and feature change of character-masked models (Figure 21a-c) as well as cell-masked and AR models (Figure 21d-f). We find that attainable decrease and feature change are associated with loss decrease and plateau escape: training runs that are stuck on the plateau have a smaller feature change and attainable decrease than runs that reach lower losses. Echoing our earlier point, we find that the attainable decrease before plateau escape does not reliably distinguish the checkpoints that go on to escape the plateau (Figure 21e), and that escape is instead associated with rapid feature change (Figure 21f). The results here differ slightly from those presented in Section 5, which were initially obtained with AdamW on a slightly different architecture, whereas Appendix I uses the same core architecture and the ADOPT optimizer as our TinyGSM analysis. We will update the paper with the full results accordingly.
How sensitive are the attainable-decrease ratios to the dimension and construction of the optimization subspace?
We have updated Appendix F.4 to investigate the sensitivity of the measured ratios to the construction of the path span, in particular the dimension of the span and the number of updates taken to form the path from which it is built. As seen in Figures 18b and 18c, the measured ratios are largely insensitive to these changes. Figure 18c also shows that they agree with the ratios of the tangent model trained in all parameters, which involves neither the quadratic approximation nor a subspace approximation.
We hope that we have addressed all of your concerns, please let us know if there are any further questions.
We thank the authors for their responses; I will keep our score.
We thank the reviewer for their thorough and thoughtful review, and we are glad they appreciated our experimental analysis and findings.
The intent of our work was not to identify the mechanism behind good vs. bad feature learning, but to trace the gap in optimization and compute inefficiency of DLM training relative to ARM training. We happened to find that feature learning was the determining factor. For greater clarity, we have expanded the explanation for what is meant by "feature learning deficiency" in the Plateau paragraph in Section 5.2, and discuss how our results support it as an alternative explanation to the hypothesis from prior work that hard prediction problems underlie DLM training inefficiency.
Essentially, we intend "feature learning deficiency" as a generic term for a failure to learn features which enable the model to reach a low loss, in contrast to the hard prediction problems hypothesis which would suggest that DLMs struggle to achieve the same loss as ARMs due to limitations in the expressivity of the model architecture/size. As noted in our conclusions, one promising avenue toward a theoretical, mechanistic understanding of feature learning deficiencies are the training dynamics of deep linear networks, which can model network abilities to escape saddle points and thereby escape loss plateaus; as saddle point escape is fundamentally non-convex optimization, it by definition implies that the features, the model Jacobians, must change. This analysis lies outside the scope of our work, whose focus is to properly assess and challenge the common explanations for the compute inefficiency of DLM training.
We agree with the reviewer that alignment between the corruption and latent structure of the data could be a key mechanism behind the effectiveness of cell-masked training. However, we would like to emphasize that the role of "appropriate initialization" should not be understated, and that the corruption process alone does not determine whether good features are learned. Our Surface Sudoku results in Figure 6 show that changing the corruption on its own is not enough: whether the DLM trained with cell masking (which is presumably better aligned with the latent structure of the data) succeeds depends entirely on its weight initialization. This suggests a much more involved mechanistic understanding for when training does and does not succeed is necessary, and would be valuable future work.
We agree with the reviewer that mechanisms explaining why good features are not learned, such as a kernel–task mismatch, are not mutually exclusive with our very general claim of a feature learning deficiency. However, as mentioned earlier, the importance of weight initialization implies that analyzing a corruption process's tangent-kernel alignment with the loss at initialization would not reliably distinguish good corruption processes from bad ones. Nevertheless, we do believe that designing corruption processes aligned with the latent structure of the data could be an effective way to induce good feature learning and close the efficiency gap between ARMs and DLMs, and our Surface Sudoku findings are intended to encourage such efforts.
We thank the reviewer again for their feedback, which has helped improve the clarity of our work. Please let us know if you have any further questions!
We are still working on new experiments on the sensitivity of our results to basis construction, and are conducting Section 3 style analysis on the Surface Sudoku experiments and interventions. We aim to update our submission with these results before the end of the discussion period.