acceptodds
Under review as a conference paper at ICLR 2027

Learn and Steer: Mechanistically Informed Interventions for Discrete Diffusion Language Models

Abstract

Discrete diffusion language models (DLMs) generate text by iteratively denoising all positions in parallel, offering an alternative to autoregressive models. Controlled generation methods for DLMs, imported from autoregressive models, apply uniform intervention at every denoising step. We show this uniform schedule is inefficient and degrades quality, and the damage compounds when multiple attributes are steered jointly. To diagnose the failure, we train sparse autoencoders on four DLMs (124M-8B parameters) and find that different attributes commit on distinct schedules, varying in timing, sharpness, and magnitude. For instance, topic commits within the first 2% of denoising on MDLM, whereas sentiment emerges gradually over 20% of the process. Motivated by these profiles, we propose an adaptive scheduling mechanism that concentrates intervention where each attribute is actively forming. An idealized allocation analysis predicts that attributes with more sharply concentrated emergence benefit more from adaptive scheduling, a prediction we confirm empirically. Across seven single- and multi-attribute steering tasks on four DLMs, adaptive steering consistently improves the control-quality balance over uniform and interval-restricted baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, steering, interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VAGaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper studies how semantic attributes emerge during denoising in discrete diffusion language models. The authors train sparse autoencoders on MDLM, SEDD, LLaDA, and DREAM, and identify features related to sentiment, topic, and style. They find differences in emergence timing, concentration, and feature-level effect sizes across models and attributes. These observations motivate an adaptive steering schedule that concentrates intervention during feature emergence, with an additional calibration to balance strengths across attributes. The paper provides an idealized allocation analysis and evaluates seven single- and multi-attribute conditions. Comparisons include uniform SAE steering, interval-restricted steering, dense activation methods, and prompting. The reported results generally show better generation quality at matched attribute confidence than uniform and interval-restricted schedules.

Critical strengths

1. Motivation & Usefulness

  • I agree that reducing the quality cost of steering is important. The paper gives a clear motivation for considering when to intervene.
  • The focus on multi-attribute control is also useful, since interactions matter and the framework provides a simple way to combine the interventions.

2. Related Works & Novelty

  • Using SAE feature dynamics to build attribute-specific schedules is interesting. It connects the interpretability analysis to a practical steering method.
  • The cross-model analysis is also informative. The different emergence patterns help motivate model- and attribute-specific schedules.

3. Experiments & Robustness

  • The experiments are fairly extensive. I appreciate the residual-stream and Step Function baselines.
  • The matched-confidence comparisons are also helpful. Adaptive improves quality in most of these comparisons against Uniform and Step Function.
  • The schedule-stability and feature-count analyses are also useful for understanding sensitivity and cross-attribute interference.

4. Presentation

The main paper is generally easy to follow. The figures help explain how the feature analysis motivates the proposed schedule.

Critical weaknesses

Overall, I find the motivation useful and the empirical results promising. However, some theoretical arguments and experimental details remain unconvincing. Addressing these concerns would strengthen the paper and could change my assessment.

1. Related Works & Novelty

  1. The novelty framing is somewhat overstated. The ILRR [1] and Shnaidman et al. [2] study stage-dependent activation steering. The additional contribution is therefore the SAE-derived emergence schedule, rather than temporal intervention itself.
  2. The exclusion of classifier guidance in Section 5.1 (lines 358–360) is not fully convincing, since discrete classifier-based and classifier-free guidance already exist [3]. The practical advantage over these alternatives remains unclear, given the SAE training and calibration required.

2. Theory & Proofs

Theorem 1 appears valid under its assumptions, but I am less convinced by some supporting arguments.

  1. In Appendix A.2.5 (lines 959–971), how is Lemma 2’s differentiation through sampled discrete tokens justified? Is g an expected score or a differentiable relaxation, rather than the score of a realized sequence?
  2. In Appendix A.2.6 (lines 1037–1041), why would an imperfect schedule be harmless when sensitivity is flat? A concentrated schedule can still underperform uniform steering at equal cost.
  3. The overlap argument in Appendix A.3.4 (lines 1183–1187) also seems insufficient. How is the intervention energy on shared features controlled? Even one shared feature could carry most of both interventions, so a small overlap count alone does not establish small coupling.

3. Experiments & Robustness

  1. The dense baselines use uniform interventions (Appendix C.10.1, lines 2033–2045). Their comparison with Adaptive therefore mixes direction choice with scheduling. The benefit over equally scheduled dense directions remains unclear.
  2. In Appendix D.5 (lines 2522–2537), MDLM–SEDD is compared in hidden space, while the other pairs use vocabulary projections. These scores are not directly comparable, and shared architecture does not guarantee aligned coordinates. The conclusion that training objective dominates architecture is therefore not well supported.
  3. The quality-gate descriptions are inconsistent. Appendix E.3.1 (lines 2731–2734) describes a PPL-only gate for DREAM, unlike Section 5.1. Table 2 reports multi-attribute Adaptive fluency of 1.34 versus 3.75 without steering, below the stated 50%-of-baseline threshold. This makes the quality-preservation claim less convincing.

4. Presentation & Reproducibility

  1. In Appendix C.9.1 (lines 1926–1932), the SAE reconstruction replaces the hidden state, whereas Section 4 preserves the reconstruction residual. Which implementation was used, and does alpha=0 recover the unsteered model?

References

[1] Avrahami and Nachmani. ILRR: Inference-Time Steering Method for Masked Diffusion Language Models. arXiv, 2026. https://arxiv.org/abs/2601.21647

[2] Shnaidman et al. Activation Steering for Masked Diffusion Language Models. ReALM-GEN Workshop at ICLR, 2026. https://arxiv.org/abs/2512.24143

[3] Schiff et al. Simple Guidance Mechanisms for Discrete Diffusion Models. ICLR, 2025. https://arxiv.org/abs/2412.10193

Questions for authors
  • How is differentiation through sampled tokens justified in Lemma 2? What assumptions support the flat-sensitivity and small-overlap claims in Appendices A.2.6 and A.3.4?

  • Was residual correction or reconstruction replacement used, and does alpha=0 recover the unsteered model? Which quality gates were enforced for DREAM, and do the gains remain under the stated fluency threshold?

  • How do dense steering directions perform with the same temporal schedule and separately tuned strengths? This would help clarify the benefit of SAE-derived directions beyond scheduling.

Reviewer_XhavOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the interpretability of diffusion language models (DLLMs) and utlizes the insights for attribute steering. To understand at which diffusion steps different semantic features emerge, the authors train Sparse Autoencoders on the hidden representation layers of four DLLMs, and find that different attributes commit at different denoising steps, with varying sharpness with per-feature discriminative strengths across models. From these observations, the authors propose an adaptive SAE-based steering approach for DLLM. Evaluations on single and multi-attribute steering demonstrate their proposed adaptive steering achieves a better trade-off between attribute control and generation quality compared to uniform steering and other baselines.

Critical strengths

The paper proposes an adaptive temporal-aware steering motivated from an interpretability perspective. The design of steering intervention, which takes into account for the per-attribute steering strengths and which denoising steps, is intuitive and well motivated by the SAE findings on commitment timing, sharpness, and effect-size asymmetry. The authors formalized the intervention as a budgeted allocation problem and provided a mathematical explanation of why adaptive temporal steering can be more efficient than uniform steering. Detailed empirical evaluations done on four different models and ablation studies strengthened the paper and justified its design choices.

Critical weaknesses

The paper evaluates uniform, step-funcion, and adaptive schedules within SAE framework against residual-stream baselines under only uniform schedule. As Contrastive Vectors already performs compatitive with uniform steering on models like SEDD and DREAM, it is unclear how much quality-control gain comes from SAE feature decomposition versus the temporal scheduling. Also, despite Theorem 1 provided a clean formulation that optimal temporal steering should follow αt∗∝st/ct\alpha_t^* \propto s_t / c_t, the actual schedule wdyn(t)w_{dyn}(t) relies entirely on positive SAE activation growth and assumes ctc_t varies slowly relative to sts_t. These assumptions are not directly validated, as acknowledged by the authors.

Minor notational issue: hh is used to represent both D-LLM hidden representations and SAE-encoded feature activations.

Questions for authors
  • The paper used Cohen's d for feature-selection while used raw mean activation difference for actual steering. What motivated this choice?
  • How do you decide the number of SAE use for steering?
  • How much of the improvement comes from SAE feature decomposition versus temporal scheduling?
Reviewer_3fSfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains sparse autoencoders (SAEs) on the residual streams of four discrete diffusion language models (MDLM, SEDD, LLaDA-8B, Dream-7B), extracts contrastive features for sentiment, topic and formality, and tracks when they emerge along the denoising trajectory. It then proposes Adaptive steering: the strength at each step is scaled by the normalized block-wise activation growth of the target features, with a per-attribute scalar for multi-attribute control. A budgeted-allocation argument (Theorem 1) motivates non-uniform schedules. Seven attribute conditions compare Adaptive with uniform and interval SAE steering, contrastive-vector, probe and PCA steering, and prompting, measured by classifier confidence, Qwen2 perplexity (PPL) and LLM-judge fluency.

Critical strengths

The evaluation is broad: four models across two objectives, seven conditions, 200 samples for each of 20 values of α, an interval baseline following Kynkäänniemi et al. (2024), matched-confidence and matched-quality analyses, and full per-condition tables. The commitment-timing characterization (Fig. 2, App. D), especially the MDLM-vs-SEDD contrast that isolates the training objective, is a useful observation about DLMs.

Critical weaknesses

My main concern is that the baselines and positioning do not isolate the claimed contribution. The SAE decoder is affine, so single-attribute SAE steering is a fixed residual-stream vector addition (x' − x = W_F αδ, as the paper notes) and the schedule is direction-agnostic, yet to my reading it is never applied to the contrastive-vector baseline, which on DREAM already beats Adaptive on quality at somewhat lower confidence (multi-attribute PPL 11.7 and fluency 2.23 vs 64.2 and 1.34, Table 2). One cannot tell whether the SAE or the schedule does the work. The positioning is also overstated: the claim that classifier guidance is "not directly applicable to masked discrete diffusion" refers to Dhariwal & Nichol, but guidance for discrete diffusion exists [1, 2] and is the natural baseline, and "all existing methods apply the same intervention at every denoising step" should be discussed against [3], which reports step-localized leverage of activation steering on LLaDA. Theorem 1 does not change this: it is Cauchy–Schwarz under an assumed linear-gain, quadratic-cost model whose quantities are not measured, and Appendix A.2.6 concedes the link to the schedule is "intuitive rather than rigorous".

Second, the evidence for the schedule itself is weaker than the headline. At matched quality, Adaptive reaches higher confidence in only 46.7% of comparisons against Uniform and 47.5% against Step (Table 3); the 97.5% at matched α is close to automatic, since w_dyn ≤ 1 injects strictly less perturbation, and the equal-budget comparison that Theorem 1 calls for is never run. Section 5.1 gates operating points on PPL < 100 and fluency ≥ 50% of baseline, but also says PPL is the primary gate except for LLaDA; the reported DREAM multi-attribute points have Adaptive fluency 1.34 of 5 (baseline 3.75) at PPL 64 (Table 2), down to 1.07 per condition (Table 38), which pass a PPL-only gate and fail the stated one. The η values validating Theorem 1 (Table 7) are hard to reconcile with §3.2: 71% of MDLM topic emergence in one of eight blocks implies η ≥ 4, yet Table 7 gives 2.58, and the LLaDA correlation holds only after the quality metric switches from PPL to fluency, as the paper discloses. All results use a single seed. I work on diffusion models rather than interpretability, so I am less confident about the SAE methodology than about the steering experiments.

[1] Nisonoff et al. Unlocking Guidance for Discrete State-Space Diffusion and Flow Models. ICLR 2025. [2] Schiff et al. Simple Guidance Mechanisms for Discrete Diffusion Models. ICLR 2025. [3] Shnaidman et al. Activation Steering for Masked Diffusion Language Models. arXiv 2512.24143, 2025 (ReALM-GEN workshop @ ICLR 2026).

Questions for authors
  1. Please apply w_dyn and the step window to the contrastive-vector baseline at the same layers. Does the SAE add anything beyond the schedule?
  2. Please report Uniform with Σ_t α matched to Adaptive's Σ_t α w_dyn(t) at the operating points of Tables 37–38, as Theorem 1 prescribes.
  3. Please re-select operating points under the stated gate and report how Table 2 and the "22 of 28" count change. How are the η of Table 7 computed, and at which layer?
  4. Why are discrete guidance methods [1, 2] not baselines, and how stable are the features and Table 2 across SAE-training and generation seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.