- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+2)
Read review
Paper summary
This paper studies how semantic attributes emerge during denoising in discrete diffusion language models. The authors train sparse autoencoders on MDLM, SEDD, LLaDA, and DREAM, and identify features related to sentiment, topic, and style. They find differences in emergence timing, concentration, and feature-level effect sizes across models and attributes. These observations motivate an adaptive steering schedule that concentrates intervention during feature emergence, with an additional calibration to balance strengths across attributes. The paper provides an idealized allocation analysis and evaluates seven single- and multi-attribute conditions. Comparisons include uniform SAE steering, interval-restricted steering, dense activation methods, and prompting. The reported results generally show better generation quality at matched attribute confidence than uniform and interval-restricted schedules.
Critical strengths
1. Motivation & Usefulness
- I agree that reducing the quality cost of steering is important. The paper gives a clear motivation for considering when to intervene.
- The focus on multi-attribute control is also useful, since interactions matter and the framework provides a simple way to combine the interventions.
2. Related Works & Novelty
- Using SAE feature dynamics to build attribute-specific schedules is interesting. It connects the interpretability analysis to a practical steering method.
- The cross-model analysis is also informative. The different emergence patterns help motivate model- and attribute-specific schedules.
3. Experiments & Robustness
- The experiments are fairly extensive. I appreciate the residual-stream and Step Function baselines.
- The matched-confidence comparisons are also helpful. Adaptive improves quality in most of these comparisons against Uniform and Step Function.
- The schedule-stability and feature-count analyses are also useful for understanding sensitivity and cross-attribute interference.
4. Presentation
The main paper is generally easy to follow. The figures help explain how the feature analysis motivates the proposed schedule.
Critical weaknesses
Overall, I find the motivation useful and the empirical results promising. However, some theoretical arguments and experimental details remain unconvincing. Addressing these concerns would strengthen the paper and could change my assessment.
1. Related Works & Novelty
- The novelty framing is somewhat overstated. The ILRR [1] and Shnaidman et al. [2] study stage-dependent activation steering. The additional contribution is therefore the SAE-derived emergence schedule, rather than temporal intervention itself.
- The exclusion of classifier guidance in Section 5.1 (lines 358–360) is not fully convincing, since discrete classifier-based and classifier-free guidance already exist [3]. The practical advantage over these alternatives remains unclear, given the SAE training and calibration required.
2. Theory & Proofs
Theorem 1 appears valid under its assumptions, but I am less convinced by some supporting arguments.
- In Appendix A.2.5 (lines 959–971), how is Lemma 2’s differentiation through sampled discrete tokens justified? Is g an expected score or a differentiable relaxation, rather than the score of a realized sequence?
- In Appendix A.2.6 (lines 1037–1041), why would an imperfect schedule be harmless when sensitivity is flat? A concentrated schedule can still underperform uniform steering at equal cost.
- The overlap argument in Appendix A.3.4 (lines 1183–1187) also seems insufficient. How is the intervention energy on shared features controlled? Even one shared feature could carry most of both interventions, so a small overlap count alone does not establish small coupling.
3. Experiments & Robustness
- The dense baselines use uniform interventions (Appendix C.10.1, lines 2033–2045). Their comparison with Adaptive therefore mixes direction choice with scheduling. The benefit over equally scheduled dense directions remains unclear.
- In Appendix D.5 (lines 2522–2537), MDLM–SEDD is compared in hidden space, while the other pairs use vocabulary projections. These scores are not directly comparable, and shared architecture does not guarantee aligned coordinates. The conclusion that training objective dominates architecture is therefore not well supported.
- The quality-gate descriptions are inconsistent. Appendix E.3.1 (lines 2731–2734) describes a PPL-only gate for DREAM, unlike Section 5.1. Table 2 reports multi-attribute Adaptive fluency of 1.34 versus 3.75 without steering, below the stated 50%-of-baseline threshold. This makes the quality-preservation claim less convincing.
4. Presentation & Reproducibility
- In Appendix C.9.1 (lines 1926–1932), the SAE reconstruction replaces the hidden state, whereas Section 4 preserves the reconstruction residual. Which implementation was used, and does alpha=0 recover the unsteered model?
References
[1] Avrahami and Nachmani. ILRR: Inference-Time Steering Method for Masked Diffusion Language Models. arXiv, 2026. https://arxiv.org/abs/2601.21647
[2] Shnaidman et al. Activation Steering for Masked Diffusion Language Models. ReALM-GEN Workshop at ICLR, 2026. https://arxiv.org/abs/2512.24143
[3] Schiff et al. Simple Guidance Mechanisms for Discrete Diffusion Models. ICLR, 2025. https://arxiv.org/abs/2412.10193
Questions for authors
-
How is differentiation through sampled tokens justified in Lemma 2? What assumptions support the flat-sensitivity and small-overlap claims in Appendices A.2.6 and A.3.4?
-
Was residual correction or reconstruction replacement used, and does alpha=0 recover the unsteered model? Which quality gates were enforced for DREAM, and do the gains remain under the stated fluency threshold?
-
How do dense steering directions perform with the same temporal schedule and separately tuned strengths? This would help clarify the benefit of SAE-derived directions beyond scheduling.