- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies monitoring of discrete diffusion language models using models trained to hide secrets, underperform and hold unstated beliefs. The main focus is on how whole-answer drafts provided by discrete diffusion models at each decoding step might facilitate monitoring. The authors show that while the drafts facilitate early detection, they primarily track intended words and not withheld knowledge, since targeted training to penalize key words weaken detection without fully removing informative hints or changing model behavior. Further hidden state measurements and causal interventions show that the underlying concepts still affect model behavior even when related words are suppressed. This motivates monitoring internal representations rather than token probabilities. I found the text very hard to follow, so I apologize if I have misrepresented the paper in this summary somehow.
Critical strengths
- Monitoring of discrete diffusion models is a relevant and timely topic that has not received much attention in the literature.
- The reported findings that the whole-answer drafts provided by discrete diffusion models enable earlier detection but do not seem to provide more faithful or robust monitoring is significant and could motivate further research.
Critical weaknesses
- The paper’s presentation could be substantially clearer. Several paragraphs combine motivation, experimental details, results, and discussion, making it difficult to follow the progression of the argument. I encourage the authors to separate these elements and establish a clearer narrative, beginning with the motivation and experimental setup before presenting the results and their interpretation. The abstract suffers from a similar problem; it is dense and full of numbers out of context, which prevents the main message from shining through.
- I am not fully convinced about the novelty of the paper.
- The paper seems to apply the same experimental setup proposed in [1] to discrete diffusion language models. The methodological novelty seems limited to the reading of the diffusion-specific draft readout.
- The main finding that internal representations are better monitoring targets than token probabilities is not particularly surprising. Prior work has already established that activation-based readouts can recover information not reflected on the model’s output [2, 3].
References
[1] Cywiński, Bartosz, et al. "Towards eliciting latent knowledge from LLMs with mechanistic interpretability." arXiv e-prints (2025): arXiv-2505.
[2] Burns, Collin, et al. "Discovering latent knowledge in language models without supervision." arXiv preprint arXiv:2212.03827 (2022).
[3] Azaria, Amos, and Tom Mitchell. "The internal state of an LLM knows when it’s lying." Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
Questions for authors
- The experimental setup was not entirely clear from the paper. Could the authors explain it in detail, including the needed innovations or adaptation required for the discrete diffusion setting?
- How does this work inform the design of future monitors besides encouraging the usage of the model's latent representations?