- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors come up with a novel method for detecting emergent misalignment while one finetunes LLM. Instead of conventionally making model generate answers to some questions and judge those answers, their method monitor changes in the model's hidden states within some low dimensional space (which captures notions like honesty, power-seeking etc.) and predicts if the model is misligned using a linear prodictor eventually (whose input is the change in hidden states along that small space). They report high (it is very close to 100%) detection accuracy on heldout training benchmarks.
Critical strengths
S1) the approach is novel to my knowledge
S2) Well written paper easy to follow very quickly I got the idea.
Critical weaknesses
The method requires some (here it is 115) forward passes at every evaluation checkpoint, which require to extract activations. It is still computationally expensive (but I accept it is probably much cheaper than evaluating the model on several prompts). In addition to that your method requires to construct a supervised learning dataset which require a lot of LLM evaluation (which is another cost that may be preliminary).
W1) Therefore, to be practical (under these preliminary costs) the method should be generalizable a lot. the heldout datasets on which they test the approach are coming from kind of the same family of emergent misalignment benchmarks in the literature. therefore the current results aren't showing generalizability of the method to fundamentally different datasets or misalignment types. Also see Q1
W2) Another thing is: I dont see why would one need to check model's alignment with such high frequency? (for instance you evaluated at every 10 iterations) (that is the regime your method becomes useful otherwise ordinary LLM judge with generation is a better approach because it may even be cheaper than the initial cost in your approach). Therefore, I am not sure if the problem you are solving a real problem?
Questions for authors
Q0) Please answer to the W2. (your answer to this question is quite important for me). ALso it would be great to pareto frontier curves for see cost vs perfromance (by defining a useful and practically relevant performance) curve of both approaches (the one you have and LLM judge one).
Q1) Could you try to find fundamentally different heldout benchmark on which you test your method so that we see if this is practically something useful?
Q2) Even, some misalignment behavior may be further fundamentally different that it is not captured by the 7 directions you mention? (this is one step further than Q1).
4 comments
Response to Reviewer CQMj (cont)
Q1/W1 - Held out EM types
We appreciate this suggestion and have conducted a new experiment on two qualitatively different distinct EM-inducing datasets that are different from the harmful-advice family in our original evaluation.
- AestheticEM (Woodruff et al., 2025): finetuning on unpopular aesthetic preferences, without harmful advice or insecure code.
- Norm-breaking: a new synthetic dataset constructed from Moral Stories and Social Chemistry 101, where assistants endorse everyday social-norm violations without providing harmful instructions or advice. Neither dataset was included in our original calibration. We use the same finetuning protocol (LoRA rank 16), across four models and four seeds.
Results. Both inducers exhibit drift aligned with our original geometric axis (median PC1 cosine ≈ 0.92), providing evidence of representational generalization. However, our frozen monitor (trained from the calibration data, as seen in the paper), exhibits opposite calibration mismatches: it overpredicts EM on AestheticEM (56.1% FPR) and underpredicts EM on norm-breaking (66.2% FNR). We therefore investigate a low-cost family-specific correction, adding just one labeled trajectory per model to calibration and evaluating on the remaining three held-out seeds:
| Held-out inducer (RF) | FNR: Frozen → Recal. (%) | FPR: Frozen → Recal. (%) | AUROC: Frozen → Recal. |
|---|---|---|---|
| AestheticEM | 0.0 → 11.9 | 56.1 → 11.4 | 0.931 → 0.961 |
| Norm-breaking | 66.2 → 14.1 | 5.9 → 10.6 | 0.872 → 0.952 |
These findings (which will be presented in the new sub section in 5, reinforce that the underlying trait-space geometry generalizes beyond the original EM-inducer families, while the mapping to behavioral severity can require family-specific calibration. Importantly, substantial improvements require only one additional labeled trajectory per model, rather than reconstructing the representation or repeating the full calibration pipeline. We believe these experiments directly address the reviewer's request while providing a more precise characterization of the method's generalization capabilities and limitations.
Q2 - Motivation and further generalization
Recent work has shown that more capable LLMs can strategically alter their behavior in black-box evaluations [1,2,3], motivating greater interest in white-box monitoring. Existing activation-based methods have demonstrated promise for specific concepts (ex: refusal, deception [3,4]). Our work extends this direction by monitoring joint changes across multiple alignment dimensions, with new evidence from distinct EM types above.
We emphasize that we do not claim to detect all forms of misalignment. As this is still an active area of research, such universal coverage would be exceedingly demanding for any monitoring method. We view our method as primarily a contribution towards internal safety monitoring and invite future work to investigate additional representation and misalignment mechanisms.
We will reinforce these points to the related work and limitation sections.
[0] Anders Cairns Woodruff. 2025. Aesthetic preferences can cause emergent misalignment. https://www.lesswrong.com/posts/gT3wtWBAs7PKonbmy/.LessWrong.
[1] Van Der Weij, Teun, et al. "Ai sandbagging: Language models can strategically underperform on evaluations." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Greenblatt, Ryan, et al. "Alignment faking in large language models." arXiv preprint arXiv:2412.14093 2.1 (2024).
[3] Parrack, Avi, Carlo Leonardo Attubato, and Stefan Heimersheim. "Benchmarking deception probes via black-to-white performance boosts." arXiv preprint arXiv:2507.12691 (2025).
[4] Goldowsky-Dill, Nicholas, et al. "Detecting strategic deception with linear probes." Forty-second International Conference on Machine Learning. 2025.
Thanks for the replies. I do not think all of my questions are answered totaly but I increased the score to weak acceptance.
Final response to Reviewer CQMj
Thank you for your follow-up and for reconsidering your assessment. We appreciate your constructive feedback, which helped us strengthen the paper.
We have incorporated our responses to the revision.
Thank you again for your time and thoughtful consideration.
Best, The Authors
Response to reviewer CQMj
We thank the reviewer for praising our paper’s novelty and clarity. We first provide clarification on the cost and benefit tradeoff (Q0/W2). Then, we provide empirical results on two different held out EM datasets (Q1/W1) and provide a summary of practical recipes informed by our experiments.
Q0/W2 - Practical value and monitoring cost
This is an insightful question. We first note that the 10-step interval is deliberately chosen to provide higher granularity in characterizing EM onset; practitioners, however, may select the monitoring cadence according to their needs and computation budget.
We further provide the following cost-benefit analysis on our system to illustrate the practical boundary. On a single NVIDIA A6000, our monitor requires 6.6-11 seconds per checkpoint, compared with approximately 34 seconds for behavior evaluation (18.7 seconds for generation and 15.5 s for GPT-4o-judging). Based on provider’s pricing, behavior evaluation costs roughly 0.09 dollar per checkpoint. Per-model calibration requires approx. 1.9 GPU hours plus $14 in judge cost, amortized after about 14 repeatedly monitor runs under our benchmark setting.
Cost–detection-delay comparison. To evaluate the practical benefit, we compare three monitoring strategies on the same 36 held-out finetuning runs: (1) final-only behavioral evaluation, which generates and grades responses only at the last checkpoint, (2) dense behavioral evaluation, which performs generation and grading at every checkpoint, and (3) our canary, which monitors activations at every checkpoint and triggers behavioral evaluation only when an alarm fires. We measure the evaluation cost and the delay between the first behaviorally dangerous checkpoint and its detection. Training is assumed to stop once danger is behaviorally confirmed.
_Our framework is designed for computing at scale._For short and/or one-off training run, simply final-checkpoint behavioral evaluation would be an appropriate choice, but it detects EM when training is already finished. In contrast, our monitor achieves near-onset detection at approximately the same marginal cost as final-only evaluation. Running behavior evaluation alone at every checkpoint would incur 7.7x the cost. Although our approach requires upfront calibration, this cost is amortized after approximately 14 monitored runs under our benchmark setting. Our monitor’s advantage grows with repeated use, making fine-grained safety monitoring practical at scale. We will expand section 5.1, the appendix to incorporate this analysis and reinforce the use case in the Introduction.