- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies white-box monitoring probes and their generalization from fine-tuned model variants to the original model. This addresses the problem of monitoring for rare misaligned behaviors, as those are typically hard to elicit from a "normal" model checkpoint, and probes are usually trained on off-policy data. The paper studies this through a variety of "model organisms" derived from the same base model, tuned for various behaviors. Under a very limited model selection, the authors show that probes transfer well when trained on model organisms or on their responses, even when evaluated under the base model prefill.
Critical strengths
-- This is a very solid (and simple) idea -- constructing probes on "less off-policy" generations and model organisms. If it indeed works and scales to larger and more recent models, this is a very promising approach for white-box monitoring of rare behaviors.
-- The methodology, data splits, and model organism selection are clearly described in a reproducible manner.
-- The main strength of the paper is the variety of evaluated misaligned behaviors.
Critical weaknesses
-- Only one base model (Llama-3.1-8B) is used to derive the model organisms. While I understand that this is a straightforward critique to make, and there is not much reason to expect that the same procedure would fail to generalize to a similar model like Qwen2.5-7B, I think it is crucial to expand the model suite along two dimensions: 1) Smaller and larger model sizes, e.g., [1] showed that the concept of deception is significantly harder to monitor in smaller models; and 2) Reasoning models. Including reasoning models would make this work more relevant and address concerns regarding the applicability of the procedure to reasoning traces (e.g., will modified-policy organisms drift too far in their reasoning to generalize to the base model? Does having reasoning drive "visible" output representations too far apart between the base model and the model organism?).
-- The judge models (Llama-3.1-70B and GPT-4.1) appear to be very outdated, and I would appreciate including verification with a newer model, e.g., GPT-6 Luna.
-- While not a critical weakness, I would appreciate it if the authors improved the figures and tables for readability (e.g., avoiding all-caps and excessive statistical significance markers like **). I would also appreciate a better structure in the discussion section, with clear and actionable takeaways on which data generation pipeline should be used for rare behavior monitoring.
[1] Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs (https://proceedings.iclr.cc/paper_files/paper/2026/hash/51510f7458011efb04791417c67798f1-Abstract-Conference.html)
Questions for authors
I think this work investigates a very promising direction, with the main limitation being the meager selection of models. Please see the weaknesses section; adding reasoning models and a model-size ablation to your evaluation suite would significantly strengthen the paper.
3 comments
Thank for your rebuttal and addressing my concerns!
I maintain my positive score and recommend this paper for acceptance.
We are grateful for your feedback! We believe that the analyses you proposed substantially strengthened the paper. Please let us know if additional concerns remain that prevent you from rating our work as a 4 - we would love to hear how we might further improve the work.
We thank the reviewer for their constructive feedback and positive assessment of our approach. We have added two additional models (including a reasoning variant), verified the accuracy of our judges against GPT-6 Luna, and improved the paper's readability. See the details below.
We thank the reviewer for pointing this out. As suggested, we extended our full pipeline to two additional, larger models: OLMo-3.1-32B-Instruct and Gemma-4-31B-it, the latter of which is a reasoning model that we evaluated with thinking both enabled and disabled. For each model, we trained the same set of variants (Risky Financial, Extreme Sports, Bad Medical, Harm Prone, and a new Combined variant trained on all three datasets), generated new evaluation sets, and trained and evaluated all probe types (Section 3, Figure 2).
For Gemma with thinking disabled, variant-trained probes significantly outperform the off-policy probe. With thinking enabled, variant-trained probes remain effective and significantly outperform the off-policy probe when evaluated specifically on naturally generated responses from the base model. In this setting, the off-policy probe performs no better than chance (Figure 5).
For OLMo, we also successfully trained misaligned model variants that could be used for probe training. We found that the base model was particularly resistant to generating harmful responses, even when refusal-ablated or subjected to prefill attacks. Our model variants successfully elicited enough harmful responses to train effective probes. One caveat is that, because the base model did not produce sufficient harmful responses for evaluation, we could only test these probes on harmful responses generated by model variants.
Results from both OLMo and Gemma show that variant-trained probes are less sensitive to the choice of training contrast than off-policy probes (see Section 3, Figure 2). This suggests an additional advantage of our approach: when the optimal training contrast is unknown, variant-trained probes may offer more consistent performance across different contrast choices.
Following the reviewer's suggestion, we validated our judge against GPT-6 Luna on 4,000 responses (Appendix E3). We find that agreement is high. For harmfulness, the judges agree on 99.5% of labels (κ=0.98, r=0.92), and for coherence on 87% (κ=0.45, r=0.58), comparable to our earlier comparison with GPT-4.1 (98%, κ=0.94 and 92%, κ=0.46, respectively). We retain Llama-3.1-70B as our primary judge for reproducibility.
We thank the reviewer for these suggestions. We have revised our figures and tables to improve readability and restructured the Discussion to provide clearer practical recommendations. Specifically, we removed small caps from variant names, simplified significance reporting to a single marker for AUC, and moved detailed tables containing threshold-based metrics to the appendix.
We also reorganized the Discussion into subsections with clearly stated takeaways and added three actionable recommendations:
Prefer same-prompt contrasts. When possible, train probes to distinguish harmful and benign responses to the same harmful prompts.
Use model variants when suitable on-policy data is unavailable. When the preferred contrast cannot be constructed from naturally generated responses, variant-trained probes, particularly those using representation transfer, are preferable to off-policy probes.
Narrow fine-tuning may be better than broad fine-tuning. Emergently misaligned models consistently outperform the Harm Prone probe when using the optimal contrast. It may be the case that narrow fine-tuning yields variants that are less off-policy than broad fine-tuning, particularly for reasoning models.
We thank the reviewer for their recommendations, as we believe these additions have substantially strengthened the paper. We are happy to address any further comments.