acceptodds
Under review as a conference paper at ICLR 2027

Misaligned model variants serve as an effective source of training for harm detection probes

Abstract

The ability to detect misaligned outputs from large language models (LLMs) is critical for their safe deployment. Linear activation probes have emerged as an effective solution, particularly when incorporated in a multilayered monitoring pipeline (Cunningham et al., 2026). They are fast, cheap, and easy to train and deploy. However, these probes can also be brittle, particularly when trained on off-policy data—data generated by a source other than the model itself (Kirch et al., 2026). On-policy data, however, are hard to source because typical model generations are mostly benign. Here, we propose a pipeline where probes are trained on outputs from misaligned model variants, derived from the base model via finetuning or other interventions. We use harm generation as a sample do- main. We find that probes trained on activations from harmful model variants—refusal-ablated, emergently misaligned, and broadly harm finetuned—can effectively identify harmful generations by the base model, outperforming off-policy probes. We also systematically vary different design choices during probe training (outputs selected as positive vs. negative labels and the model supplying activations for probe training), showing that these choices can strongly affect the representation learned by the probe and its ability to classify harmful out- puts. In particular, we show that the typical probe training setup, where benign responses to benign prompts are contrasted with harmful responses to harmful prompts, is brittle compared to contrasting benign versus harmful responses to harmful prompts. Overall, we provide evidence that misaligned model variants are an effective source of data for probe training and outline a comprehensive set of guidelines for better probe design.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Activation Monitoring, Linear Probes, Harm Generation, Emergent Misalignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BvCzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies white-box monitoring probes and their generalization from fine-tuned model variants to the original model. This addresses the problem of monitoring for rare misaligned behaviors, as those are typically hard to elicit from a "normal" model checkpoint, and probes are usually trained on off-policy data. The paper studies this through a variety of "model organisms" derived from the same base model, tuned for various behaviors. Under a very limited model selection, the authors show that probes transfer well when trained on model organisms or on their responses, even when evaluated under the base model prefill.

Critical strengths

-- This is a very solid (and simple) idea -- constructing probes on "less off-policy" generations and model organisms. If it indeed works and scales to larger and more recent models, this is a very promising approach for white-box monitoring of rare behaviors.

-- The methodology, data splits, and model organism selection are clearly described in a reproducible manner.

-- The main strength of the paper is the variety of evaluated misaligned behaviors.

Critical weaknesses

-- Only one base model (Llama-3.1-8B) is used to derive the model organisms. While I understand that this is a straightforward critique to make, and there is not much reason to expect that the same procedure would fail to generalize to a similar model like Qwen2.5-7B, I think it is crucial to expand the model suite along two dimensions: 1) Smaller and larger model sizes, e.g., [1] showed that the concept of deception is significantly harder to monitor in smaller models; and 2) Reasoning models. Including reasoning models would make this work more relevant and address concerns regarding the applicability of the procedure to reasoning traces (e.g., will modified-policy organisms drift too far in their reasoning to generalize to the base model? Does having reasoning drive "visible" output representations too far apart between the base model and the model organism?).

-- The judge models (Llama-3.1-70B and GPT-4.1) appear to be very outdated, and I would appreciate including verification with a newer model, e.g., GPT-6 Luna.

-- While not a critical weakness, I would appreciate it if the authors improved the figures and tables for readability (e.g., avoiding all-caps and excessive statistical significance markers like **). I would also appreciate a better structure in the discussion section, with clear and actionable takeaways on which data generation pipeline should be used for rare behavior monitoring.

[1] Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs (https://proceedings.iclr.cc/paper_files/paper/2026/hash/51510f7458011efb04791417c67798f1-Abstract-Conference.html)

Questions for authors

I think this work investigates a very promising direction, with the main limitation being the meager selection of models. Please see the weaknesses section; adding reasoning models and a model-size ablation to your evaluation suite would significantly strengthen the paper.

3 comments

We thank the reviewer for their constructive feedback and positive assessment of our approach. We have added two additional models (including a reasoning variant), verified the accuracy of our judges against GPT-6 Luna, and improved the paper's readability. See the details below.

Only one base model is used. The reviewer suggests expanding to other model sizes and to reasoning models.

We thank the reviewer for pointing this out. As suggested, we extended our full pipeline to two additional, larger models: OLMo-3.1-32B-Instruct and Gemma-4-31B-it, the latter of which is a reasoning model that we evaluated with thinking both enabled and disabled. For each model, we trained the same set of variants (Risky Financial, Extreme Sports, Bad Medical, Harm Prone, and a new Combined variant trained on all three datasets), generated new evaluation sets, and trained and evaluated all probe types (Section 3, Figure 2).

For Gemma with thinking disabled, variant-trained probes significantly outperform the off-policy probe. With thinking enabled, variant-trained probes remain effective and significantly outperform the off-policy probe when evaluated specifically on naturally generated responses from the base model. In this setting, the off-policy probe performs no better than chance (Figure 5).

For OLMo, we also successfully trained misaligned model variants that could be used for probe training. We found that the base model was particularly resistant to generating harmful responses, even when refusal-ablated or subjected to prefill attacks. Our model variants successfully elicited enough harmful responses to train effective probes. One caveat is that, because the base model did not produce sufficient harmful responses for evaluation, we could only test these probes on harmful responses generated by model variants.

Results from both OLMo and Gemma show that variant-trained probes are less sensitive to the choice of training contrast than off-policy probes (see Section 3, Figure 2). This suggests an additional advantage of our approach: when the optimal training contrast is unknown, variant-trained probes may offer more consistent performance across different contrast choices.

The judge models (Llama-3.1-70B and GPT-4.1) appear outdated. The reviewer suggests verification with a newer model such as GPT-6 Luna.

Following the reviewer's suggestion, we validated our judge against GPT-6 Luna on 4,000 responses (Appendix E3). We find that agreement is high. For harmfulness, the judges agree on 99.5% of labels (κ=0.98\kappa = 0.98, r=0.92r = 0.92), and for coherence on 87% (κ=0.45\kappa = 0.45, r=0.58r = 0.58), comparable to our earlier comparison with GPT-4.1 (98%, κ=0.94\kappa = 0.94 and 92%, κ=0.46\kappa = 0.46, respectively). We retain Llama-3.1-70B as our primary judge for reproducibility.

Figures and tables should be more readable (avoiding all-caps and excessive significance markers), and the Discussion should give clear, actionable takeaways on which data generation pipeline to use.

We thank the reviewer for these suggestions. We have revised our figures and tables to improve readability and restructured the Discussion to provide clearer practical recommendations. Specifically, we removed small caps from variant names, simplified significance reporting to a single marker for AUC, and moved detailed tables containing threshold-based metrics to the appendix.

We also reorganized the Discussion into subsections with clearly stated takeaways and added three actionable recommendations:

  1. Prefer same-prompt contrasts. When possible, train probes to distinguish harmful and benign responses to the same harmful prompts.

  2. Use model variants when suitable on-policy data is unavailable. When the preferred contrast cannot be constructed from naturally generated responses, variant-trained probes, particularly those using representation transfer, are preferable to off-policy probes.

  3. Narrow fine-tuning may be better than broad fine-tuning. Emergently misaligned models consistently outperform the Harm Prone probe when using the optimal contrast. It may be the case that narrow fine-tuning yields variants that are less off-policy than broad fine-tuning, particularly for reasoning models.

We thank the reviewer for their recommendations, as we believe these additions have substantially strengthened the paper. We are happy to address any further comments.

Thank for your rebuttal and addressing my concerns!

I maintain my positive score and recommend this paper for acceptance.

We are grateful for your feedback! We believe that the analyses you proposed substantially strengthened the paper. Please let us know if additional concerns remain that prevent you from rating our work as a 4 - we would love to hear how we might further improve the work.

Reviewer_CsRtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose to use misaligned model variants(refusal-ablated, emergently misaligned, and broadly harm-finetuned) as effective training sources for linear probes that detect harmful outputs from a base model. This paper addresses an interesting problem: how to effecivetively generate useful and on-line negative instances. And the authors systematically compare different generators, activation sources, and negative-class contrasts on LLAMA-3.1-8B-INSTRUCT, finding that model-variant probes outperform off-policy probes, and that benign responses to harmful prompts as negatives outperform the conventional benign responses to benign prompts.

Critical strengths

This paper addresses the scarity challenge of the on-policy harmful instances for harm detection, which is practical useful. And the idea of using misaligned model variants as training sources is reasonable. The diagnostic analysis of negative-class contrast also can provide insight for further studies.

Critical weaknesses

All experiments are conducted on a single model (LLAMA-3.1-8B-INSTRUCT) and a single behavior domain (harm generation), therefore the effectivenss and the generalization of the method should be further verified. Currently, all findings and analysis are draw on experimental performance and comparsions, I think it is better to provide in-depth analysis(e.g., intervention analysis) and explanation.

Questions for authors

See weaknesses in above.

3 comments

We thank the reviewer for their positive feedback. We are glad you found our idea of using model variants as training sources useful and highlighted our analysis of negative-class contrast as “insightful” for future work. We have substantially expanded the set of conducted analyses by adding two additional models (one of which is a reasoning model with reasoning turned on vs. off). See specific responses below.

All experiments use a single model and a single behavior domain

We extended our full pipeline to two additional models, OLMo-3.1-32B-Instruct and Gemma-4-31B-it. The latter is a reasoning model that we evaluated both when reasoning was disabled and enabled (Section 3, Figure 2). In Gemma, we find that variant probes significantly outperform the off-policy baseline under our recommended contrast. In OLMo, all probes perform extremely well, approaching the ceiling. We found that OLMo never generates harmful content when its base variant is used; therefore, our approach is the only way to train probes for this model architecture. Overall, our main findings replicate across models. The recommended contrast (using benign responses to harmful prompts as the negative contrast) outperforms the benign-prompt contrast for nearly every probe, and model variant probes are more robust to differences in probe training contrasts than off-policy probes.

Extending the analyses to other behavioral domains, beyond harm generation, is an important direction for future work. We chose harm generation as a first test case because it is well documented in the literature and one of the few misaligned behaviors for which forced compliance (prefilling or a compliance system prompt) can produce on-policy reference probes for comparison. That lets us benchmark variant probes against an on-policy reference, which is not possible for behaviors such as sandbagging or deception. We have performed some initial analyses on other domains (e.g., deception), but an in-depth treatment of this question deserves its own paper due to substantial cross-domain differences that should be considered. Here, our goal was to establish the methodological foundation that we can then apply across domains. We discuss this in Section 5.3.

Currently all analysis draw on experimental performance and comparisons. It is better to provide an in-depth analysis and explanation

We thank the reviewer for pointing this out. In our original analysis, Figure 3 plots how each probe learns to represent and classify different responses. This figure reveals to us why probes trained on the wrong contrast (benign responses to benign prompts as the negative class) do so poorly. While they are excellent at distinguishing harmful responses to harmful prompts from benign responses to benign prompts, they classify refusals as harmful. They learn to distinguish between the prompts and not the responses. Our results in Figure 4 further support this. When combining the probes trained on the benign prompt responses with refusal probes—that is, probes that distinguish refusal and non-refusal—the performance of all probes instantly hits the ceiling.

We have amended the main text to explicitly distinguish our core results (probe efficacy) from follow-up analyses that examine probe performance in more depth. That distinction was not explicit in our initial writeup; we hope that it’s clearer now.

As for the intervention analyses, they are valuable, but lie slightly out of scope of this particular project: our goal was to train detectors of misaligned behaviors, not necessarily to use them for steering. We note that our refusal ablation variant is a result of a causal intervention; the fact that our probes still perform well on the refusal-ablated variant indicates that they do not simply pick up on refusals, a conclusion also supported by the analysis in Figure 3. We are in the process of reframing our refusal ablation results to highlight this causal relationship. Additional in-depth analyses are subject to future work, which is feasible now that we have established the efficacy of our probes!

We are happy to answer any additional questions.

One additional update following our previous response. We have now completed the reframing of our refusal-ablation results, as mentioned in our earlier reply.

In the revised manuscript (Section 4.3, Figure 4), we explicitly test our hypothesis that refusal-related representations are spuriously correlated with prompt harmfulness. We do so by ablating the refusal direction in Llama's activation space and showing that probes trained on the resulting activations remain robust across training contrasts. Together with our probe-combination results, this provides further evidence for why probes trained on the suboptimal contrast misclassify refusals as harmful.

We hope this additional analysis helps address your request for a deeper explanation of our findings, and we would appreciate any further feedback!

Thank you again for your review. As the discussion period draws to a close, we wanted to check whether our revisions have sufficiently addressed your concerns. To summarize our revisions, we have extended our entire pipeline to two additional models (OLMo-3.1-32B and Gemma-4-31B, the latter with reasoning enabled and disabled), and found that the results generally replicate across them. We also added a causal analysis (Section 4.3, Figure 4) explaining why probes trained on the conventional contrast (harmful responses to harmful prompts vs benign responses to benign prompts) perform worse.

Please let us know whether our expanded set of analyses addresses your concerns.

Reviewer_N1A1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose training harmful response detectors using data and activations from specially modified versions of the model, and then applying them to the original Llama-3.1-8B-Instruct. They compare several data sources and methods for selecting safe examples. In the main evaluation, probes trained on model-variant data outperform the Gemma-based off-policy probe, but the best result is achieved by prefilling the original model. Additionally, the authors show an interesting detail - safe responses to dangerous queries are usually more useful as negative examples than responses to regular queries.

Critical strengths

Modified models help extract harmful examples that the original model rarely produces. Comparing activation sources, testing with the same number of training examples, and testing on held-out prompt categories are useful. Selecting safe responses to a dangerous queries helps the detector avoid confusing a dangerous topic with a harmful response.

Critical weaknesses

It is unclear how well the detector works on harmful responses generated by the original model without intervention. The test mixes regular responses, prefilled responses, and responses from the model with rejection disabled (Section 3.7). According to Tables 2 and 4, the regular model produced only 41 and 225 harmful responses on the two test sets. If responses weren't resampled, this represents no more than 8.9% of the 3,000 harmful test responses. Separate results by generation source are needed to understand whether the advantage persists for regular responses.

Comparisons that demonstrate the benefit of the modified models activations are missing. The same texts are run through both models only for Harm Prone and Refusal Ablated: for the first one, the activations of the original model are better, while for the second - worse. No such comparison is available for Risky Financial. Moreover, the simpler prefill gives an AUC of 0.945 versus 0.918, and the external generator for comparison is only one Gemma-2B.

Questions for authors
  1. What is the proportion of regular, prefilled, and refusal-ablated responses in the test? Does the advantage of regular responses hold true separately, with confidence intervals for the queries?

  2. What results do Risky Financial, Extreme Sports, and Bad Medical produce if the same texts are run through the original model and the detectors are trained on its activations?

  3. Table 2 shows 5771 refusals and 5879 benign responses for 6250 responses. Does benign include refusals, and how many examples of each class remain after the filtering in Section 3.5?

4 comments

We thank the reviewer for their constructive feedback. We are glad the reviewer appreciated the use of modified models in eliciting harmful examples that the original models would rarely produce. We are also encouraged that reviewers highlight our evaluation design as a core strength: comparing activation sources, controlling the number of training examples, testing on a held-out prompting categories, and using benign responses to dangerous queries so that the detector avoids confusing harmful topics with harmful responses. Below, we respond to specific questions raised by the reviewer:

Q1. What is the proportion of regular, prefilled, and refusal-ablated responses in the test? Does the advantage of regular responses hold true separately, with confidence intervals for the queries?

As suggested by the reviewer, we separated the test set into groups by generation source. We now report group-specific results in Appendix H. The number of responses in each group is: 1,986 regular (3.9% harmful), 1,666 prefilled (60% harmful), and 2,348 refusal-ablated responses (81.9% harmful). Figure 17 shows AUROC by response source. To ensure that the AUROC measure accurately reflects probe performance even for heavily class-imbalanced groups, we also computed the AUPRC (area under the precision-recall curve) and included it as Figure 18. Probes perform well for each group, suggesting that the method should be applicable to regular model generations.

Q2. What results do Risky Financial, Extreme Sports, and Bad Medical produce if the same texts are run through the original model and the detectors are trained on its activations?

Thank you for asking this question! We ran experiments and added the results to our main Figure 2. Under our recommended contrast (harmful vs. benign responses to harmful prompts; left panel), modified-policy probes perform comparably to their rep-transfer counterparts, and all significantly outperform the off-policy baseline. Conversely, under the benign-prompt contrast (right panel), every rep-transfer probe outperforms its modified-policy counterpart. This suggests that when the training contrast isolates response harmfulness, not only are the representations transferable but so are the outputs used to extract those representations. When the contrast is confounded with prompt harmfulness, training on the variants' activations partly mitigates that confound. We thank the reviewer for prompting this analysis. We have revised Section 4.1 and the Discussion to reflect this change in our interpretation of the results.

Q3. Table 2 shows 5771 refusals and 5879 benign responses for 6250 responses. Does benign include refusals, and how many examples of each class remain after the filtering in Section 3.5?

We thank the reviewer for catching this. Yes, the "Benign" column in Table 2 included refusals. In the paper we use "benign" in two senses: narrowly, for non-refusal responses without harmful content (Section 3.5), and in places more broadly, for all non-harmful responses including refusals. To avoid further confusion, we now use "benign" only in the narrow sense and "non-harmful" for the union of benign responses and refusals, and we have corrected this terminology throughout the paper. As described in Section 3.5, probe training excludes refusals, incoherent responses (coherence below 50), and ambiguous responses (harmfulness between 30 and 70). We have corrected Table 2 to report refusals and benign responses separately after this filtering. For example, the Base model's training split contains 5,771 refusals, 279 benign responses, and 64 harmful responses. Our core results are not affected by this change.

We believe that addressing these points has substantially strengthened the paper and are happy to address any additional comments.

We thank the reviewer again for their previous feedback. We wanted to let you know that we have substantially revised the manuscript, expanding our experiments to two additional models (OLMo-3.1-32B and Gemma-4-31B, the latter with reasoning enabled and disabled), adding new analyses, and improving the paper's clarity and presentation.

In particular, we appreciated your suggestion to evaluate probe performance separately by response source. We have now emphasized this analysis in our main results (Figure 5), where we find that variant-trained probes substantially outperform the off-policy baseline on naturally generated responses from reasoning-enabled Gemma!

We believe these revisions have substantially strengthened the paper and would greatly appreciate any further feedback you might have!

Thank you for the substantial revisions and additional experiments!

The updated manuscript addresses my main concerns. Additional experiments on OLMo-32B and Gemma-31B, including reasoning, also significantly strengthen the paper.

One question remains: Appendix H describes bootstrapping over responses, while Figures 26–27 state that confidence intervals are computed by resampling prompts. Could you clarify which method was used? Since multiple responses can share the same prompt, this distinction matters for the reported uncertainty.

Overall, I find the revised paper substantially stronger. The authors have addressed my main concerns with new experiments and clearer analysis.

I am updating my overall assessment from 2 to 3.

Thank you for your updated assessment! We also thank you for pointing out the inconsistency in our bootstrap sampling procedure .

Appendix H describes bootstrapping over responses, while Figures 26–27 state that confidence intervals are computed by resampling prompts. Could you clarify which method was used? Since multiple responses can share the same prompt, this distinction matters for the reported uncertainty.

Figures 26–27 used bootstrap samples of individual responses but their captions incorrectly said “prompts.” We have corrected those captions. Your question also prompted us to audit the other reported intervals and significance tests. We reran the affected analyses using responses as the resampling unit, regenerated Figures 2, 4, and 5, and updated the corresponding significance marks, tables, and Results text. The point estimates, results, and takeaways do not change. We’ve also updated appendix H to explain the procedure explicitly.

We are grateful for your revised assessment and comments. If a specific remaining concern or additional analysis would make the evidence sufficient for a score of 4, we would welcome your guidance and address it if feasible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.