acceptodds
Under review as a conference paper at ICLR 2027

Alignment via Training Against Probes Without Losing Monitorability

52 views

Abstract

Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without internalizing the intended behavior, for example by faking compliance during training. Such superficial compliance could be harder when the objective is defined on model internals rather than outputs. Therefore, we study probe-guided fine-tuning, using probes that detect undesired properties in model activations as a direct training signal. We evaluate linear and non-linear probes with different numbers of probes per layer across two alignment objectives: harmlessness and honesty. We find that training against probes that do not update during training is an easily exploitable objective, while continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility. Probe-guided fine-tuning achieves better safety–utility trade-offs than DPO and inference-time steering, while being substantially more robust against jailbreak and abliteration attacks. Moreover, the concepts stay linearly encoded after fine-tuning, meaning oversight is not lost by our method. Training against probes thus offers a way to shape what models represent rather than only what they output, which may become increasingly important as models get better at making their outputs look aligned.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
interpretability, linear probes, monitorability, harmfulness, honesty, AI safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7ZWSOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors ask: can we use a model internal representations during training as a gradient signal to make the model more aligned? And they show that the answer is yes. On harmfulness (refusal) and honesty tasks, their method with dynamically updating probes shows stronger results than the main DPO baseline, while also generalizing better (as shown by the robustness experiments with GCG/prefill attacks) and converging faster. The authors also show that the naive method, where the probe weights are fixed once at the start of training, is not effective: the model learns to bypass the probe in its activations, which is consistent with the findings of prior works. The paper itself is very well written, with a simple and clear methodology and fairly broad ablations, which makes me more confident in the results.

Critical strengths
  1. A very important and timely research question about whether interp should be used during training. The authors give a positive answer and show that it can even beat baselines like DPO.

  2. Compelling results on fixed vs. dynamic probes. With the fixed probe, you can see how the model bypasses it (classic Goodharting), but the dynamic probe doesn't have this problem and lets the model become truly harmless and honest.

  3. Important results showing that with the dynamic probe the model becomes robust to GCG/prefill attacks (unlike with DPO).

  4. A nice qualitative analysis of the model's behavior after this fine-tuning (Figures 2 and 7). It's quite interesting that in this setup the model generates Nonsensical / Disrupted, Soft Refusal or Pseudo-Compliance outputs instead of clear refusals.

  5. A simple and clear methodology for all experiments, and the paper itself is very clearly and well written.

Critical weaknesses
  1. Only a small number of models were tested. I understand that there are few open instruct models that don't refuse, but in that case you could have used more refusal-abliterated model variants.

  2. I think the comparison with DPO is not entirely fair. This is my main concern: maybe the advantage of your method is really just the advantage of online methods in general (it's known that they generalize and work better than offline methods like DPO). It might be worth running some online RL method without access to the latents and comparing against it.

Questions for authors

How do you explain that after this probe training, removing the refusal direction no longer jailbreaks the model?

3 comments

Dear Reviewer 7ZWS,

Thank you for the positive feedback and for finding our research question important and the paper clearly written. We follow your advice and add another baseline that uses online RL to our comparison, and provide clarification on why abliteration is not effective to jailbreak our models.

  1. Comparison to online training

    We add GRPO as a baseline comparison as a representative for an online RL method (Table 1, Figure 5, Appendix G.3). GRPO samples eight responses per prompt from the current model at every step. A reward model trained on the same paired data as our probes then scores these responses. Within 50 model update steps, probes and polytopes reach lower StrongREJECT scores at high utility (Figure 5). Even with its full budget of 300 steps, GRPO on Mistral reaches 0.31 on direct queries, 0.52 under GCG and 0.74 under prefill. The continuously updated probe reaches 0.07, 0.01 and 0.06 within 50 steps (Table 5).

  2. Abliteration

    Abliteration removes the direction that mediates explicit refusals. Our models, however, almost never refuse explicitly: at the selected checkpoints, at most 1% of responses are explicit refusals (Figure 7). Instead, the models produce soft refusals, pseudo-compliance and disrupted text. Their safety therefore likely does not depend on a single refusal direction that abliteration targets.

  3. More models

    We agree that more models would strengthen our results. Our harmfulness experiments currently use Mistral 7B Instruct v0.1, which was never trained to refuse, and an abliterated Llama 3 8B Instruct. The honesty experiments use Qwen3-14B. Following the reviewer's suggestion, we will add an abliterated model from a different model family, and update the manuscript again, over the course of the discussion period.


Thank you again for the constructive feedback and positive assessment of our paper!

To demonstrate that our results transfer to other model families as well, we repeat the continuously updated linear probe and non-linear polytope runs for the harmlessness setting on Phi-4, abliterated with Heretic. Section E.5 shows that our technique also works with this model: it becomes significantly safer while retaining at least 95% of the base utility and keeping the over-refusal rate at or below 5%.

Thank you for the additional experiments and clarifications! The GRPO comparison and the results on Phi-4 address my main concerns. I am happy to keep my rating of 4.

Reviewer_8NSJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the superficial compliance problem, where the model might learn to seemingly align in the output space without internalizing the intended behavior. To this end, this paper studies probe-guided fine-tuning. Specifically, they study whether internal probes can be directly used to train LLMs for safety and honesty. Experiments show that probe-guided fine-tuning shows better safety–utility trade-offs than DPO and inference-time steering, while being substantially more robust against jailbreak and ablation attacks.

Critical strengths
  • Interesting research question. The paper investigates an important question: whether internal probes can serve as alignment training objectives without losing their ability to monitor model behavior.
  • Interesting empirical findings. The paper found that continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility.
Critical weaknesses
  • Unclear motivation for using probing as a guide for fine-tuning. The problem motivation stems from the "fake alignment" (Lines 36 - 38). It remains unknown whether activation-based probes are useful for instilling safety (Lines 47 - 49). Although it is true that limited existing works study alignment from the activation perspective, I am curious what the intuition is for adopting this perspective? In other words, why is optimizing internal representations fundamentally less vulnerable to superficial alignment than optimizing outputs?
  • Baseline is not complete. Apart from the existing baseline, I would suggest that the authors add more baselines from the DPO variants to fully demonstrate the effectiveness of your method. Many papers demonstrate the limitation of vanilla DPO in safety alignment, e.g., [1, 2].
  • In Section 4.3, the authors acknowledge that explicit refusals account for at most 1% of responses, while many responses exhibit pseudo-compliance or disrupted text, particularly for Mistral. This raises concerns about whether the proposed probe-guided method can really improve safety alignment.

[1] Hu M, Datla V, Kumar A, Guan Z, Li S, Samuel A, Liu D. Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment. In International Conference on Learning Representations 2026 Apr 20 (Vol. 2026, pp. 63876-63899). [2] Zhao X, Cai W, Shi T, Huang D, Lin L, Mei S, Song D. Improving llm safety alignment with dual-objective optimization. arXiv preprint arXiv:2503.03710. 2025 Mar 5.

Questions for authors

See Weakness. My final scores will be determined based on the author's responses.

3 comments

Dear Reviewer 8NSJ,

Thank you for your thorough review, and we are glad that you find our research idea and findings interesting and important!

We are pleased to address your concerns by clarifying the motivation for our approach and in what way we think our method enables the model to become safer, and by providing new experimental results, namely adding GRPO (Shao et al., 2024) and DOOR (Zhao et al., 2025) as baselines.

  1. Why train against internal representations?

    Output-based training objectives can be underspecified (D'Amour et al., 2022). They specify what a "good" output looks like, but not how the model should get there. A model that still represents the undesired behavior internally but hides it in its outputs can therefore score as well as one that is actually aligned. For example, we see this in reward hacking. When Baker et al. (2025) penalized a reasoning model for reward-hacking intentions in its CoT, the model kept hacking its coding tests but stopped mentioning it. Their objective was satisfied, but the unwanted reward hacking behavior persisted. Similarly, models can learn to satisfy the grader instead of the intended goal (Højmark et al., 2026), and more capable models may even fake alignment (Greenblatt et al., 2024).

    With internal representations we cannot only shape what the model outputs, but also what it represents internally. Our probe-based objective targets the harmful or dishonest concept directly, so a model that only changes its outputs is still penalized as long as the probe detects the concept in its activations.

  2. Additional baselines

    Besides DPO, we now include comparisons against GRPO and DOOR (as proposed by the reviewer), as well as probe-reward DPO (Table 1, Figure 5, Appendix G). All methods use the same paired data. At the matched number of gradient updates, probes and polytopes reach lower StrongREJECT scores at high utility than all three output-level training baselines (Figure 5). We also find probe-trained models more robust under prefill attacks on both models and under GCG on Mistral (Table 1). For example, on Mistral, the continuously updated probe reaches 0.01 on GCG and 0.06 on prefill attacks, compared to 0.66 and 0.78 for DOOR. DOOR lowers StrongREJECT on direct queries but gains little robustness (Appendix G.4). Both DPO and GRPO need more gradient updates than probe-trained models to significantly lower StrongREJECT (Figure 23).

  3. Can you improve safety without explicit refusals?

    A model can be harmless without refusing explicitly. What matters is that it does not produce usable harmful content. We measure this with the established jailbreak judge StrongREJECT (Souly et al., 2024). The examples in Figure 2 and Appendix D.2 illustrate this. For instance, the model reinterprets a harmful request as a cooking question and gives a harmless answer. We also make sure that the model stays helpful on harmless questions. All selected checkpoints retain at least 95% of base utility and refuse at most 5% of the safe prompts in XSTest (Table 1).

    Not relying on explicit refusals also makes the model more robust to abliteration, a common uncensoring intervention that removes the refusal direction (Arditi et al., 2024). After abliteration, our continuously updated checkpoints reach StrongREJECT scores of 0.10–0.39, compared to 0.76 for abliterated Llama 3 8B Instruct (Section 4.3, Table 7).

References

Arditi, A., Obeso, O., Syed, A., et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024.

Baker, B., Huizinga, J., Gao, L., et al. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv:2503.11926.

D’Amour, A., Heller, K., Moldovan, D., et al. (2022). Underspecification Presents Challenges for Credibility in Modern Machine Learning. JMLR 23(226).

Greenblatt, R., Denison, C., Wright, B., et al. (2024). Alignment Faking in Large Language Models. arXiv:2412.14093.

Højmark, A., Scheurer, J., Nitishinskaya, E., et al. (2026). Measuring Reward-Seeking via Contrastive Belief Updates. arXiv:2607.18966.

Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.

Souly, A., Lu, Q., Bowen, D., et al. (2024). A StrongREJECT for Empty Jailbreaks. NeurIPS 2024 (Datasets and Benchmarks Track).

Zhao, X., Cai, W., Shi, T., et al. (2025). Improving LLM Safety Alignment with Dual-Objective Optimization. ICML 2025.


We thank you again for the helpful feedback and suggestions. The added stronger baselines have made our comparison stronger and shows more clearly where probe-guided training improves on output-based alignment. We hope our responses have addressed your questions and concerns and would be grateful if you would consider raising your score.

  1. Are there any guarantees (or at least intuitions) that, if you use internal representation, the mentioned problems can be mitigated? You mainly answer by discussing the disadvantages of "output-based" alignment, but this doesn't directly motivate your research on "representation-based" alignment, right? I think more explanation of the necessity of using "representation-based alignment" would be important.

2 & 3. Thanks for the authors' rebuttal. My concerns are addressed for these two.

Thank you for your response! In computer vision, theoretical results have established that representation-level objectives can provably isolate invariant content from changing style [1], while appropriate structural constraints on learned representations enable compositional generalization [2].

These early results motivate extending the idea of representation-level supervision to LLM alignment, with [3] further prove that autoregressive objectives are underspecified with respect to desirable model properties, providing a rationale for imposing additional constraints on internal representations.

While theoretical arguments for provable efficacy of representation engineering in LLMs currently lacking, empirical research on this topic, including Representation Engineering [4] and Circuit Breakers [5], has already demonstrated substantial improvements in model safety and adversarial robustness through interventions on internal representations.

We can offer an intuition that producing an aligned-looking response can satisfy an output- based objective while leaving the probed internal property unchanged. A representation-level objective continues to penalize that property, providing a constraint on how the model can satisfy the training objective.

Our paper builds on this established direction but addresses a specific concern related to scalable oversight and AI control: whether training against model internals compromises monitorability. To answer this question, we introduce a method based on continuously updated probes, and through substantial experimental evidence show that it achieves competitive or better alignment than output-based methods, with improved safety–utility trade-offs and adversarial robustness, while preserving linear decodability and monitorability of the targeted properties.


We kindly ask the reviewer to consider raising their score in light of the specific concern our work addresses within representation engineering field and the demonstrated effectiveness of our method in improving alignment while preserving monitorability.

References

[1] von Kügelgen et al. (2021). Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style. NeurIPS 2021.

[2] Wiedemer et al. (2024). Provable Compositional Generalization for Object-Centric Learning. ICLR 2024.

[3] Reizinger et al. (2024). Position: Understanding LLMs Requires More Than Statistical Generalization. ICML 2024.

[4] Zou et al. (2023). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv:2310.01405.

[5] Zou et al. (2024). Improving Alignment and Robustness with Circuit Breakers. NeurIPS 2024.

Reviewer_z3LPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper studies probe-guided fine-tuning, which trains a language model on probe scores from its internal activations. The motivation is that output-only training may yield models that seem aligned without learning the intended behavior.
  • The probes are small classifiers on intermediate-layer activations that flag harmfulness or dishonesty. Four detectors are tested, combining linear or non-linear probes with one or several probes per layer.
  • The harmfulness experiments compare two probe update regimes. A frozen probe keeps its initial weights. A continuously updated probe is refit on the same labeled data after each model update.
  • At each step, LoRA adapters are trained with a hinge loss that pushes the activations of the model's responses to the harmless or honest side of the probe's decision boundary. A KL penalty on benign examples keeps the model close to the base model.
  • Harmfulness is scored by StrongREJECT, an LLM-judge score, on JailbreakBench and on held-out ClearHarm prompts, the latter also under GCG and prefill jailbreak attacks. The harmfulness models are Mistral 7B Instruct v0.1 and an abliterated Llama 3 8B Instruct whose refusal direction has been removed. Honesty is measured on Qwen3-14B with MASK, which checks whether answers under pressure contradict the model's stated belief.
  • Baselines are DPO on both harmfulness models and two inference-time steering methods, CAA and SafeFlow, on Mistral. Utility is the mean of MMLU, GSM8K, and IFEval relative to the base model. Each fine-tuning method is reported at the checkpoint with the lowest JailbreakBench StrongREJECT or highest MASK honesty that keeps at least 95% of base utility.
  • With a frozen probe, the probe loss nears zero but harmfulness remains high. With a continuously updated probe, harmfulness falls on both harmfulness models and MASK honesty rises on Qwen3-14B.
  • On Mistral, the method achieves a better safety-utility trade-off than steering, and than DPO when data is limited. Under GCG and prefill attacks, the selected continuously updated checkpoints score lower than the base models. Abliteration has little effect on these checkpoints' direct-query ClearHarm score but sharply raises that of standard Llama 3 8B Instruct.
  • For monitorability, a linear probe trained from scratch on each selected continuously updated checkpoint reaches a held-out AUROC close to that of a base-model probe.
Critical strengths
  • The paper identifies a failure mode of training against a frozen probe and proposes a fix.
    • In Section 4.1 and Figure 3, JailbreakBench StrongREJECT stays above 0.40 with a frozen probe even after the probe loss reaches zero.
    • Figure 11 shows why. Scores of harmful completions cross the frozen probe's decision boundary, so it stops flagging them.
    • In Figure 4, continuous updates largely avoid this failure for all four detectors on both harmfulness models.
  • Harmfulness decreases and MASK honesty increases while utility remains largely preserved.
    • In Table 1, MMLU, GSM8K, and IFEval scores stay within 0.07 of the base model.
    • In Figure 6, MASK honesty rises on Qwen3-14B with utility at the base-model level.
Critical weaknesses
  • Monitorability is evaluated as linear decodability, which does not by itself show that the probe-finetuned model's harmful responses are reliably flagged. Tables 2 and 3 show the target property stays linearly decodable, meaning a newly trained linear probe separates the two classes about as well as on the base model under both probe regimes. The abstract concludes from this that "oversight is not lost by our method."
    • Table 2 does not say whether its held-out AUROC uses fixed completions or the model's own responses. Since the objective in Section 3.4 optimizes probe scores on the model's own responses, detection should be tested on those responses.
    • A similar AUROC can hide lower recall at a fixed threshold because AUROC summarizes all thresholds while a monitor uses one. In Figure 11, the frozen probe's scores shift down for both classes, which stay separable, yet harmful completions fall below the original boundary.
  • The need for probe updates is shown only for harmfulness and only under this paper's training objective.
    • Frozen probes are tested only for harmfulness, so it is unclear whether honesty training needs updates. The abstract, covering both objectives, calls training against non-updating probes "an easily exploitable objective."
    • The need may also depend on the training algorithm, which Section 2 lists as a factor in whether probe feedback causes evasion. In Wehner & Fritz (2025), DPO on base-model preference pairs labeled by a frozen probe still reduced toxicity, while SFT with a frozen-probe penalty suggested partial evasion. Both results concern toxicity in a different setting.
    • The paper should therefore scope this claim to its direct probe-score objective. Comparing frozen and updated probes in iterative DPO, where the probe labels pairs from the current model's responses each round, would test whether updates are needed for other objectives.

References

  • Wehner, Jan, and Mario Fritz. "Probe-based Fine-tuning for Reducing Toxicity." arXiv preprint arXiv:2510.21531 (2025).
Questions for authors
  • How well do the newly trained and base-model probes detect the probe-finetuned model's own harmful responses?
    • Is the held-out AUROC in Table 2 computed on fixed completions or on the checkpoints' own responses?
    • Could you evaluate the newly trained probe on the selected checkpoints' responses to held-out prompts such as ClearHarm, including under GCG and prefill attacks? The layer should be chosen on a separate validation set and the responses labeled by a judge independent of the probe. Please report the AUROC, the number of responses per class, and the recall and actual false-positive rate at a threshold set on benign validation responses for a target false-positive rate such as 1%.
    • Is a monitor built before training still usable? Without retraining, what are the base-model probe's AUROC on these responses and its recall and false-positive rate at its original decision boundary?
  • Do the honesty experiments also need probe updates?
    • If Qwen3-14B is trained against a frozen probe, how do probe loss and MASK honesty compare with the continuously updated run? How well does the frozen probe then detect dishonest responses, in AUROC and in recall at its original boundary?
  • What responses account for the gains on ClearHarm and MASK?
    • Does the lower ClearHarm StrongREJECT come from refusals, pseudo-compliance that looks helpful but gives no usable instructions, or disrupted text? Extending the JailbreakBench analysis of Figure 7 to ClearHarm, including under GCG and prefill, would answer this.
    • Does higher MASK honesty reflect more honest answers or more evasive ones, which MASK also counts as honest? Please report the shares of honest, lying, and evasive outcomes for the base model and the selected checkpoints, with cases lacking a stated belief counted separately.
2 comments

Dear Reviewer z3LP,

Thank you for your questions and feedback. We now address your concerns with new experimental results, namely: expanding our monitoring analysis, running the frozen probe setting for honesty, adding a "probes-as-reward-signal" baseline, and extending our output-type analysis to attack outputs and to honesty.

  1. Monitorability

    Table 2 reports AUROC using off-policy test split completions (harmfulness: BeaverTails, dishonesty: Liars' Bench). We now extend this in Table 4 (page 23) to all training and one held-out distribution.

    To evaluate on the model's own responses, we need harmful generations from our selected trained model checkpoint. As these are rare (Llama with a continuously updated probe at our selected checkpoint has StrongREJECT ≥ 0.5 on only 25 of 120 ClearHarm answers across direct, GCG and prefill), we sample more direct and prefill responses until we have 50 harmful ones, labeled by StrongREJECT, and pair with 50 benign ones. A fresh linear probe fit on BeaverTails (same setup and layer as Table 2) reaches an AUROC of 0.94 on these on-policy responses. With the threshold set on 200 separate benign responses for a 5% FPR, recall is 0.90. As a result, the model is also monitorable on on-policy completions.

    A monitor fit on the base model before training (step-0 probe), however, is not usable afterwards. As shown in Appendix B.4 (Figure 10), recall drops to almost 0 within 4–6 steps as the base-model probe scores almost all responses below its original boundary. Our claim is that the property remains detectable by a refit linear probe.

  2. Frozen probes for honesty

    We now demonstrate that frozen probes are also evaded in the honesty setting, by training Qwen3-14b. The probe loss reaches 0 by step 10, and dishonest activations shift across the fixed boundary while remaining separable (Figure 13). Evasion happens slower than for harmfulness, which we hypothesize is because we train against several layers. MASK honesty also improves less than with continuously updated probes (Figure 18). The step-0 probe still separates the classes on the frozen-probe checkpoint (AUROC 0.976 vs. 0.988 on the base model). However, its recall at the original boundary falls from 0.98 to 0.36. Since the classes remain separable, this drop only reflects a shifted threshold.

  3. Training against probes vs. using probes as reward signal

    To compare our method with Wehner & Fritz (2025), we now add their approach as a baseline to Table 1. In their approach, the probe ranks sampled responses to form DPO preference pairs. With a frozen probe, we find that this objective is not leading to evasion. Instead, it teaches the model to refuse even on harmless prompts: on Mistral, more than 40% of safe XSTest prompts are refused. Under our selection rule (at least 95% utility, at most 5% over-refusal on XSTest), its selected checkpoint scores 0.55 on ClearHarm direct queries, compared to 0.07 for our continuously updated probe (Table 5), and it is less robust to attacks.

    Refitting the probe after every update roughly halves over-refusal, which we find consistent with overall improvements over continuously updated regime for original objective.

  4. Output types on ClearHarm and MASK

    For ClearHarm, we classify the outputs of the selected Mistral checkpoint trained with a continuously updated probe (Table 1):

    CategoryDirectGCGPrefill
    Nonsensical/Disrupted202333
    Pseudo-compliance5165
    Soft refusal900
    Harmful compliance412
    Explicit refusal100
    Different topic100

    As on JailbreakBench (Figure 7), the lower score comes from disrupted text, pseudo-compliance and soft refusals rather than explicit refusals. Since the model has learned a particular way of generating, this changes little with a different prompt set, a suffix or a short prefill.

    For MASK, we report the share of each outcome (%):

    ModelHonestLieEvasiveNo consistent beliefHonesty
    Base165310210.47
    Cont. update probe (step 17)22376350.63
    Cont. update polytope (step 18)253021240.70

    Both checkpoints lie less and give more honest answers.

References:

Wehner, J., & Fritz, M. (2025). Probe-based Fine-tuning for Reducing Toxicity. arXiv preprint arXiv:2510.21531.


We thank you for your insightful questions and suggested experiments, which have significantly improved our paper! We hope we have addressed your concerns and remain open to any further questions. We would be very grateful if you would consider increasing your score.

As the discussion period is ending soon, we wanted to follow up on our response. We would love to engage with any remaining questions or concerns you may have.

In our response, we addressed your points with new results:

  • Monitorability on on-policy responses: a freshly fit linear probe reaches an AUROC of 0.94 on the model's own responses, with 0.90 recall at 5% FPR. We also clarify that the step-0 probe is not usable after training.
  • Frozen probes for honesty: frozen probes are also evaded in the honesty setting, and MASK honesty improves less than with continuously updated probes.
  • Probes as reward signal: we added the Wehner & Fritz (2025) DPO approach as a baseline (Table 1, Table 5).
  • Output types: we report the output categories on ClearHarm (direct, GCG, prefill) and the honest/lie/evasive shares on MASK.

If we have addressed your concerns, we would be very grateful if you would consider increasing your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Conclusion

As the discussion period is coming to an end, we wanted to briefly summarize the strengths the reviewers identified, and how we addressed their concerns.

Strengths

The reviewers found that our paper addresses an “interesting” and “important” (8NSJ), as well as “very important and timely” (7ZWS), research question. They described the paper as “very clearly and well written”, with a “simple and clear methodology” and “fairly broad ablations” (7ZWS). They also highlighted our “interesting empirical findings” (8NSJ), in particular the comparison of frozen and continuously updated probes (7ZWS, z3LP), the robustness to GCG and prefill attacks (7ZWS), and the qualitative analysis of the model’s behavior (7ZWS).

Concerns and how we addressed them

  1. Monitorability on the model’s own responses (z3LP): We show that the model is monitorable on both off-policy and on-policy completions. That means that the trained model’s own harmful responses can be detected.
  2. Frozen probes for honesty (z3LP): Our frozen-probe results originally covered only harmfulness. We added the corresponding run on Qwen3-14B and find that the frozen probe is evaded in the honesty setting as well (Figures 13 and 18).
  3. Stronger baselines (8NSJ, 7ZWS, z3LP): We added DOOR, GRPO, and probe-reward DPO as baselines, all trained on the same paired data. At matched gradient updates, probe-trained models reach lower StrongREJECT scores at high utility than all three (Table 1, Figure 5).
  4. Safety without explicit refusals (8NSJ, z3LP): The reviewers asked whether the model is really safer when it rarely refuses explicitly, and which responses account for the gains. We extended our output-level analysis to ClearHarm under attacks and to MASK.
  5. Motivation (8NSJ): The reviewer asked why training against internal representations should be less vulnerable to superficial alignment. We clarified that output-based objectives are underspecified, while a probe-based objective still penalizes a model that only changes its outputs.
  6. More models (7ZWS): Following the reviewer’s suggestion, we added Phi-4, abliterated with Heretic, as a model from a different family. Our technique also works with this model (Section E.5).

Overall, our results show that training against continuously updated probes makes models substantially more harmless and honest, while preserving utility and monitorability. As models become more capable, alignment that is only based on observed outputs or chain-of-thought becomes less reliable, since both can look aligned while the underlying behavior persists. We therefore believe our paper provides an important piece of evidence that representation-based alignment is a practical and promising direction. We thank all reviewers for their constructive feedback, which has substantially improved the paper, and remain available for any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.