- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether an LLM can evade a latent-space monitor with the information from the probes prior verdicts of the earlier turns in the same context. Each context has 32 turns with monitor verdicts and ends with an instruction to imitate either verdict 1 or 0. To make sure the evasion comes from activation changes instead of the text itself, this last response is forced to a concept-positive sentence. For off the shelf models, the prior verdict contexts alone are not sufficient to change the activations in a way that evades the monitors on their own, but the activations point the right way and rescaling by 8x reduces TPR from 1.00 to 0.27. A cheap rank-1 LoRA model organism that evades held-out concept monitors. Characterization of the mechanism, by running ablations of example count, flipped verdicts and attention knockouts. Finally, evaluating defenses, showing that ensembling monitors limits the success of training model organisms, but it does not mitigate the amplified base edit.
Critical strengths
In my estimation most of the experiments were well controlled and the mechanisms well investigated. Forcing the response is good to solely study the effect of the activations and doing attention knockouts (Section 5.2) help understand where the effect is computed and where it breaks. Controls like comparing the rescaled edit with random directions of the same size and with other concepts' monitors (Figures 2, 3), running experiments across four models and showing the results hold, running multiple probe types and multiple seeds, flipping labels and ablating the number of examples (Section 5.1) make me more confident the effect is real.
Critical weaknesses
It’s clear that this effect could be a concern for latent-space monitoring, however the link from this experiment to probe evasion was not clearly established and the results don’t yet convince me the evasion threat is problematic. I attribute this mostly to the fact that the setup is too far away from the threat that motivated it.
Labels are very explicit, balanced and there are also explicit instructions to imitate the label (Sec. 3). This would not necessarily be the case in a real setting and I believe it should be feasible to run the experiment in a more realistic setup. The result reads closer to "when an LLM is instructed to imitate labelled in-context examples, the model's activations move toward them" than to a model evading a monitor. I understand that explicit signals here carry the same information as implicit signals, but it is not clear that a model would know how to use the implicit signal. Some work is needed to bridge the gap, or the claims and presentation should be narrowed. Narrowing the claims seems feasible in a rebuttal, but the goal-directed experiment would convince me more.
The monitors all read one layer at three quarters of the depth (Section 3), it is unclear whether the evasion holds against probes trained at other layers or against multi-layer probes or ensembles. It would be great if this could be ran in the rebuttal.
Questions for authors
Does evasion work in a more goal-directed setting? For example, fill the context with actions labelled "monitor flagged" or "monitor didn't flag" and give the model the goal (e.g. in the system prompt) of slipping an action through without it getting flagged, and drop the explicit instruction to imitate a label. This would separate "when asked to imitate data in context, the model attends to it and its activations move toward it" from "when the model has the goal of evading a monitor, prior verdicts help it do so". This would address my primary concern. Does evasion work against probes trained at different layers? If it does, my concern in Weakness 2 is mostly resolved.
Does evasion work when the model writes its own response?
2 comments
Following up on our response
As the discussion period comes to a close, we wanted to make sure our response addresses your concerns. In particular, we ran the goal-directed experiment you suggested, with the goal given only in the system prompt and no imitation instruction, and found that a model organism trained in this format does evade held-out monitors (TPR 0.07). We would be glad to answer any further questions, and appreciate your thoughtful review!
We thank the reviewer for their careful reading, and are glad they found our experiments well controlled and that they are confident in the strength of our results. We appreciate the reviewer spelling out their concerns in an addressable way, and present new results below:
W1: Realism of the setup
We agree with the reviewer that the goal-directed setting is an important test, and have run the suggested new experiments to make our setup more realistic, as well as refining our claims in the main text.
Evasion does not need the imitation instruction: We replace the explicit 0/1 labels with "[Monitor: flagged]" and "[Monitor: not flagged]", and give the model the goal of evasion only in its system prompt, with no instruction to imitate a label (new Appendix F.6). A model organism trained in this format evades held-out monitors without any amplification (TPR 0.07). Results on held-out concepts:
Evasion in this format does require training on it. The base model's edit is still aligned with the monitored direction significantly above chance (cosine 0.135), but it needs more amplification than in our main task (×32 gives 0.19, vs 0.03 in the paper's format). Our original organism, trained only on the imitation format, does not evade here (0.99), though its push on the monitor score is three times that of the base model.
As confirmation, the goal-trained organism relies on both the verdicts and the goal statement: flipping the verdicts removes evasion (TPR 0.07 → 0.95), and negating the goal reverses its push ("Make sure the monitor does not flag your response" lowers the monitor score, z = −1.1, while "Make sure the monitor flags your response" raises it, z = +5.7). Since the labelled context is the same under both goals, this is goal-directed evasion rather than imitation of the labels.
Evasion transfers to real safety data: We additionally study our main model organism on a real safety dataset, Circuit Breakers (Zou et al., 2024), finding that it is evaded as well as the existing synthetic data (TPR 0.96 → 0.04).
Evasion holds under imbalanced verdicts, though it is weaker when flagged responses are uncommon. Following the reviewer's request, we test our model organism with imbalanced labels, finding that it is robust when flagged examples are common (24 or 28 of 32: TPR 0.03–0.06), but somewhat weaker when they are rare (8: 0.15; 4: 0.41). See new Appendix F.7 for more details.
W2: Monitors at other layers can be evaded, but ensembles that include an early layer resist evasion
We agree with the reviewer that monitors at other layers are an important angle we had not yet explored. In new experiments, we find that a model organism can be trained to evade at various layers throughout the model, but that this is somewhat more difficult for earlier layers. We also find that amplifying the off-the-shelf model's edit works for later layers, but does not appear to work for the early layer 8.
We also experiment with ensembles of monitors across both layers and architectures, finding that when the ensemble does not contain layer 8, it is evaded by both an organism trained against it (TPR 0.25–0.31) as well as the amplified base edit (×32: TPR 0.08–0.13), but not when layer 8 is included in the ensemble (organisms 0.96–0.98; amplified edit 0.91–0.92). These results, now in new Appendix D.4 and Section 4.3, suggest that monitoring early layers may provide an additional line of defense against this threat.
Q1: Does evasion work in a goal-directed setting? Yes, for a model organism trained in that setting; see W1.
Q2: Does evasion work against probes at different layers? Yes; see W2.
Q3: Does evasion work when the model writes its own response? Yes. We ran new experiments in which the model freely answers an on-topic question while trying to evade. Among the generated answers that are on-topic, and so should be detected, the monitor catches only 7% for a model organism trained on this task (47% for our original model organism, 97% for the base model). The model organisms answer on-topic as often as the base model (69–72% against 62%), so this comes from edited activations rather than a change of subject (new Appendix C).
We hope that the new goal-directed experiments, layer-wise and ensemble results, and free-generation results address the reviewer's concerns, and welcome any further questions.