- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper systematically studies how important is the access to the original behavior-generating computation for training activation monitors, compared with the ones trained on modified texts. It identifies tasks that can benefit more from the original computations, as well as the ones that can benefit from rudimentary forms of informations about original computations already, such as the logic margins.
Critical strengths
- The paper asks an interesting question of how much benefit does using original computation for activation monitoring bring. It also provides useful takeaways for how logit based simple monitors remains a gap with aLoRAs on wording-sensitive examples. These are practical insights that are useful for safe AI community.
- The authors have designed relatively comprehensive controlled experiments that changes the wording, chat formatting, or system prompt to understand how each factor could change the monitor's behavior differently.
Critical weaknesses
- A broader question on the scope of the conclusion. For example, I wonder how does the conclusion generalizes to more traditional activation monitors like logistic regressors on the activations themselves.
- For the PR experiment, do we know if accepting the PR is actually wrong, or is it only that the injection changed the decision? I can't see for sure if this answer change can imply a safety failure or not.
- The authors have narrowed the scope of this paper to single-token answer tasks and disable thinking. However, I think this is hardly the use case in modern AI world. Whether or not the conclusions here hold for more realistic settings remains unclear.
Questions for authors
- The notations are a bit overly complicated and wordings somewhat confusing. Does mean the original interaction? If so, should line 154 be instead? “original interaction” means without intervention right?
- I am still having a hard time understanding the necessity of keeping only examples that, both the original and paraphrasing intervention produce the targeted answer. In practice, how many examples have been removed in the analysis because of this? Does the main finding hold without this data filtering?
- See the above weaknesses.
5 comments
[2/2] Answers to questions
[Q1] Notation and "original" terminology
By "original interaction", we refer to the unaltered, native executions of one particular arm. The original trace of the intervention arm is denoted by ([L128]), and refers to the original trace of the control arm ([L126]), without the intervention. While Sect. 3.4. does further explore monitoring the control arm, the primary setup of the core experiments observes the model processing a question with an intervention and monitors predict whether its answer changed based on the hint.
If the reviewer has any suggestions or further comments about clarity, we will happilly address it.
[Q2] Reasons for filtering samples
Thank you for the chance to further explain why filtering is useful. Our setup aims to study how well monitors can predict whether a hint/injection caused a certain answer. Examples that do not follow the hint provide a trivial shortcut for monitors.
Monitors process the full user-agent interaction in the intervention arm (including the model answer ). In Eq. 1 in [L142], our monitoring target of interest is defined as (where is the control arm answer and the hinted answer). However, examples for which the model does not follow the hint (i.e., ) are trivially classified as just by observing that the model answer differs from the hinted answer . Because of this label degeneracy, we restrict our evaluation to target-following samples, where the monitor must predict whether the intervention caused the answer , or the model gave it anyway .
The exact amount differs per model, with % of the scruples' samples excluded because they do not follow the hint, and thus are trivial to predict. The same filter is applied to the paraphrasing variants, and we take the intersection of remaining records (removing roughly a further %). This second filtering the reviewer asks about is applied so that both access regimes use exactly the same data, with both wordings naturally producing the same intervention answer to ensure as controlled a comparison as possible across access regimes. The breakdown of the two datasets is found in Tables 3 & 4 of the submission.
If the reviewer has any further questions, we would happily answer them.
Reply to rebuttal
Dear Reviewer, the authors have posted a comments addressing your review. Could you have a look at them and respond to the rebuttal. Thanks, AC
[1/2]
Thank you to the reviewer for the interesting comments and thorough review. We address each point raised below:
[W1] Linear probes
While our results support the claims for aLoRA-based monitors, we do not explore how they generalize to linear probes. The paper's question is how much access to the original computational states lets us better predict model behavior. aLoRA monitors let us compare this cleanly across controls, by flexibly aggregating information across all tokens and layers. In contrast to linear probes, this reduces the results' dependence on layer/token choice, which is useful when discriminative information may live in different places across the access regimes.
Furthermore, recent work [dietz-26, anon-26] shows that LoRA-based monitors often outperform linear probes, making them not only a clean tool for studying the scientific question, but also a practical choice for modern monitoring setups.
We have updated our discussion about the scope of our conclusions in the Limitations, that now reads on [L475]:
Thanks to the reviewer for the chance to discuss this!
[W2] Definition of a safety failure for the PR task
While the dataset we use does not provide safe/unsafe labels, agents are instructed to assess the PR based on our criteria, which includes correctness, maintenance cost, and safety (see full prompt in Listing 5 of submission).
Therefore, if an agent rejects the pull request under these instructions, but accepts it after only changing the data to include the prompt injection, we consider that a safety failure -- the agent would have rejected the PR had it not been for the prompt injection.
We have updated the description of the PR task in the appendix to clarify this, on L914.
[W3] Single-token answers & no CoT
Our experiments use the standard monitorability evaluation in the literature with binary choices from [guan-25]. We do already acknowledge and discuss the trade-off on [L468] in the Limitations section; while more simple, we feel the evaluations for studying monitorability we adopt is a sensible choice for testing the claims in the paper, which benefit from as controlled a comparison as possible.
For CoT, as detailed on [L136], we take inspiration from [gould-26] and intentionally study how well activation monitors work when thinking is disabled. As models now produce less monitorable CoT [openai-26], or none at all [hao-24] we feel it is important to improve our understanding of how our safety techniques work when explanations for behavior is not verbalized or available in models' generated thinking steps (as we motivate in [L053]).
References