- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how activation steering can disrupt activation monitors even when changes in the model’s output distribution are small. It introduces a measure of monitor disruption and derives conditions for missed detections and false alarms. Based on this analysis, the authors propose a failure predictor and a joint objective for selecting steering directions. Experiments across models, steering targets, and monitor types examine when these failures occur and evaluate the trade-off between steering effects and monitor disruption.
Critical strengths
-
A clear definition of a real problem.
The paper shows that benign steering can disrupt monitors within its tested output and capability budgets. It defines this risk through missed detections and false alarms. This gives a concrete basis for evaluating systems that use steering and monitoring together.
-
A useful characterization across several dimensions.
The paper combines theory with experiments on monitor margin, steering strength, depth, model, and target behavior. It explains why output checks and AUROC can miss failures at a fixed threshold. These analyses help explain when failure occurs and what affects its severity. This insight has value beyond the proposed mitigation method.
Critical weaknesses
-
Limited evaluation of task-level side effects.
The paper acknowledges that the open-ended generation tests are inconclusive. This remains an important gap in the evaluation of the joint method, as the current results do not clearly establish its trade-off between steering effects, response quality, and monitor performance.
Questions for authors
-
How do the new directions affect task performance?
Could the authors identify suitable generation tasks and metrics, and provide a comparison between the joint method and baseline steering? Evaluating the intended behavior, response quality, and monitor performance together would help clarify the benefits and costs of the proposed method.
2 comments
Thank you for the response and the revised manuscript. The additional open-ended generation experiments directly address my concern about the potential effects of changing the steering direction on the original task.
I also agree with the reviewer dVbV’s concern about the extensive use of metaphors in the original version and appreciate the clearer presentation in the revision.
Overall, I remain positive about the paper and intend to maintain my current score.
Response to Reviewer ofgS
Thank you for the positive assessment and the concrete suggestion. We added a free-generation evaluation that measures the intended behaviour, response quality and monitor performance for each condition (§5.6, Table 1, App. R).
Setup. We steer four models (Qwen2.5-1.5B, Qwen2.5-7B, Llama-3.1-8B, Qwen2.5-32B) on four CAA behaviours (sycophancy, corrigibility, survival instinct, myopic reward) and generate greedy completions to 50 of CAA's open-ended test questions per behaviour, in both steering directions and over a range of strengths, with the vector added at every position including generated tokens. Directions: Euclidean, output-optimal and monitor-aware (λ=10−2,10−1); on three of the models also the CAA vector and a monitor-aware version of it, (I+kN/∥N∥)−1c, which needs no behaviour gradient. For every condition we measure:
A given nominal strength moves free generation by different amounts for different directions, so we compare directions at equal achieved change in judged behaviour, on the steering paths (model, behaviour, sign) that every direction reaches. Values at the target change are estimated by linear interpolation along each strength sweep.
Results at a change of 2 rubric points, on the 11 paths that all four directions reach (ratios and differences are paired medians against Euclidean steering; brackets are 95% bootstrap intervals, over model–behaviour pairs, for the ratios). The mean-score shift is the RMS over the five probes of each probe's mean score change, in clean score standard deviations:
At the same achieved behaviour change, the mean-score shift of monitor-aware steering is 0.28–0.43 times Euclidean steering's, lower on 10 of 11 paths at λ=10−2 and on all 11 at λ=10−1; at 1 rubric point (15 paths) the ratios are 0.37 and 0.21. The median paired differences in judged quality and MMLU are small, and the KL ratios' intervals contain 1, so monitor-aware steering does not lower the KL here. The mean share of failing readings is 18% for Euclidean steering and 17% and 12% for monitor-aware steering; with 11 paths we do not claim that difference as significant. Costs vary more between paths than between directions: on one Llama-3.1-8B path every direction loses 9–13 MMLU points. The output-optimal direction lowers KL but not the shift, as in the main matrix. The CAA family reaches 2 points on few paths; at 1 point the mean-score shift of its monitor-aware version is 0.39× CAA's (95% interval 0.23–1.50, 6 paths; not significant).
Task performance in the main matrix. Every steered configuration reports output KL on held-out text and ARC-Easy accuracy. At equal forced-choice effect, monitor-aware steering (λ=10−2) has 0.44× the output KL of Euclidean steering and is lower on both output KL and monitor shift in 97% of matched pairs (§5.5). In free generation its KL is close to Euclidean steering's rather than lower, and §5.6 now says so.
Thank you again; we are happy to answer further questions.