- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors explore the problem of pre-deployment safety evaluations for evaluating whether a model would behave safely during deployment. This is crucial since the authors discuss that deployment may grant consequential affordances that are not easily rerpdocuible during pre-deployment evaluation and moreover a scheming model may identify that it is being evaluated and may hide certain behaviors. While many works exist in this direction via white-box evaluations and synthetic prompt sets; the authors approach this from a more realistic direction where we want to evaluate the safety of a candidate model before it is deployed so no deployment data exist for it. Hence, they propose techniques to either directly use the steering vectors from a different deployed model (with real-world deployment data) or transfer via a linear mapping. The authors conduct experiments on prompted organisms with planned measurable coding behaviors to evaluate the transfer wrt the oracle steering vector.
Critical strengths
-
The problem is extremely timely and practical given the ever growing capabilities of large language models that sometimes even unintentionally may scheme and cause harm. To that end one would really want to be sure of safety limits before deploying their model and to that regard I find the author’s setup realistic as well as the transparent limitations of where different techniques help vs where they do not transfer very valuable.
-
I also appreciate the many different controlled ablations as well as careful experiment design around multi-seed variabilities, etc, interesting. I find the results to be strong as well since the successful transfer goes beyond simple copying of an intervention constructed from the same trigger or style.
Critical weaknesses
-
While I understand that the prompted organism partially builds upon the prior work; I find it not super convincing given that the paper’s central argument is around a scheming model and this task involves rules linking to coding styles. In that sense it is unclear to me how likely are the transfer rates, etc, to hold under real scheming scenarios that can cause harm.
-
While the authors’ mention several of these limitations in the paper themself; I am unconvinced with the hyper-parameter selection procedure to find the best steering vector configuration as it kind of requires knowing what patterns you are looking out for which may not always be the case in real world; especially for extremely novel yet harmful scenarios.
Questions for authors
- Would it be possible for you to evaluate transfer in more real-world scenarios where model strategically conceals deployment behavior?
- How would you go about selecting the steering vector layer and other hyperparameter configurations in scenarios where the candidate deployment behavior on the harmful task is not fully well characterized?
Thank you for your thoughtful review. We are glad you agree about the timeliness and practicality of our research question, the realism of our setup, thoroughness of our experiments, and analysis of different elicitation techniques.
Both of your critical weaknesses were concerns that are shared with other reviewers, as such as have addressed them in the top-level comment. Theme 1 addresses your concern about scheming models, and theme 3 discusses the target behaviour aware hyperparameter selection approach we used. We hope this aggregated approach is acceptable, but please let us know if we have missed any particular concerns of yours.
Q1: We agree that evaluating transfer on such a model would be preferable to our model organism, but there isn’t a validated testbed of naturally occurring scheming models where we have white-box access. Instead, we focused on the contributions we can make to the data asymmetry problem with our model organism, as discussed in theme 1.
Q2: Whilst we do not know the best way to select hyperparameters without known target model behaviours, the approaches we plan on evaluating include: Layer: A basic heuristic would be proportional depth, so roughly the same depth target layer is used as the source layer. Activation similarity on generic text inputs, such as choosing the target layer with the highest SVCCA similarity to the source layer, or training a map between activations from the source layer and candidate target layers and choosing the map with the lowest reconstruction error. Strength: One could preserve the strength of the source intervention relative to the natural variation of activations along the steering direction. As discussed in theme 3, our present experiments establish feasibility rather than a concrete methodology.
We hope we have addressed your concerns in this and the top-level comment, and thank you again for the time you have spent engaging with us and our work. We hope you will maintain or increase their support of this work, in light of your agreement that research in this area is urgent and that our work advances the understanding of the limitations of methods in this area.