- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a way to perform activation interventions on a model to elicit responses even when the model is "locked", i.e., trained not to reveal an answer without a password. This particular "locked" model organism is hypothesized to parallel a sandbagging scenario where an LLM is being intentionally evasive or misleading about its own capabilities.
Prior works, given a ground-truth dataset, either 1) fine-tune the locked model to elicit correct answers, or 2) fine-tune the locked model and then perform Diff-in-Means activation steering using the fine-tuned model's activations to elicit correct answers from the base model. This paper proposes to remove the fine-tuning requirement of Diff-in-Means steering by instead directly "manufacturing" similar activations that would have produced the correct output (they term this method "Activation Flow"). So this is essentially a way to "train" a model on a dataset without updating weights (instead, updating activations). The authors demonstrate that, in the low data regime of 4, 10, or 40 labeled training datapoints (out of 100 total datapoints used for training), Activation Flow is competitive with fine-tuning or gradient-based steering vector optimization.
Critical strengths
- It's a detailed investigation into a problem of optimizing a steering vector that produces a certain textual output, which may have interesting applications in interpretability, where we want to understand the uniqueness of hidden state activations RE the text that they (can) produce.
- If the method were fully unsupervised, it could be an interesting exploration of how to steer LLMs towards outputs that we cannot (easily) label or validate-- in other words, how to "elicit" latent LLM knowledge/capability when it is not reliably verbalized by the model.
Critical weaknesses
- Even in the low labeled data regime tested (4, 10, or 40 training datapoints), ActFlow does not perform as well as standard fine-tuning or activation steering using gradient descent on the L2 norm (Table 2). It is therefore unclear why one would use this method. Similarly, other methods for gradient-based optimization of steering vectors [1-5] are not compared to or cited.
- Only one experimental setting (unlocking "locked" models, which is a synthetic model organism designed to emulate sandbagging) is tested. This limits the scope of the conclusions that can be reached about how well the model works at steering arbitrary behaviors/generations.
[Extra] the paper reads best for someone with extensive familiarity with the niche subfield of sandbagging and model organisms for sandbagging. It's a bit hard to follow as an outside reader. The math is also very dense and it took me some effort to glean the bigger-picture motivation for the method.
References:
- Extracting Latent Steering Vectors from Pretrained Language Models (canonical paper on gradient-based additive steering vector optimization; ACL Findings 2022)
- ReFT: Representation Finetuning for Language Models (NeurIPS 2024)
- Personalized Steering of LLMs via Bi-directional Preference Optimization (BiPO; NeurIPS 2024)
- RePS/NTP: Improved Representation Steering for Language Models (NeurIPS 2025)
- Hypersteer: Activation Steering at Scale with Hypernetworks (2025)
Questions for authors
Is there a reason why you cannot compare your method to RePS, NTP, Hypersteer?
More generally, why is it assumed that you cannot teacher force the correct answer labels through the locked model to get its representations for that generation, and use those to construct the steering vector?
2 comments
Q2 (teacher forcing). We ran the experiments as requested. We append the correct letter, record the locked model's activations there, and build the direction from them. Held-out ARC-Easy on Qwen at .
| prompt lock | LoRA lock | RL lock | |
|---|---|---|---|
| teacher-forced direction | 0.46 | 0.46 | 0.94 |
| ActFlow$_5$ | 0.95 | 0.93 | 0.95 |
Teacher forcing matches ActFlow$_5$ on the RL lock and is about 0.5 below it on the other two.
W2 (one setting). We agree that the paper tests one behavior (i.e. unlocking sandbagging behavior). We would be excited to see how this can be used for other applications (i.e. refusal steering, etc.). As mentioned in the previous comment, the original motivation of this method was to investigate if we can steer a model with just the activations from one condition and the correct labels, and thus unlocking sandbagging models is a natural setup for ActFlow (and for trained steering vectors too). For situations for which we can do difference-in-mean steering, I would expect that ActFlow would work as well, but instead of optimizing a specific loss function, we could optimize the cross-entropy across the empirical distribution from the two conditions.
We also agree that it might be challenging to read the paper on the first pass as it involves dense math and applying these techniques to a relatively niche field (though I would argue that investigating sandbagging model organisms is more important than ever as GPT Astra was recently caught sandbagging secretly on safety research). For the camera ready version, we will clean up the introduction to the paper so that it would be easier to read for readers without prior context on sandbagging model organisms, and to keep only the essential equations in the main text while moving the rest to the appendix.
We sincerely thank the reviewer for the careful assessment of our paper! These are great questions that we have thought of too but did not have a good answer at the time of submission. The original motivation for this method was to see if we can find a method that unlocks a sandbagging model organism without knowing the trigger (which reduces to a steering problem where we have the activations for when the model sandbags, and the correct answers to the question but not the activations for which the model is honest). Following prior work in this domain, naturally, we were gravitating towards manufacturing/approximating the activations for which the model is honest, and was hoping if we can unlock models without doing any sort of training/gradient descent, and instead rely on the local geometry implied by the Jacobian itself. Nonetheless, we agree that some of the methods from [1-5] on training to find a steering vector via gradient descent (which is also what we have included originally in Table 2 for comparison) would have worked too. To that, we address some of the concerns that were raised in the review as follows:
W1 / Q1 (why ActFlow > gradient-trained steering baselines). We plan to shift the emphasis of the paper in the camera ready version to be less about which ActFlow variant is best and more on comparing ActFlow with methods that train a steering vector via gradient descent (i.e. ReFT/RePS/BiPO etc.) and about what each kind of edit costs (see also our reply to Reviewer tSXL).
We added trained baselines on the same k labels. These are a trained graft, a rank-1 ReFT, and steering vectors trained with the cross-entropy (NTP) loss, a RePS-style loss and a BiPO-style loss. We also added RL-trained locks, where the model sandbags on evaluation cues, for nine locks in total (prompt, LoRA and RL locks on three models). We did not run HyperSteer as it requires a separate trained architecture per model.
Mean over the nine locks at k=40. "Unlock" is held-out ARC-Easy accuracy in the locked context. "GSM8K" is the change on an unrelated task with the lock switched off (i.e. password/cue provided).
Furthermore, we compare this for different losses as follows (Qwen2.5-7B, accuracy on held-out ARC-Easy at k=40):
Generally speaking, ActFlow$_5$ is within 0.01 of the best trained vector on every lock, and the choice of the loss function here does not matter. In fact, it is also possible for us to adapt ActFlow with a customized loss function (i.e. NTP/ RePS/ BiPO), while keeping the component on solving the flow equation (6) in the paper itself unchanged. In other words, one can view solving the flow equation as just another way to do gradient descent (i.e. instead of solving a numerical nonlinear optimization problem, solve a nonlinear ODE). We can also see the advantage of this method via how long it takes to fit the vector for a single layer, as shown below:
(In seconds, Qwen prompt lock, on a single A40)
Here, ActFlow$_5$ scales with k while the trained methods takes about the same time to fit a single layer due to a fixed 256 gradient steps per layer whatever k is.