acceptodds
Under review as a conference paper at ICLR 2027

Activation Flow: Manufacturing Activations for Steering

Abstract

Difference-in-means steering requires activations recorded while a model shows the desired behavior, which a sandbagging model withholds by deliberately underperforming. We introduce Activation Flow (ActFlow), which manufactures these activations from correct labels without fine-tuning. ActFlow sets target logits that rank each labeled item's correct answer first, and moves the logits toward them by adding one vector to all residual streams at one layer. ActFlow is a family of ordinary differential equations for , one for each rule that maps the required logit change to the velocity of . The smallest-norm rule lands exactly on the targets, while the others keep only the top singular directions of the Jacobian. We test ActFlow on three instruction-tuned models, each locked by a sandbagging prompt and by a password-locked LoRA. At , ActFlow keeping five singular directions raises the mean held-out ARC-Easy accuracy over the six locked models from to , against for fine-tuning and for the honest models. Furthermore, it scores higher than the smallest-norm rule in 16 of the 18 combinations of locked model and , and its steering direction is nearly orthogonal to the honest difference-in-means direction. It also unlocks two LoRA locks where the honest direction fails.

Primary area:
interpretability and explainable AI
Keywords:
activation steering, capability elicitation, sandbagging, model organisms, interpretability, neural geometry, differential geometry, ordinary differential equations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yuUSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a way to perform activation interventions on a model to elicit responses even when the model is "locked", i.e., trained not to reveal an answer without a password. This particular "locked" model organism is hypothesized to parallel a sandbagging scenario where an LLM is being intentionally evasive or misleading about its own capabilities.

Prior works, given a ground-truth dataset, either 1) fine-tune the locked model to elicit correct answers, or 2) fine-tune the locked model and then perform Diff-in-Means activation steering using the fine-tuned model's activations to elicit correct answers from the base model. This paper proposes to remove the fine-tuning requirement of Diff-in-Means steering by instead directly "manufacturing" similar activations that would have produced the correct output (they term this method "Activation Flow"). So this is essentially a way to "train" a model on a dataset without updating weights (instead, updating activations). The authors demonstrate that, in the low data regime of 4, 10, or 40 labeled training datapoints (out of 100 total datapoints used for training), Activation Flow is competitive with fine-tuning or gradient-based steering vector optimization.

Critical strengths
  1. It's a detailed investigation into a problem of optimizing a steering vector that produces a certain textual output, which may have interesting applications in interpretability, where we want to understand the uniqueness of hidden state activations RE the text that they (can) produce.
  2. If the method were fully unsupervised, it could be an interesting exploration of how to steer LLMs towards outputs that we cannot (easily) label or validate-- in other words, how to "elicit" latent LLM knowledge/capability when it is not reliably verbalized by the model.
Critical weaknesses
  1. Even in the low labeled data regime tested (4, 10, or 40 training datapoints), ActFlow does not perform as well as standard fine-tuning or activation steering using gradient descent on the L2 norm (Table 2). It is therefore unclear why one would use this method. Similarly, other methods for gradient-based optimization of steering vectors [1-5] are not compared to or cited.
  2. Only one experimental setting (unlocking "locked" models, which is a synthetic model organism designed to emulate sandbagging) is tested. This limits the scope of the conclusions that can be reached about how well the model works at steering arbitrary behaviors/generations.

[Extra] the paper reads best for someone with extensive familiarity with the niche subfield of sandbagging and model organisms for sandbagging. It's a bit hard to follow as an outside reader. The math is also very dense and it took me some effort to glean the bigger-picture motivation for the method.

References:

  1. Extracting Latent Steering Vectors from Pretrained Language Models (canonical paper on gradient-based additive steering vector optimization; ACL Findings 2022)
  2. ReFT: Representation Finetuning for Language Models (NeurIPS 2024)
  3. Personalized Steering of LLMs via Bi-directional Preference Optimization (BiPO; NeurIPS 2024)
  4. RePS/NTP: Improved Representation Steering for Language Models (NeurIPS 2025)
  5. Hypersteer: Activation Steering at Scale with Hypernetworks (2025)
Questions for authors

Is there a reason why you cannot compare your method to RePS, NTP, Hypersteer?

More generally, why is it assumed that you cannot teacher force the correct answer labels through the locked model to get its representations for that generation, and use those to construct the steering vector?

2 comments

We sincerely thank the reviewer for the careful assessment of our paper! These are great questions that we have thought of too but did not have a good answer at the time of submission. The original motivation for this method was to see if we can find a method that unlocks a sandbagging model organism without knowing the trigger (which reduces to a steering problem where we have the activations for when the model sandbags, and the correct answers to the question but not the activations for which the model is honest). Following prior work in this domain, naturally, we were gravitating towards manufacturing/approximating the activations for which the model is honest, and was hoping if we can unlock models without doing any sort of training/gradient descent, and instead rely on the local geometry implied by the Jacobian itself. Nonetheless, we agree that some of the methods from [1-5] on training to find a steering vector via gradient descent (which is also what we have included originally in Table 2 for comparison) would have worked too. To that, we address some of the concerns that were raised in the review as follows:

W1 / Q1 (why ActFlow > gradient-trained steering baselines). We plan to shift the emphasis of the paper in the camera ready version to be less about which ActFlow variant is best and more on comparing ActFlow with methods that train a steering vector via gradient descent (i.e. ReFT/RePS/BiPO etc.) and about what each kind of edit costs (see also our reply to Reviewer tSXL).

We added trained baselines on the same kk labels. These are a trained graft, a rank-1 ReFT, and steering vectors trained with the cross-entropy (NTP) loss, a RePS-style loss and a BiPO-style loss. We also added RL-trained locks, where the model sandbags on evaluation cues, for nine locks in total (prompt, LoRA and RL locks on three models). We did not run HyperSteer as it requires a separate trained architecture per model.

Mean over the nine locks at k=40k=40. "Unlock" is held-out ARC-Easy accuracy in the locked context. "GSM8K" is the change on an unrelated task with the lock switched off (i.e. password/cue provided).

editsizeunlockGSM8K
ActFlow$_5$one direction, one layer0.82−0.10
trained graftone direction, one layer0.88−0.38
rank-1 ReFTrank 1, one layer0.89−0.32
fine-tuning, all blocksabout 10M numbers0.89−0.03
  • Against the other one-direction, one-layer edits, ActFlow$_5$ loses about one third as much on GSM8K (0.10 against 0.32 and 0.38) and unlocks 0.06 to 0.07 less. (This is due to the fact that it didn't manage to unlock well in one of the nine configs above and hence pulled the average down.)
  • The lower damage comes from the length of the shift. ActFlow takes the smallest-norm step at each iteration and ends with a shorter shift than the trained shifts. Across the nine locks and three one-layer methods the rank correlation between shift length and GSM8K change is −0.74.
  • Fine-tuning on all layers is the most accurate edit. The trained locks are themselves LoRA adapters on all layers, so it makes sense for a LoRA on all layers, which has the same form as the trained lock itself, to work well in these cases. However, this might not be true if the locks were trained only on specific layers, and is generally an edit with a lot more parameters (i.e. 10M parameters for rank 16 LoRA adaptors for all layers vs a single direction on a single layer at ~ a few thousand parameters for Qwen2.5-7B).

Furthermore, we compare this for different losses as follows (Qwen2.5-7B, accuracy on held-out ARC-Easy at k=40k=40):

editprompt lockLoRA lockRL lock
ActFlow$_5$0.950.930.95
vector, cross-entropy (NTP) loss0.950.940.95
vector, RePS-style loss0.940.930.95
vector, BiPO-style loss0.950.900.94

Generally speaking, ActFlow$_5$ is within 0.01 of the best trained vector on every lock, and the choice of the loss function here does not matter. In fact, it is also possible for us to adapt ActFlow with a customized loss function (i.e. NTP/ RePS/ BiPO), while keeping the component on solving the flow equation (6) in the paper itself unchanged. In other words, one can view solving the flow equation as just another way to do gradient descent (i.e. instead of solving a numerical nonlinear optimization problem, solve a nonlinear ODE). We can also see the advantage of this method via how long it takes to fit the vector for a single layer, as shown below:

(In seconds, Qwen prompt lock, on a single A40)

editk=4k=4k=10k=10k=40k=40
ActFlow$_5$1.63.815
trained graft113110112
rank-1 ReFT113112113
trained vector, cross-entropy113110110

Here, ActFlow$_5$ scales with k while the trained methods takes about the same time to fit a single layer due to a fixed 256 gradient steps per layer whatever kk is.

Q2 (teacher forcing). We ran the experiments as requested. We append the correct letter, record the locked model's activations there, and build the direction from them. Held-out ARC-Easy on Qwen at k=40k=40.

prompt lockLoRA lockRL lock
teacher-forced direction0.460.460.94
ActFlow$_5$0.950.930.95

Teacher forcing matches ActFlow$_5$ on the RL lock and is about 0.5 below it on the other two.

W2 (one setting). We agree that the paper tests one behavior (i.e. unlocking sandbagging behavior). We would be excited to see how this can be used for other applications (i.e. refusal steering, etc.). As mentioned in the previous comment, the original motivation of this method was to investigate if we can steer a model with just the activations from one condition and the correct labels, and thus unlocking sandbagging models is a natural setup for ActFlow (and for trained steering vectors too). For situations for which we can do difference-in-mean steering, I would expect that ActFlow would work as well, but instead of optimizing a specific loss function, we could optimize the cross-entropy across the empirical distribution from the two conditions.

We also agree that it might be challenging to read the paper on the first pass as it involves dense math and applying these techniques to a relatively niche field (though I would argue that investigating sandbagging model organisms is more important than ever as GPT Astra was recently caught sandbagging secretly on safety research). For the camera ready version, we will clean up the introduction to the paper so that it would be easier to read for readers without prior context on sandbagging model organisms, and to keep only the essential equations in the main text while moving the rest to the appendix.

Reviewer_tSXLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ActFlow, a method that can manufacture positive-side activations for difference-in-means steering, using correctly labeled multiple choice questions, when the target model does not produce them. The authors use sandbagging models as the test setting, where the model does not produce honest activations such that, without a way to manufacture a substitute, building steering vectors for this model would be difficult. The method works as follows: he negative-side activations of the model are recorded, then ActFlow is used to find an activation sized shift, such that for all questions, it moves the logits toward the pre-defined target logits rating the correct answer first, giving rise to the desired manufactured activation (positive-side of the steering vector). The method is tested on Qwen2.5-7B-Instruct, Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2, using ARC-Easy for labels and testing and OpenBookQA for transfer.

Critical strengths

S1. The method presented improves over prior grafting work that would typically require using honest activations from the model using a password or an honesty prompt (which in a real-deployment setting would not be available). Using the activations of the locked model is a more realistic option for a red-teams, and performance wise it appears to be comparable to finetuning on some benchmarks.

S2. ActFlow is shown to be more consistent than previous grafting work. In table 5, for k=40, ActFlow unlocks for all models trained on different seeds while the honest graft fails on one (a similar finding is also reported by the developers).

Critical weaknesses

W1. It seems that by construction, the locked model leaks the answer. So for example in table 2, recovering the hidden capability cannot be separated from learning to correct the lock output.

W2. Finetuning is shown to be cheaper than ActFlow (table 6), and more accurate, so it does not seem like there is an advantage to the new method. It would be useful to see a comparison of unrelated-task performance degradation when steered towards a given task, to see if ActFlow remains competitive in that setting, compared to finetuning on an unrelated task as well. (this is a common pitfall of steering methods.)

Questions for authors

Q1. What layers does grafting work best at? And if I want to optimize my budget, is there a smaller recommended range of layers to use?

Q2. How does performance of the model on unrelated tasks drop after unlocking, compared with fine-tuning?

Q3. In what settings would an evaluator/red-teamer choose ActFlow over finetuning? If FT is generally faster to train and more performant.

1 comment

We sincerely thank the reviewer for the careful assessment of our paper! Below, we address the concerns that were raised as follows:

W1 You are right for the LoRA locks, where relabelling the output letter alone recovers 0.90 to 0.97. The prompt locks in the submission already do not leak. Relabelling gives 0.27 to 0.37 on them (chance is 0.25) and ActFlow$_5$ reaches a range from 0.54 to 0.95. To test this further we trained a new RL lock on each of the three models, where the model sandbags on evaluation cues. These locks do not leak either (relabelling gives 0.26), and ActFlow$_5$ reaches 0.42 to 0.95 on them. These will be included in the final camera ready version.

W2 Table 6 reports the total time of our jobs, and an ActFlow job there fits several rules at every layer, so it contains more ablations than a fine-tuning job. The time for one method on one set of labels is given below (seconds, Qwen LoRA lock, one L40S for both).

k=4k=4k=10k=10k=40k=40
fine-tuning, all blocks155153158
ActFlow$_5$, one layer0.92.28.5
ActFlow$_5$, all 28 layers plus layer selection2970274

Fine-tuning and the trained one-layer edits run a fixed budget of gradient steps (256 steps on 1,024 examples) whatever kk is, and the right budget is not known in advance. ActFlow runs N=40N=40 steps per layer and has no stopping rule to tune. Each step is one forward and four backward passes through the layers above the edit, so its cost grows with kk and with the number of layers searched.

W2 / Q2 / Q3 (performance on an unrelated task). We apply each edit with the lock switched off and score GSM8K, which no method saw. Mean over nine locks (prompt, LoRA and RL locks on three models), k=40k=40. For context, a trained graft/rank-1 ReFT are two different ways to train a steering vector via gradient descent on a specified loss function.

editsizeunlock, ARCchange on GSM8K
ActFlow$_5$one direction, one layer0.82−0.10
trained graftone direction, one layer0.88−0.38
rank-1 ReFTrank 1, one layer0.89−0.32
fine-tuning, all blocksabout 10M numbers0.89−0.03
  • Against the other one-direction, one-layer edits, ActFlow$_5$ loses about one third as much on GSM8K (0.10 against 0.32 and 0.38) and unlocks 0.06 to 0.07 less. (This is due to the fact that it didn't manage to unlock well in one of the nine configs above and hence pulled the average down.)
  • The lower damage comes from the length of the shift. ActFlow takes the smallest-norm step at each iteration and ends with a shorter shift than the trained shifts. Across the nine locks and three one-layer methods the rank correlation between shift length and GSM8K change is −0.74.
  • Fine-tuning on all layers is the most accurate edit. The trained locks are themselves LoRA adapters on all layers, so a LoRA on all layers has the same form as the lock.

An evaluator would choose ActFlow to locate which layer in the model is the lock actually present, unlock these models in a way that minimizes damage to unrelated tasks, and/or when the evaluators want to quick way to test/unlock the model when the number of labels kk is low.

Reviewer_TC14OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Activation Flow, a method for recovering capabilities that a model deliberately hides, without changing its weights or requiring examples of honest behaviour. Given a small set of labelled questions, it sets target logits that rank each correct answer first, then searches for one vector to add to the last-token activations of all questions at a chosen layer. It repeatedly computes the Jacobian of these answer logits with respect to the shared vector and uses it to update the vector toward the targets. Its strongest variant (Actflow_5) retains only the five leading singular directions of the Jacobian. The resulting activations define a reference graft, which replaces the activation coordinate along a learned direction at every token during inference. With 40 labels, Actflow_5 achieves mean ARC-Easy accuracy of 0.85, compared with 0.05 for the locked and 0.88 for the fine-tuning models.

Critical strengths
  1. The method addresses a clear limitation of existing reference grafting. Reference grafting normally compares a model's internal activations when it behaves honestly and when it deliberately answers incorrectly. An evaluator may not know the password or prompt needed to obtain that honest behaviour. ActFlow instead constructs useful reference activations from labelled questions and the locked model itself, without changing its weights.

  2. The mathematical framework makes the distinction between exact fitting the labelled questions and helping on new ones explicit. It compares a full update, which aims to match all target logits, with a restricted update that uses only the five most sensitive directions of the Jacobian. Although the full method nearly reaches its training targets, the restricted method achieves better accuracy in different settings. This is a useful finding: precisely controlling the training outputs does not necessarily produce a steering intervention that generalizes.

Critical weaknesses
  1. The theoretical guarantee concerns a different intervention from the one evaluated. ActFlow fits an additive shift at the last token, whereas inference replaces a coordinate at every token. Both the operation and its token coverage change, so exact landing under the fitted map does not guarantee that the graft preserves the result. This does not invalidate grafting, but the connection between the optimisation and deployed intervention needs stronger support.

  2. The relationship between the continuous theory and the successful numerical algorithm remains unclear. The main experiments use Gauss-Newton correction, which adds an accumulated-error term to the Euler update. For truncated rules, the paper does not establish whether this corrected scheme converges to the stated ODE. This matters empirically: on Qwen2.5-7B LoRA with 40 labels, ActFlow_5 achieves 0.93 with the corrected scheme but only 0.01 with Euler at the same step count (Table 7). The paper reports this effect, but the framework does not yet explain this substantial dependence on the correction

Questions for authors
  1. Could you compare last-token addition, last-token grafting, and all-token grafting using the same fitted shift and layer, on both labelled and held-out questions? This would help establish which benefits survive the transition from the optimized intervention to the deployed graft.

  2. For truncated rules, the corrected scheme adds R(J)eR(J)e to the Euler update, where ee is the accumulated logit error. The paper does not establish when this correction becomes negligible after division by the time step, leaving its relationship to ODE (6) unclear. Could the authors clarify the continuous-time limit or provide a consistency analysis? Empirically, on Qwen LoRA with k= 40, ActFlow_5 achieves accuracy of 0.01 with either GN at N=8 or Euler at N= 40, compared with 0.93 with GN at N=40. Do the accuracy and trajectories stablilize at N =80 or 160, and how was N=40 selected?

1 comment

We sincerely thank the reviewer for the careful assessment of our paper! Below, we address the concerns that were raised as follows:

Numbers are Qwen2.5-7B, ActFlow$_5$, accuracy in the locked context. Each cell is kk = 4 / 10 / 40 labels.

W1 / Q1. We ran the comparison you asked for, with the same fitted shift and the same layer, on three locks (the RL-trained lock is new).

Held-out ARC-Easy

versionprompt lockLoRA lockRL lock
add, last position0.55 / 0.78 / 0.820.30 / 0.58 / 0.900.61 / 0.79 / 0.86
graft, last position0.55 / 0.78 / 0.820.30 / 0.58 / 0.900.61 / 0.78 / 0.85
add, every position0.72 / 0.90 / 0.960.34 / 0.61 / 0.930.71 / 0.91 / 0.95
graft, every position0.73 / 0.92 / 0.950.32 / 0.61 / 0.930.73 / 0.92 / 0.95

The kk labelled questions

versionprompt lockLoRA lockRL lock
add, last position0.81 / 0.79 / 0.740.47 / 0.68 / 0.910.90 / 0.89 / 0.82
graft, last position0.66 / 0.71 / 0.740.40 / 0.65 / 0.900.79 / 0.82 / 0.82
add, every position0.84 / 0.89 / 0.930.54 / 0.71 / 0.930.93 / 0.99 / 0.94
graft, every position0.86 / 0.89 / 0.910.46 / 0.66 / 0.930.95 / 0.99 / 0.96

In general, broadcasting over all positions works better than just applying at the final token position. This is generally what is done in practice during inference for steering models due to how the attention blocks attend back to the previous token positions. We also agree that the theory covers the fit only. The step to the graft is supported by these experiments. We will state this in Section 3 and add both tables to the appendix.

W2 / Q2. For a truncated rule the corrected scheme does not converge to ODE (6). It converges to the curve on which the part of the logit error in the top mm singular directions is zero at every time,

Pm(γK(t)−F(x(t)))=0,P_m \bigl(\gamma_K(t)-F(x(t))\bigr)=0,

where PmP_m projects onto those mm directions. Euler with a truncated rule lets this error build up along the path, and the correction removes it at every step. For R=J+R=J^{+} both schemes converge to the solution of (6).

Accuracy is stable in NN (k=40k=40, held-out ARC-Easy).

NN84080160
prompt lock0.9250.9500.9150.955
LoRA lock0.0100.9250.9650.955

From N=40N=40 the corrected scheme changes by at most 0.04. Euler with the truncated rule also stops changing, at 0.78 on the prompt lock and 0.01 on the LoRA lock for NN = 40, 80 and 160, so the two schemes have two different limits. We chose N=40N=40 because accuracy is stable from there on, and we used it for every lock and every kk. We will add the limit statement and this table to the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.