acceptodds
Under review as a conference paper at ICLR 2027

Activation Steering Is a Data Problem

Abstract

Activation steering offers dynamic control over language model behavior at a small computational cost, by adding precomputed “steering vectors” to internal activations at inference time. Central to the success of this approach are the steering vectors extracted from internal activations, recorded as the target model processes a set of prompts, they aim to encode the desired behavioral shift. Prior work has treated optimizing steering vectors as an algorithmic problem over fixed data. In this work, we find that data construction, drawing on the full expressiveness. of natural language at its disposal, largely shapes steering vectors. We show that (1) steering vectors are better learned from data that demonstrates a behavior than from data that instructs it, (2) data quality matters more than data quantity, and (3) implicit biases can enter the extraction data and propagate to the steering vector. Together, these results show that careful data design improves steering, prevents common failures, and clarifies what makes steering vectors effective.

Primary area:
interpretability and explainable AI
Keywords:
activation steering, representation engineering, mechanistic interpretability, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iwwQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines obtaining higher quality activation steering vectors from a data perspective, focusing on language and persona steering. Compared across multiple methods of steering vector extraction, the authors find that demonstrating this feature other than instructing and data quality can give better vectors. Overall, they highlight the importance of data design when obtaining steering vectors.

Critical strengths

The question is interesting and important. The paper focuses on the data quality perspective and the design of demonstration versus instruction is quite interesting.

Critical weaknesses
  1. The assumption of better steering vectors need smaller alpha (steering strength) needs evidence.
  2. The task setup is mainly language and persona, which can consist of multiple subspaces. Maybe the story would be stronger if the subject is a cleaner feature. Considering the evaluation is mainly a yes/no question, I feel like conclusion like "a single well-constructed demonstration" is sufficient is a bit thin.
  3. Missing figure and section references in multiple places (currently shows ??).
Questions for authors

I have concern over semantic information conveyed in the paired prompts. For example, in figure2, for demonstration, the pair shows full sentences in their respective language. However, for instruction, only the word "English" or "German" differs. Thus it feels like both what the task is demanding and semantic richness differ in demonstration versus instruction, thus confounding the setup.

Reviewer_D5nCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how extraction data affects activation steering. It argues that data construction matters more than the extraction algorithm. It compares different extraction data strategy: instruction prompts (Ins), prompt demonstrations (p-demo), activation differences between a rank-1 lora fine-tune and base (m-demo), across three models (llama-3.1-8b, qwen2.5-7b, gemma-9b). Additionally, the paper argues that one pair can suffice, studies an yes/no answer bias and its correction and also claims a limit of fixed last-layer steering.

Critical strengths
  • The study covers 3 models and 8 target languages, with a pool of 135 personas (15 evaluated for gemma)
Critical weaknesses
  • main comparison is confounded - for language steering, p-demo uses target-language user sentences, while Ins uses english translation commands. Activations are collected at the assistant tag so input language and task format may explain part of the gap too.
  • section 3.1 says demonstrations outperform instructions across all settings. however, the p-demo and ins persona scores tie for rfm on all three models and for PCA on llama.
  • various results tables conflict with each other. averaging the eight llama mean-diff language rows in table 4 gies 0.11 for p-demo and 0.25 for ins. Table 2 and 5 instead report 0.08 for p-demo and they report 0.28 and 0.23 for ins, so the ins value is inconsistent in multiple ways.
  • the programming and algorithm evidence is incomplete - appendix B.3 defines p-demo for code but tables 14 and 18 compare only ins with m-demo and appendix E.13 has no llama code results.
Questions for authors
  • which llama Mean-Diff language averages are correct? how were Tables 2 and 5 computed from Table 4, given that the single-pair and M-Demo columns reconcile but p-demo and Ins do not?
  • which qwen persona scores are correct, Table 2 or Table 11?
  • why are p-demo code results and Llama code results missing?
Reviewer_ThMYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper examines the role of data vs. algorothms in computing/learning steering vectors. It makes the three interesting claims listed in the abstract, and supports them with a somewhat limited empirical evaluation.

Critical strengths

The paper asks sensible and relevant questions and makes reasonable and interesting contributions.

The writing is clear.

Critical weaknesses

This would make a strong workshop paper, but the experimental evaluation is too cursory for a conference paper. I would expect a more thorough set of experiments, e.g. with substantially more variation in the data used, given that this is the focus of the work, as well as more variation in tasks. The conclusions are also at times too strong given the limited nature of the empirical evaluation (e.g. Result 3.1).

This is something I think can be fixed, but the changes needed are substantial, hence I rate it a strong reject, but I encourage the authors to expand on the work for resubmission.

I'm also unconvinced by the approach to evaluation; the authors note issues with naive evaluation methods, but isn’t the right approach just to consider performance as multi-faceted: how well is the model steered while still maintaining overall performance according to some other metric(s)? Specifically, the metric used to evaluating complete steering seems to have the following 2 issues: 1) it depends on the magnitude of v and not just the direction, and 2) it doesn’t actually measure degredation of performance, and instead uses alpha as a proxy, but the sensitivity of model’s performance to a perturbation depends on the direction of the perturbation and not only the magnitude.

Questions for authors

Are the steering vectors injected at every timestep?

RE the "yes/no" steering result in section 3.3:

  • I would like to know if there are similar results for things other than yes/no directions.
  • The explanation provided here is intriguing, but should be validated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.