acceptodds
Under review as a conference paper at ICLR 2027

Contextual Linear Activation Steering of Language Models

Abstract

Linear activation steering is a powerful approach for controlling and specializing large language models by adding linear representations of concepts to their activations at inference. While effective, existing methods often steer with a fixed strength, independent of how much steering any given input requires. In this work, we introduce Contextual Linear Activation Steering (CLAS), which decomposes activation steering into a fixed steering direction and a context-dependent steering coefficient. This decomposition preserves the interpretability of the steering direction while enabling the steering strength to vary with context. Across a wide range of steering benchmarks and language models, CLAS consistently outperforms standard linear activation steering and matches or exceeds ReFT and LoRA in settings with limited labeled data. In addition, for concept monitoring, the directions used by CLAS are superior to those learned via fine-tuning. Furthermore, CLAS enables *selective steering*: steering multiple behaviors across different portions of a single generation—a capability not possible when steering with a fixed strength. We therefore propose CLAS as an interpretable and accurate method for controlling and specializing large language models.

Primary area:
interpretability and explainable AI
Keywords:
activation steering, activation probing, learned representations, linear representation hypothesis, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mKV1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CLAS extends linear activation steering by retaining a fixed, probe-derived direction and learning an activation-dependent coefficient that varies across tokens and layers. The coefficients are trained using next-token prediction on a small set of target completions. Experiments across 11 tasks and four models demonstrate improvements over fixed-strength steering and competitive performance against LoRA and ReFT. The paper also evaluates cross-task interference, concept monitoring, and selective steering within structured outputs.

Critical strengths

The strongest contribution is the controlled evidence that contextual coefficients improve steering. The paper evaluates 11 tasks across four models and includes learned per-layer constant coefficients, higher-rank LoRA/ReFT, four-seed results, and cross-task interference measurements. These controls strengthen the comparison with fixed-strength steering.

Critical weaknesses
  1. Monitoring accuracy does not fully establish interpretability. Section 7 mostly tests whether directions distinguish explicit prompt instructions, such as “C++” versus “Python” or “answer correctly” versus “incorrectly.” This supports recognizing those prompt conditions, but does not establish reliable monitoring of actual generated behavior or explain the learned coefficients.

  2. The advantage over LoRA/ReFT has several possible causes. CLAS typically receives 200 labeled probe examples in addition to the 10 training completions available to the other methods. The experiments do not isolate the contribution of concept alignment from extra supervision, freezing one adapter factor, or reduced trainable capacity.

    1. Selective steering is promising but narrowly demonstrated. Appendix B shows opposite ends of one attribute applied to fixed JSON fields, for example, negative sentiment in one field and positive sentiment in another. It does not test independent concept combinations, unfamiliar field arrangements, or comparison with LoRA/ReFT on this setting.
Questions for authors
  1. What evidence distinguishes genuine concept monitoring from recognizing the explicit wording of task/non-task prompt templates? Do any reported results evaluate actual generated behavior or generalization to unseen instruction formulations?

  2. CLAS receives additional labeled probe data unavailable to LoRA/ReFT. What evidence separates the benefit of concept-aligned directions from extra supervision and the regularization introduced by freezing one adapter factor?

  3. Is the claim limited to applying opposite ends of one attribute within a fixed, trained JSON layout, or intended to cover more general contextual control? What evidence supports the broader interpretation?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer mKV1

We thank Reviewer mKV1 for their feedback and address their individual questions and suggestions below.

W1. Monitoring accuracy does not fully establish interpretability. Section 7 mostly tests whether directions distinguish explicit prompt instructions, such as “C++” versus “Python” or “answer correctly” versus “incorrectly.” This supports recognizing those prompt conditions, but does not establish reliable monitoring of actual generated behavior or explain the learned coefficients.

Q1. What evidence distinguishes genuine concept monitoring from recognizing the explicit wording of task/non-task prompt templates? Do any reported results evaluate actual generated behavior or generalization to unseen instruction formulations?

The probe datasets for JailbreakBench and ToxicChat do not use task/non-task prompt templates, so their two classes differ only in prompt content rather than instruction wording. For both, concept monitoring results are strong. The directions evaluated in Section 7 are the same directions used for steering in Section 6.1, which evaluates actual generated behavior. Section 6.2 further evaluates the steered models on unseen tasks and instruction formulations. Moreover, the selective steering results in Appendix B use instructions not seen in the probe data. All evaluations are on held-out test sets.

Regarding the learned coefficients, a coefficient only measures how much the activation changes along the steering direction, not where it ends up. For example, two activations can receive very different coefficients yet end up close together along the steering direction, whereas two activations that receive the same coefficient can end up far apart. To show where activations steered with CLAS end up along the steering direction, we add a new appendix (Appendix H) that tracks this across blocks, tokens and JSON fields. Since the steering direction used by CLAS reliably captures the concept, as shown in Section 7 and discussed above, where activations end up after steering can be interpreted in terms of the concept.

W2. The advantage over LoRA/ReFT has several possible causes. CLAS typically receives 200 labeled probe examples in addition to the 10 training completions available to the other methods. The experiments do not isolate the contribution of concept alignment from extra supervision, freezing one adapter factor, or reduced trainable capacity.

Q2. CLAS receives additional labeled probe data unavailable to LoRA/ReFT. What evidence separates the benefit of concept-aligned directions from extra supervision and the regularization introduced by freezing one adapter factor?

We examined the effect of increasing the number of prompt-completion pairs in Table 16, as referenced in Section 4.2. With over 10x as many prompt-completion pairs, CLAS remains competitive with LoRA and ReFT, and at the largest dataset size it is the best method on three of four models. We also note that dataset sizes are not directly comparable between the probe data and the prompt-completion pairs, since the latter carry significantly more information per example and are therefore more expensive to obtain in practice.

The steering direction used by CLAS is not used as a regularizer but is obtained using the activation steering methodology, where the direction is extracted from a probe [1–3]. Since this direction is concept-aligned, steering along it can be interpreted in terms of the concept, as described in our previous response (W1, Q1).

[1] Li et al., "Inference-Time Intervention: Eliciting Truthful Answers from a Language Model", NeurIPS 2023

[2] Rimsky et al., "Steering Llama 2 via Contrastive Activation Addition", ACL 2024

[3] Beaglehole et al., "Toward Universal Steering and Monitoring of AI Models", Science 2026

AuthorsOpenReview ↗

Author Response to Reviewer mKV1 (Continued)

W3. Selective steering is promising but narrowly demonstrated. Appendix B shows opposite ends of one attribute applied to fixed JSON fields, for example, negative sentiment in one field and positive sentiment in another. It does not test independent concept combinations, unfamiliar field arrangements, or comparison with LoRA/ReFT on this setting.

Q3. Is the claim limited to applying opposite ends of one attribute within a fixed, trained JSON layout, or intended to cover more general contextual control? What evidence supports the broader interpretation?

The claim is not limited to the JSON layout. We use the JSON layout because failures to steer selectively, or failures due to violating the JSON schema, can be evaluated automatically. The experiment tests whether a single steering vector can steer different portions of a single generation toward different behaviors, comparing context-dependent and fixed coefficients on the same steering vector. CLAS steers the two fields toward different behaviors with a single steering vector at each block, extracted from a single dataset. To our knowledge, this is the first demonstration of steering different portions of a single generation toward opposite behaviors using a single steering vector at each block. In contrast, LAS cannot achieve this with any fixed coefficient. This reveals a fundamental limitation of steering with a fixed coefficient, the primary approach to activation steering. More general contextual control and concept combinations can be achieved by using more than one steering vector at a given block. Evidence for this is shown in [1], which supplies CLAS with multiple steering vectors at each block.

[1] Gurgurov et al., "Compositional Activation Steering Across Languages and Behaviors with Contextual Gates", arXiv preprint 2026

AuthorsOpenReview ↗

Author Response to Reviewer mKV1 (Continued)

Thank you again for your review. If our responses have addressed your concerns, please consider raising your score. We are happy to answer any further questions.

Reviewer_emmoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Contextual Linear Activation Steering (CLAS), which involves (i) steering vectors built from probes trained on 50-200 prompts with binary concept labels; and (ii) block and context specific coefficients, obtained from sensing vectors trained on 10 prompt-answer pairs.

CLAS coefficients are tuned 20 to 50 times faster than linear activation steering, and it can steer towards different behaviors in a single generation.

The method is evaluated across 11 steering tasks and 4 LLMs. It is shown to outperform baselines (linear activation steering, LoRA, ReFT, DSAS) on target tasks, and remains competitive in degradation tests for tasks unrelated to the task being steered towards.

Critical strengths

S1. The method is well motivated and practical, separated into (i) steering vectors built from probes trained on 50-200 prompts with binary concept labels, and (ii) block and context specific coefficients obtained from sensing vectors trained on 10 prompt-answer pairs (as stated in the summary). They show in table 3 that the tuning speed up is anywhere between 20 and 50 times over normal linear activation steering. It makes steering cheaper to deploy.

S2. The evaluation of the method proposed is extensive and commendable both generally and specifically for steering literature. The authors use: 4 LLMs spanning 2 families and 3 sizes; competitive baselines like LoRA and ReFT; 11 steering tasks spanning different topics (code, knowledge, math, sentiment, toxicity, among others); and evaluate on both (i) target task steering effect and (ii) performance degradation on tasks unrelated to the steering target task. Great evaluations!

Critical weaknesses

W1. I think the authors are missing a few standard baselines in activation steering literature like CAA for full activation engineering, or BiPO if optimization steps are involved. (see references below)

Other: I want to note that the JailbreakBench reference in this paper has an almost fully hallucinated author list.

Questions for authors

Q1. Why do the authors choose to not run other popular activation steering baselines such as CAA (https://arxiv.org/abs/2312.06681) or BiPO (https://arxiv.org/pdf/2406.00045)? I think comparisons against these would be useful.

Q2. Can you provide some statistics on the learned coefficients? How do they vary across blocks and tokens for the toy examples? Also, for the selective steering example with the JSONs.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer emmo

We thank Reviewer emmo for their feedback and address their individual questions and suggestions below.

W1. I think the authors are missing a few standard baselines in activation steering literature like CAA for full activation engineering, or BiPO if optimization steps are involved. (see references below) Other: I want to note that the JailbreakBench reference in this paper has an almost fully hallucinated author list.

Q1. Why do the authors choose to not run other popular activation steering baselines such as CAA (https://arxiv.org/abs/2312.06681) or BiPO (https://arxiv.org/pdf/2406.00045)? I think comparisons against these would be useful.

LAS with the mean-difference steering vector corresponds to CAA [1], as cited in Section 2, and we evaluated it in Table 2. BiPO [2] learns a vector that is applied with a fixed coefficient. This is complementary to CLAS, which modulates a steering vector with a context-dependent coefficient, and Table 2 shows that CLAS improves steering regardless of the steering vector used.

Thank you for catching the JailbreakBench reference. We have corrected it and verified the references.

[1] Rimsky et al., "Steering Llama 2 via Contrastive Activation Addition", ACL 2024

[2] Cao et al., "Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization", NeurIPS 2024

Q2. Can you provide some statistics on the learned coefficients? How do they vary across blocks and tokens for the toy examples? Also, for the selective steering example with the JSONs.

We add a new appendix (Appendix H) that shows where activations steered with CLAS end up during generation. Specifically, we report the concept readout, which is where each steered activation lies along the steering direction. Since the steering direction reliably captures the concept (Section 7), this readout can be interpreted in terms of the concept. For every block, we report its median and spread, grouped by position in the generated completion and, for selective steering, by JSON field. We report this rather than the coefficients, since a coefficient only measures how much the activation changes along the steering direction, not where it ends up. For example, two activations can receive very different coefficients yet end up close together along the steering direction, whereas two activations that receive the same coefficient can end up far apart.

AuthorsOpenReview ↗

Author Response to Reviewer emmo (Continued)

Thank you again for your review. If our responses have addressed your concerns, please consider raising your score. We are happy to answer any further questions.

Thank you for your rebuttal and for adding Appendix H. I think the current score is fair so I will keep it.

Reviewer_2pDjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Contextual Linear Activation Steering, CLAS, which keeps the steering direction of linear activation steering fixed but computes the steering coefficient at each block from the current activation through a learned sensing vector. The sensing vectors are trained with next-token loss on a small set of prompt-completion pairs. Across 11 tasks and four models, the authors report gains over fixed-coefficient steering, results comparable to or better than LoRA and ReFT with limited data, better concept monitoring, and the ability to steer different parts of one generation selectively.

Critical strengths
  1. The paper targets a real limitation of linear steering, namely a single fixed coefficient shared across inputs and blocks, and the fix is simple: keep the probe direction and learn only how much of it to apply.
  2. The experiments cover 11 tasks and four models up to 70B, and CLAS clearly improves over fixed-coefficient steering while being much cheaper to tune than a grid search.
Critical weaknesses
  1. Limited novelty. CLAS adds α d to the residual stream with α = c·[h, 1], which is a rank-one affine intervention whose output direction is frozen to a probe direction. This is close to rank-one DiReFT with a fixed output vector, and the step beyond conditional steering methods such as CAST and DSAS is small.
  2. The comparison with LoRA and ReFT is not on equal footing. CLAS also uses the concept-labeled probe data, up to 200 prompts per task, while LoRA and ReFT only see the ten prompt-completion pairs, so part of the gain in this very low-data setting may come from the extra supervision.
  3. The source of the gain over LAS is not isolated. CLAS changes three things at once, namely per-block coefficients, context dependence and training with next-token loss instead of a judge-based grid search, and the main text does not separate them, so the claim that the improvement comes from context dependence is not established.
  4. The additional claims are weakly supported. Selective steering, highlighted in the abstract and Figure 1, is supported in the main text only by that single example, and the concept monitoring results compare RFM probe directions with adapter directions, which says more about RFM than about CLAS.
  5. The paper is poorly structured. The method, which is the main contribution, takes only about half a page, while the background on linear steering is repeated in the introduction, preliminaries and related work. Related work is placed between the setup and the results, the concept monitoring section appears abruptly at the end with little connection to the steering story, and Figures 2 and 3 are hard to read.
Questions for authors
  1. What is the essential difference between CLAS and a rank-one DiReFT whose output direction is fixed to the probe direction?
  2. How much of the gain over LAS remains when LAS uses per-block fixed coefficients learned with the same next-token objective?
  3. Can selective steering be evaluated quantitatively on a test set, and what data are the sensing vectors trained on in this setting?
  4. Since the sensing vectors are the only learned part of CLAS, how well do they themselves serve as concept monitors?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer 2pDj

We thank Reviewer 2pDj for their feedback and address their individual questions and suggestions below.

W1. Limited novelty. CLAS adds α d to the residual stream with α = c·[h, 1], which is a rank-one affine intervention whose output direction is frozen to a probe direction. This is close to rank-one DiReFT with a fixed output vector, and the step beyond conditional steering methods such as CAST and DSAS is small.

Q1. What is the essential difference between CLAS and a rank-one DiReFT whose output direction is fixed to the probe direction?

The essential difference lies in how the intervention is obtained rather than its form. CLAS is an activation steering method that extracts the steering direction from a probe and learns only a context-dependent coefficient, extending LAS beyond a fixed coefficient. In contrast, ReFT (DiReFT) is a fine-tuning method that learns all of its parameters from task completions. As a result, the steering direction in CLAS remains a concept direction, whereas it is unclear what concept a direction learned through backpropagation corresponds to. Empirically, ReFT generally performs worse than CLAS in the low-data setting (see our response to W2), and its directions are unreliable concept monitors (Section 7).

Regarding conditional steering, we do not believe the step beyond CAST and DSAS is small. CAST was only demonstrated to condition on the prompt, not during generation, so during generation it either steers exactly like LAS or does not steer. We also compare against DSAS in Table 2, where CLAS substantially outperforms it. More practically, as discussed in Section 5, both CAST and DSAS require grid search sweeps over the same hyperparameters as LAS plus additional ones, making them combinatorially more expensive to tune. CLAS is already over 20x faster to tune than LAS (Table 3), so its speedup over CAST and DSAS is even larger.

W2. The comparison with LoRA and ReFT is not on equal footing. CLAS also uses the concept-labeled probe data, up to 200 prompts per task, while LoRA and ReFT only see the ten prompt-completion pairs, so part of the gain in this very low-data setting may come from the extra supervision.

We examined the effect of increasing the number of prompt-completion pairs in Table 16, as referenced in Section 4.2. With over 10x as many prompt-completion pairs, CLAS remains competitive with LoRA and ReFT, and at the largest dataset size it is the best method on three of four models. Moreover, probe datasets are widely used to construct steering vectors in activation steering [1–3]. We also note that dataset sizes are not directly comparable between the probe data and the prompt-completion pairs, since the latter carry significantly more information per example and are therefore more expensive to obtain in practice.

[1] Li et al., "Inference-Time Intervention: Eliciting Truthful Answers from a Language Model", NeurIPS 2023

[2] Rimsky et al., "Steering Llama 2 via Contrastive Activation Addition", ACL 2024

[3] Beaglehole et al., "Toward Universal Steering and Monitoring of AI Models", Science 2026

W3. The source of the gain over LAS is not isolated. CLAS changes three things at once, namely per-block coefficients, context dependence and training with next-token loss instead of a judge-based grid search, and the main text does not separate them, so the claim that the improvement comes from context dependence is not established.

Q2. How much of the gain over LAS remains when LAS uses per-block fixed coefficients learned with the same next-token objective?

We evaluated this in Table 14, as referenced in Section 6.1. With the same setup and next-token objective as CLAS, LAS with per-block fixed coefficients performs worse than LAS with a single fixed grid-searched coefficient, and is therefore also worse than CLAS.

W4. The additional claims are weakly supported. Selective steering, highlighted in the abstract and Figure 1, is supported in the main text only by that single example, and the concept monitoring results compare RFM probe directions with adapter directions, which says more about RFM than about CLAS.

Q3. Can selective steering be evaluated quantitatively on a test set, and what data are the sensing vectors trained on in this setting?

We evaluated selective steering quantitatively on held-out test prompts in Appendix B, as referenced in Section 1 and Figure 1. The results show that CLAS can steer the two fields toward different behaviors, whereas LAS cannot with any fixed coefficient. The full setup, including the training data for the sensing vectors, is described in Appendix B.

Regarding concept monitoring, Section 7 shows that the directions CLAS steers along also serve as concept monitors, so steering along them can be interpreted in terms of the concept. In contrast, learned adapter directions may steer effectively but are unreliable monitors, so it is unclear what they correspond to.

AuthorsOpenReview ↗

Author Response to Reviewer 2pDj (Continued)

Q4. Since the sensing vectors are the only learned part of CLAS, how well do they themselves serve as concept monitors?

The purpose of the sensing vector is not to detect concepts but to determine how much to steer along the steering direction given the current activation. To show where activations steered with CLAS end up along the steering direction, we add a new appendix (Appendix H) that tracks this across blocks, tokens and JSON fields. Since the steering direction used by CLAS reliably captures the concept (Section 7), where activations end up after steering can be interpreted in terms of the concept.

W5. The paper is poorly structured. The method, which is the main contribution, takes only about half a page, while the background on linear steering is repeated in the introduction, preliminaries and related work. Related work is placed between the setup and the results, the concept monitoring section appears abruptly at the end with little connection to the steering story, and Figures 2 and 3 are hard to read.

Thank you for the suggestions. We will improve the structure and readability of the paper in the final version, addressing each of the points raised.

AuthorsOpenReview ↗

Author Response to Reviewer 2pDj (Continued)

Thank you again for your review. If our responses have addressed your concerns, please consider raising your score. We are happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Rebuttal

We thank all reviewers for their reviews. As the discussion period comes to a close, we briefly summarize our contributions, the strengths noted by reviewers, and the updates made during the rebuttal.

Contributions. Our work addresses the problem of how much to steer in linear activation steering. Traditionally, this is treated as a hyperparameter: a steering direction is scaled by a fixed coefficient, tuned by grid search over model generations. Even with an exhaustive sweep, however, a fixed coefficient steers every input by the same amount regardless of how much steering it requires. As a result, fixed coefficients fall short on complex tasks such as step-by-step reasoning and multilingual translation (Table 4). Moreover, sweeping is expensive and restrictive. For a given task, finding the optimal fixed coefficients would require a search that grows exponentially with the number of steering vectors. CLAS addresses these limitations by keeping the steering direction fixed and learning a sensing vector that computes an adaptive, context-dependent coefficient for each block and input. We conduct experiments across 11 diverse tasks (including reasoning, code, translation, and summarization) and four models from 1B to 70B. This is in contrast to prior work, which often evaluates steering on a narrow set of tasks or on tasks where fixed coefficients are commonly found to be effective. Our main findings are:

  • Improvement over fixed-coefficient steering. Existing steering directions become substantially more effective when modulated with context-dependent coefficients. CLAS outperforms fixed-coefficient steering for all three common steering vector extraction methods and all four models (Table 2), and is over 20x faster to tune than grid search (Table 3). In addition, Table 14 shows that the improvement in accuracy comes from context dependence.
  • Selective steering. With CLAS, a single steering vector at each block can steer different parts of one generation toward different behaviors, which is not possible with a fixed coefficient (Appendix B).
  • Improvement over prior adaptive steering. DSAS, an adaptive steering method, performs comparably to or below fixed-coefficient steering with identical steering vectors, while CLAS substantially outperforms both (Table 2). Unlike DSAS, which does not outperform both LoRA and ReFT on any model, CLAS outperforms both on three of four models (Tables 2 and 16). Prior adaptive methods also require grid search sweeps over the same hyperparameters as fixed-coefficient steering plus additional ones, making them even more expensive to tune (Section 5).
  • Interpretability. With CLAS, the same directions serve both to monitor a concept and to steer toward it. CLAS steers along directions that are reliable concept monitors, whereas the directions learned by rank-1 LoRA and ReFT are unreliable monitors (Section 7). As a result, steering with CLAS can be interpreted in terms of the concept (Appendix H), whereas it is unclear what concept LoRA and ReFT steer along.

Strengths noted by reviewers.

  • CLAS addresses a real limitation of linear steering and clearly improves over fixed-coefficient steering while being much cheaper to tune (2pDj).
  • The method is well motivated and practical, outperforms baselines on target tasks, and the evaluation is extensive and commendable both generally and for the steering literature (emmo).
  • The paper provides strong, controlled evidence that contextual coefficients improve steering, supported by ablations, seed variance, and interference measurements (mKV1).

Updates during the rebuttal. We added Appendix H, which analyzes where activations steered with CLAS end up along the steering direction across blocks, tokens, and JSON fields. This lets us interpret what CLAS does during generation in terms of the concept. We also pointed to existing results that address the main questions: the comparison with LoRA and ReFT with more training data (Table 16) and the quantitative evaluation of selective steering (Appendix B).

We sincerely thank the reviewers again for their time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.