acceptodds
Under review as a conference paper at ICLR 2027

Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods

22 views

Abstract

Activation steering provides a lightweight and flexible way to control large language model (LLM) behavior. However, effective steering requires more than inducing the intended behavior: it should also limit unintended changes and remain robust across inputs and training data. Existing evaluations cover these dimensions only in fragments. As a result, the trade-offs between efficacy and side effects have not been systematically characterized. We introduce SteerScope, a two-axis, multi-dimensional evaluation suite that jointly characterizes steering outcomes and method properties through 15 metrics. We score target efficacy and side effects on language quality, task capabilities, and safety and reliability, and further assess generalization and data dependence through steering-specific metrics for sample efficiency and sample sensitivity. Rather than comparing methods at a single operating point, we characterize their efficacy–side-effect trade-offs. Under matched models, tasks, and evaluation protocols, we benchmark 23 methods spanning four families, including prompting, LoRA, and SFT as baseline methods, and release the suite as an extensible codebase. We find that current activation steering methods do not yet surpass the prompting baseline in their overall balance between steering efficacy and side effects: across both model scales, no evaluated activation steering method achieves higher efficacy without incurring greater composite side effects. We further uncover a consistent coupling between steering efficacy and side effects. Under out-of-distribution (OOD) prompts, target efficacy is often preserved, whereas side effects tend to become more pronounced, particularly through declines in instruction relevance and fluency. Methods also exhibit sharply different sample-efficiency profiles.

Primary area:
datasets and benchmarks
Keywords:
large language models, activation steering, model evaluation, benchmark, side effects, steering vector

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9NsYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SteerScope, a multidimensional evaluation suite for LLM steering that jointly assesses target efficacy, unintended side effects, cross-lingual generalization, and dependence on training-data size and composition through 15 metrics. The authors evaluate 23 methods spanning activation steering, prompting, and parameter fine-tuning on Gemma-2-2B-it and Gemma-2-9B-it, using steering-strength sweeps to characterize efficacy–side-effect trade-offs. Under the evaluated configurations and default aggregation, no activation steering method achieves both higher efficacy and lower composite side effects than Prompt Steering. The experiments further show that stronger steering generally incurs greater side effects, cross-lingual shifts can preserve target efficacy while worsening response quality, and methods differ substantially in their data dependence.

Critical strengths
  1. The paper addresses a meaningful gap in steering evaluation: achieving the target behavior is only useful if the model remains capable and reliable. Bringing capability, safety, and response-quality measurements into the same evaluation makes it possible to see costs that existing steering scores can miss. This is the strongest reason I see value in the benchmark.

  2. The strength sweeps make the comparison more informative than a ranking at a single operating point. Evaluating a broad set of methods this way shows where stronger control comes at the expense of other behaviors. The metric-weighting analysis also lends support to the finding that Prompt Steering remains a competitive baseline, rather than leaving that conclusion entirely dependent on equal weighting.

Critical weaknesses

My main concern is that the breadth of the method comparison is not matched by the breadth of the model evaluation. Both backbones belong to the Gemma-2 family, and single-site interventions are fixed at layer 20, which corresponds to quite different relative depths in the two models. This provides a controlled comparison, but leaves open whether the observed trade-offs reflect the methods themselves or their suitability for these particular configurations. Since a central finding concerns activation steering’s competitiveness with Prompt Steering, a small validation on another model family, with reasonable intervention-layer selection, would substantially strengthen the evidence. As it stands, I find the benchmark useful, but the scope of its empirical conclusions limited.

Questions for authors
  1. Clarification: What motivated fixing single-site interventions at layer 20 in both models, and what evidence suggests this choice does not disadvantage particular methods?
  2. Additional validation: Can you test Prompt Steering and a few representative activation steering methods on another model family, with intervention layers selected on held-out data, to assess whether the main trade-off finding holds?
4 comments

Thank you for recognizing the value of SteerScope's joint evaluation of concept control, model capabilities, safety, and response quality, as well as its characterization of trade-offs through steering-strength sweeps. We agree that extending evaluation to other model families and examining the effects of intervention layer selection help clarify the scope of the main findings. Below, we explain the rationale for using L20 and a shared intervention layer, and report evaluations at different Gemma layers and a comparison of fixed-layer and per-method layer selection on Qwen3-8B.

Q1: Why use L20 in both models? Could this disadvantage some methods?

This question concerns two choices: using L20 in both Gemma models and sharing an intervention layer across methods. To examine the former, we added evaluations on Gemma-2-2B at L10 and Gemma-2-9B at L31 to study method performance at different relative depths. To examine the latter, we compare a fixed L14 with per-method layer selection on Qwen3-8B to assess whether method rankings and the main trade-off finding change.

(a) Rationale for L20 and evaluations at different intervention layers.

Within each evaluation configuration, AxBench [1] fixes a common intervention layer for different single-layer intervention methods. It evaluates four configurations: L10 and L20 of Gemma-2-2B, and L20 and L31 of Gemma-2-9B. Among these configurations, works including HyperSteer [2], Steer Like the LLM [3], FLAS [4], EasyEdit2 [5], and SteerEval [6] use L20 in their respective main Gemma evaluations. We follow this configuration for comparability with prior work. In addition, SteerScope covers many methods, full steering-strength grids, and multiple evaluation dimensions; extending this to an exhaustive layer sweep for every method would substantially increase computational cost. We therefore use a common configuration for the main experiments.

We agree that L20 corresponds to different relative depths in the two models. To examine method performance at different relative depths, we added experiments on Gemma-2-2B at L10 and Gemma-2-9B at L31. Each configuration covers 20 methods plus a Random control on 10 concepts, with each method's full steering-factor grid. Both configurations support the paper's main observation: no evaluated activation steering method achieves both a larger gain in Concept Expression and a lower equally weighted composite side-effect score than Prompt Steering. Complete results have been added to Appendix D.10.1 (Alternative Intervention Layers), on page 62 of the revised manuscript.

(b) Rationale for a shared intervention layer and the per-method layer-selection experiment.

Sharing an intervention layer is a widely adopted evaluation setup in steering research. Examples include method studies such as ODESteer [7] and A Unified Understanding and Evaluation of Steering Methods [8], benchmarks such as AxBench [1], SteerEval [6], and FaithSteer-BENCH [9], and the AxBench evaluation in EasyEdit2 [5]. Following this controlled comparison approach, our choice reflects three considerations:

(1) Consistent comparison conditions.

Comparing methods at a common representation site controls for the effect of intervention location and focuses the comparison on the intervention methods themselves; steering strengths are still swept separately for each method.

(2) Alignment between SAE features and concept data.

SAE feature dictionaries are layer-specific. Extracting new features at different layers to define evaluation concepts would also change the concept set. We therefore maintain consistency between the intervention layer, feature dictionary, and concept data to compare methods on the same evaluation targets.

(3) Balancing evaluation coverage and computational budget.

In an evaluation covering many methods, full strength grids, and multiple metrics, sharing a layer allows us to allocate the computational budget to broader evaluation coverage.

To examine how a shared layer affects different methods, we conducted a per-method layer-selection experiment on Qwen3-8B with separate validation data and compared it with the shared-layer setting, as detailed in Q2.

Q2: Can the main trade-off finding be tested on another model family with layers selected on separate validation data?

(a) Validation and evaluation protocol.

On Qwen3-8B, we include 11 methods plus a Random control. Six methods undergo per-method layer selection: S-PSR, RePS, ReFT-r1, DiffMean, Linear Probe, and ODESteer. Random follows the same layer-selection protocol to compare concept-related interventions with random perturbations. The other five methods are Prompt Steering, Simple Prompt Steering, A-PSR, LoRA, and LoReFT.

We select each method's intervention layer on a validation set, with L8, L14, L20, and L28 as candidates. At each candidate layer, we sweep the method's 14 nonzero steering strengths and select the layer based on Overall. Specifically, we first take the highest mean validation Overall score across the strength grid at each layer, then select the layer with the highest score.

The shared layer is also determined on the validation set. At each candidate layer, we average the best mean Overall scores across the methods participating in layer selection and select L14, which has the highest average, as the shared intervention layer.

After layer selection, we evaluate the full strength grids on a separate evaluation set covering 10 concepts, comparing efficacy–side-effect trade-offs under shared L14 and per-method layer selection. The other five methods retain identical configurations in both settings.

(b) Per-method layer-selection results.

The table below reports each method's validation performance at the four candidate layers. Each cell reports the highest mean Overall score attained by the method over its strength grid at that layer. The selected layer is bolded. Methods are ordered by their validation Overall score at the selected layer, from highest to lowest.

MethodL8L14L20L28
S-PSR0.8381.0600.7260.074
RePS0.8340.8740.7980.490
ReFT-r10.2820.5420.2200.384
DiffMean0.2400.2160.2880.234
Linear Probe0.1560.1280.0600.024
ODESteer0.0240.0720.0960.048
Random0.0300.0200.0240.024

S-PSR, RePS, and ReFT-r1, which have higher validation Overall scores, all select L14. This may suggest that the advantage of an intervention layer can generalize to some extent across methods.

(c) Trade-offs under shared-layer and per-method layer selection.

The table below compares evaluation results under shared L14 and per-method layer selection. For each setting, we report the steering factor with the highest mean Overall score and its corresponding Overall, Concept Gain, and Composite scores. Methods are ordered by Overall under the shared-layer setting, from highest to lowest. Entries for methods that do not undergo per-method layer selection are marked with “—”.

MethodShared factorShared Overall ↑Shared Concept Gain ↑Shared Composite ↓Per-method layerPer-method factorPer-method Overall ↑Per-method Concept Gain ↑Per-method Composite ↓
A-PSR21.2331.3100.0731—————
Simple Prompt Steering11.1691.4200.0401—————
S-PSR31.1251.1900.1243L1431.1251.1900.1243
Prompt Steering11.0951.0700.0241—————
RePS7.50.9401.1800.2563L147.50.9401.1800.2563
LoRA10.7050.8900.2044—————
LoReFT10.6820.7400.0735—————
ReFT-r110.4920.6100.1040L1410.4920.6100.1040
DiffMean1.50.3640.4400.2226L2020.4750.6700.2047
Linear Probe7.50.1480.2200.3013L850.1490.1400.2034
ODESteer540.1300.0800.2394L20720.2010.1300.0832
Random0.010.1000.010−0.0056L80.010.0870.0100.0054

Per-method layer selection improves some methods, while the Overall ranking remains largely unchanged: only ODESteer and Linear Probe exchange positions. For example, DiffMean's Overall increases from 0.364 to 0.475, and ODESteer's from 0.130 to 0.201, with lower Composite scores at the corresponding strengths for both methods. S-PSR, RePS, and ReFT-r1 all select L14, the layer used in the shared-layer setting.

Comparisons over the full strength grids show that both shared-layer and per-method layer selection support the paper's main observation: no evaluated activation steering method achieves both a larger gain in Concept Expression and a lower equally weighted composite side-effect score than Prompt Steering.

Dear Reviewer 9NsY,

Thank you again for your constructive feedback. In response to your concerns, we have added evaluations on Gemma-2-2B at L10 and Gemma-2-9B at L31, and compared shared-layer and validation-based per-method layer selection on Qwen3-8B. These additional evaluations continue to support our main trade-off finding. All changes in the revised manuscript are highlighted in blue.

As the discussion period is ending soon, we look forward to your feedback on whether our revisions and responses have addressed your concerns.

Best regards,

The Authors

Reviewer_M29aOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SteerScope, a multi-dimensional evaluation suite for LLM steering that measures target efficacy, side effects, cross-lingual generalization and data dependence. The authors evaluate 23 steering methods across two Gemma-2 family models and analyze tradeoff between efficacy and side effects under different steering strengths. The results suggest that existing activation steering methods do not outperform Prompting Steering baseline in terms of overall efficacy-side effect tradeoff.

Critical strengths
  • The paper provides a comprehensive evaluation framework that integrates multiple dimensions of steering performance, including language quality, task performance, safety and data dependence.
  • The systematic comparison of diverse steering methods across intervention strengths provides useful empiricial evidence of their efficacy-side effect tradeoffs.
Critical weaknesses
  • Limited novel insights beyond existing observations. The central finding that stronger steering leads to greater side effects is consistent with known oversteering problems. Although the benchmark includes adaptive steering methods designed to mitigate such issues (e.g. PSR, AUSteer), the paper provides limited analysis of why these methods still suffer from efficacy-side effect tradeoffs.
  • Insufficient analysis of the causes of side effects. The evaluation explicitly encourages target concepts to appear even when unrelated to the original instruction. Therefore, some degradation might arise from conflicts between the steering objectives and the underlying task rather than unintended interference. The paper does not sufficiently distinguish these effects or explain how different steering methods exhibit different degradation patterns.
  • Potentially unfair intervention configuration. All methods that intervene at a single layer use layer 20 for both Gemma-2 models, corresponding to different relative depths. Since optimal intervention layers may vary across methods, this fixed configuration may not represent the steering effect of each method.
  • Limited comparability of the fine-tuning baselines. SFT is trained and evaluated on only small fraction of concepts, whereas other methods are evaluated on larger concept sets. This makes direct comparisons potentially misleading, and the paper provdies limited evidence that the SFT configuration is sufficiently optimized.
Questions for authors
  • Could the authors provide further analysis of whether adaptive or fine-grained steering methods exhibit different efficacy-side effect tradeoffs? In particular, which designs appear most effective in mitigating oversteering and when these methods fail to keep steered results plausible in terms of both concept expression and task performance?
  • Could the authors clarify whether method-specific layer selection would change the relative rankings or whether the current reslts would hold?
  • Could the authors explain what the intended role of SFT is in the comparison given its restricted concept set?
9 comments

Thank you for recognizing the value of SteerScope's multidimensional evaluation framework and systematic comparisons across methods and steering strengths. Your suggestion to analyze adaptive and fine-grained methods provides a useful direction for discussing trade-offs across different designs. We address your comments and questions below.

W1: Contributions and main findings

In response to the reviewer's comments on novelty and research insights, we would like to clarify our contributions. By examining steering efficacy and side effects more comprehensively, SteerScope reveals how the performance advantages of existing methods depend on the evaluation dimensions considered, and provides findings on intervention design and generalization. Our contributions have two aspects.

(1) A comprehensive steering evaluation framework. SteerScope organizes evaluation around two complementary aspects, Steering Outcomes and Method Properties, with 15 metrics covering target concept expression, language quality, task capabilities, safety and reliability, cross-lingual generalization, and training-data dependence. We compare 23 methods under a unified protocol and characterize their efficacy–side-effect trade-offs through full steering-strength sweeps. This supports systematic comparisons of existing methods and the evaluation of future improvements in both control effectiveness and side effects.

(2) Findings on method performance and design from multidimensional evaluation. This framework yields several findings:

  • Broader side-effect evaluation reveals the scope of existing performance advantages. Some methods, such as A-PSR, have outperformed Prompt Steering on AxBench's Concept Expression and Overall metrics [3]. However, when we include a broader range of task capability, safety, and response quality metrics, no activation steering method achieves both a larger gain in Concept Expression and a lower composite side-effect score than Prompt Steering within the evaluated configurations and strength grids. This reveals a gap between improvements on existing metrics and advantages in the broader trade-off. Together, this finding and our evaluation framework provide research directions and an empirical basis for developing steering methods that balance target efficacy and side effects.

  • Methods that learn interventions through optimization generally achieve better trade-offs. Compared with optimization-free methods, methods that learn intervention parameters generally exhibit more favorable efficacy–side-effect trade-offs. This comparison provides empirical guidance for method design and motivates further analysis of how training objectives and intervention mechanisms affect the trade-off, as discussed in Q1 below.

  • Low-strength interventions can improve capabilities beyond the target. We observe positive off-target effects for some methods at low steering strengths. In additional direction-reversal experiments, some MMLU gains from positive-direction interventions disappear or turn into degradation. This indicates that these changes depend on the intervention direction and suggests possible connections between target concepts and other capabilities, providing further clues about how steering works.

  • Target efficacy transfers more readily than behavioral quality. Under cross-lingual shifts, most methods retain concept gains, while instruction relevance and fluency degradation often increase. This highlights the need to evaluate both target efficacy and side effects when assessing generalization.

Q1: Trade-offs and causes of failure in adaptive and fine-grained steering methods

Thank you for suggesting further analysis of performance differences across methods. We have added this analysis to the Discussion section on page 9 of the revised manuscript. Drawing on each method's control mechanism and training approach, we discuss design factors that may improve the efficacy–side-effect trade-off, with the aim of informing the design of future activation steering methods.

To this end, among the 23 methods and baselines evaluated, we focus on eight methods from six works that involve adaptive directions or fine-grained strength control:

  • Adaptive directions: HyperSteer, FLAS, HiDRA [11], ODESteer, and StepODESteer, whose effective intervention directions depend on the input or current hidden state.
  • Fine-grained strength control: A-PSR, S-PSR, and AUSteer, which control intervention strength through state-dependent gating or activation-unit scaling.

Even with similar control mechanisms, these methods exhibit markedly different trade-offs in our evaluation. We summarize the overall observations, then examine the implementations of these two groups and discuss their possible effects on concept expression and side effects separately.

(1) Overall observations.

  • Learning interventions through optimization generally yields better trade-offs. Among these eight methods, A-PSR, S-PSR, HyperSteer, and FLAS learn intervention parameters by backpropagating through the language model. They generally achieve more favorable trade-offs between concept expression and side effects than HiDRA and AUSteer, which construct interventions from activation statistics, consistent with our overall findings. ODESteer and its Step variant also involve optimization: they learn a barrier function through classification training on positive and negative activations and use its gradients to construct intervention directions. However, their concept gains are small and their overall trade-offs are relatively weak in our evaluation. These differences suggest that the benefits of optimization may depend on the specific training objective and its connection to generation behavior.

  • Joint learning at different levels may further improve the trade-off. We consider three levels of joint learning. For joint learning of direction and strength, A-PSR and S-PSR jointly optimize a concept direction and an activation-dependent strength gate under the same objective. The direction determines how representations change, while the gate determines how much to intervene in the current state. Adapting both to the target behavior may help balance concept expression and side effects. For joint learning across layers, A-PSR further optimizes these interventions jointly across multiple layers, allowing them to coordinate in producing the target behavior. For example, at operating points with Concept Gain of approximately 1.0 on Gemma-2-9B-it, A-PSR and S-PSR have Composite scores of 0.041 and 0.110, respectively, with lower side effects for A-PSR. This suggests that coordination across layers may further improve the trade-off beyond jointly learning direction and strength. For joint learning across concepts, HyperSteer and FLAS train shared control networks on multiple concepts, allowing control knowledge to be shared across concepts and potentially enabling more general concept control.

  • Intervention construction and training objectives also affect performance. In terms of intervention construction, PSR learns directions and gates; HyperSteer and FLAS learn control networks that generate interventions; and ODESteer and its Step variant learn a barrier function. HiDRA constructs concept directions in a transformed space from activation statistics, while AUSteer selects and scales activation units. In terms of training objectives, PSR imitates prompt-induced behavior through hidden-state matching, HyperSteer and FLAS learn interventions through the generation loss on target responses, and ODESteer uses a concept classification objective. These methods therefore draw on different supervision signals. Supervision more closely tied to the target generation behavior may help methods learn both concept expression and its compatibility with task responses.

(2) Methods with adaptive directions.

These differences in training information provide a concrete basis for comparing methods with adaptive directions. Across the two Gemma configurations we evaluate, HyperSteer and FLAS generally exhibit more favorable trade-offs between concept expression and side effects than HiDRA, ODESteer, and its Step variant.

For concept expression, HyperSteer and FLAS train shared control networks across multiple concepts and learn interventions through the generation loss on target responses. This provides supervision directly tied to generation behavior and allows control knowledge to be shared across concepts. HiDRA constructs interventions from the activation statistics of individual concepts, while ODESteer uses gradients of a per-concept classifier to determine update directions. We hypothesize that shared training across concepts and richer response supervision may help methods learn more general concept control. The FLAS paper also observes that expanding the training concept set mainly improves Concept scores, while Instruction and Fluency remain roughly stable, providing further evidence for the benefits of joint learning across concepts for concept expression [4, §4.3].

For side effects, the target responses used by HyperSteer and FLAS contain concept content, task-relevant information, and linguistic structure, giving training an opportunity to learn how concept expression can be compatible with task responses. The concept mean differences used by HiDRA and the classification gradients used by ODESteer primarily capture information that distinguishes concepts in hidden space. These differences in supervision may help explain why methods with state-dependent directions still exhibit different trade-offs between concept expression and side effects.

(3) Methods with fine-grained strength control.

Beyond intervention direction, how strength is allocated according to the current state may also affect the relationship between concept expression and side effects. PSR and AUSteer offer two different implementations: PSR learns directions and gates through training, while AUSteer selects and scales units based on activation statistics.

For concept expression, PSR jointly learns concept directions and state-dependent strength gates by imitating prompt-induced changes in hidden states. AUSteer selects a small number of units based on activation differences between positive and negative samples and scales them multiplicatively [10]. PSR can therefore adjust its interventions through training to approach the target behavior, while AUSteer's effectiveness depends more on how much the selected units actually control generation of the target concept. In our experiments, PSR achieves markedly larger concept gains, consistent with these differences in mechanism.

For side effects, PSR's gates allocate intervention strength according to the current activations. The original paper's ablation provides direct evidence: under the 9B all-layer MSE training setup, replacing constant strength with dynamic gating slightly reduces Concept from 1.260 to 1.202, while increasing instruction relevance from 1.550 to 1.743 and coherence from 1.044 to 1.096. This supports the role of dynamic strength allocation in improving response quality at some cost to concept expression [3, Appendix E.3, Table 9]. AUSteer's sparse control reduces the number of modified units, but these units may also serve other functions, and their scaling magnitudes are not calibrated through target responses. Thus, the relationship between a smaller intervention scope and lower side effects also depends on the functions of the selected units and their effects on generation.

W2: Effects of task design and prompting strategies on the trade-off

Our templates for generating contrastive training data and the prompt templates for both prompting baselines are taken directly from AxBench [1]. This prompting strategy prioritizes expression of the target concept. We agree with the reviewer's observation that prompting strategies can affect concept expression and response quality, reflecting an efficacy–side-effect trade-off in the choice of strategy. A strategy that emphasizes concept expression may strengthen the intended effect while increasing side effects; a strategy that emphasizes instruction following and coherence may reduce side effects while weakening concept expression. To examine how these strategies perform in practice, we additionally evaluated a milder prompting strategy that requires the model to express the target concept while remaining coherent and directly answering the question.

Specifically, we conducted additional experiments on Gemma-2-9B-it at L20 with 10 concepts, covering 20 methods and a Random control, and retaining each method's full steering-factor grid. We replaced the original template wording that permits departing from the question or disregarding coherence with the milder requirements above, and regenerated the contrastive training data and steering prompts accordingly.

Under the milder strategy, Prompt Steering's gain in Concept Expression is markedly lower, while its composite side-effect score changes little. Thus, the strategy we use achieves stronger concept expression with comparable composite side effects, supporting our choice. For Simple Prompt Steering, both the gain in Concept Expression and the composite side-effect score decrease, illustrating the trade-off between concept expression and response quality across prompting strategies. Under the milder strategy, no evaluated activation steering method achieves both a larger gain in Concept Expression and a lower composite side-effect score than Prompt Steering. Our main trade-off finding therefore continues to hold under the milder prompting strategy. The specific template changes and complete results have been added to Appendix D.10.3 (Sensitivity to Steering Prompts), on page 65 of the revised manuscript.

For a discussion of performance differences across steering methods and their possible causes, please see our response to Q1.

Reviewer_JQCgOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SteerScope, an evaluation suite for LLM steering comprisig 15 metrics, such as target efficacy, language quality, task capabilities, safety, cross-lingual generalization, and training-data dependence. It also compares 23 methods, including activation steering, prompting, LoRA, and SFT. Overall, the paper characterizes efficacy-side-effect tradeoffs in comparison to relying on a single operating point. Also, they find that no activation method achieves higher efficacy while also reducing composite side effects relative to Prompt Steering.

Critical strengths
  • The experiments are super extensive, which broadly measures intended behavior and unintended effects across a wide range of axes.
  • The trade-off analysis were informative and interesting. It also provides actionable generalization findings by separating concept expression from response quality. For instance, steering can retain its intended effect while becoming less usable under language shifts.
Critical weaknesses
  • The task design partly induces the measured conflict. Training and prompting templates explicitly prioritize concept insertion even when irrelevant or incoherent. Some quality degradation may therefore reflect the chosen objective and data recipe, limiting conclusions about steering more generally.
  • I'm writing this for a lot of my papers that I'm reviewing this ICLR: why is there only a single model (Gemma2)? The broad analysis should be adopted to other base models such as Qwen3 or open-source models.
  • Steering strengths are selected on the full in-distribution evaluation set. Reduced-data experiments then retain the full-data-selected strength, so measured sample efficiency can also reflect how well that operating point transfers across data budgets.
  • Core metrics depend on GPT-4o-mini, while human-validation evidence is inherited from a smaller AxBench study. Additional validation is needed for the broader method set and multilingual evaluations.
Questions for authors

see weaknesses above

3 comments

Thank you for recognizing SteerScope's broad evaluation coverage, its analysis of trade-offs across steering strengths, and its findings on cross-lingual generalization. We have added experiments and analyses to address your four comments, as detailed below.

W1: How does task design affect the efficacy–side-effect trade-off?

Our templates for generating contrastive training data and the prompt templates for both prompting baselines are taken directly from AxBench [1]. This prompting strategy prioritizes expression of the target concept. We agree with the reviewer's observation that prompting strategies can affect concept expression and response quality, reflecting an efficacy–side-effect trade-off in the choice of strategy. A strategy that emphasizes concept expression may strengthen the intended effect while increasing side effects; a strategy that emphasizes instruction following and coherence may reduce side effects while weakening concept expression. To examine how these strategies perform in practice, we additionally evaluated a milder prompting strategy that requires the model to express the target concept while remaining coherent and directly answering the question.

Specifically, we conducted additional experiments on Gemma-2-9B-it at L20 with 10 concepts, covering 20 methods and a Random control, and retaining each method's full steering-factor grid. We replaced the original template wording that permits departing from the question or disregarding coherence with the milder requirements above, and regenerated the contrastive training data and steering prompts accordingly.

Under the milder strategy, Prompt Steering's gain in Concept Expression is markedly lower, while its composite side-effect score changes little. Thus, the strategy we use achieves stronger concept expression with comparable composite side effects, supporting our choice. For Simple Prompt Steering, both the gain in Concept Expression and the composite side-effect score decrease, illustrating the trade-off between concept expression and response quality across prompting strategies. Under the milder strategy, no evaluated activation steering method achieves both a larger gain in Concept Expression and a lower composite side-effect score than Prompt Steering. Our main trade-off finding therefore continues to hold under the milder prompting strategy. The specific template changes and complete results have been added to Appendix D.10.3 (Sensitivity to Steering Prompts), on page 65 of the revised manuscript.

W2: Do the main findings extend to other model families?

We conducted additional experiments on Qwen3-8B, using L14 for single-layer interventions and evaluating 11 methods plus a Random control on 10 concepts, with each method's full steering-factor grid. The results show that no evaluated activation steering method achieves both higher Concept Expression and a lower composite side-effect score than Prompt Steering. In cross-lingual generalization, several methods preserve or increase Concept Gain while incurring greater Instruction Degradation, supporting the finding that target efficacy transfers more readily than behavioral quality. The experimental setup and complete results have been added to Appendix D.10.2 (Cross-Model Evaluation), on page 64 of the revised manuscript.

W3: How does Sample Efficiency reflect the applicability of an operating point across data budgets?

We aim to examine how much steering effectiveness interventions constructed from limited data retain at a fixed operating point. This objective considers robustness to changes in data quantity and quality in real-world settings, as well as the additional cost of recalibrating the steering strength for every concept and data budget. We therefore fix the steering factor to the value selected in the full-data setting. We have further clarified this objective and setup in the definition of Sample Efficiency in Section 3.2 on page 5 of the revised manuscript.

Prior studies adopt similar approaches: FaithSteer-BENCH [2] evaluates robustness under different stress conditions at a fixed, calibrated operating point; DSG [3] evaluates different data subsets with the configuration selected using the full dataset; and COLD-Steer [4], DSPA [5], STA [6], and VIAS [7] also analyze data efficiency under fixed intervention settings.

W4: How reliable is the LLM judge across more methods and in cross-lingual evaluation?

Thank you for suggesting further validation of the LLM judge's reliability. We added independent evaluations by five human experts, who rated 222 responses from the Gemma-2-2B-it L20 experiments on Concept Expression, Instruction Relevance, and Fluency, using the same scoring rubric as the LLM judge.

The efficacy–side-effect evaluation includes 150 responses across 10 concepts, covering DiffMean, ODESteer, FLAS, A-PSR, Prompt Steering, LoRA, and the unsteered model. For each of the first four methods, we selected three steering factors: the factor maximizing Overall and two other randomly sampled factors. The cross-lingual evaluation includes 72 responses across three concepts, covering the six steering methods listed above in Chinese, Korean, Italian, and Spanish.

Table 1: Spearman correlations between the LLM judge and the mean ratings of the five human experts. Concept and Instruction denote Concept Expression and Instruction Relevance, respectively. Following the paper's definition, Overall is computed from each rater's scores on the three dimensions; human Overall scores are computed separately for each expert and then averaged across the five experts.

Evaluation settingNConceptInstructionFluencyOverall
Efficacy and Side Effects1500.8160.8190.7150.734
Cross-lingual Generalization720.8220.5760.5810.761
All2220.8180.7530.6770.737

Table 2: Pairwise Spearman correlations for Overall across all 222 responses. H1–H5 denote the five human experts.

H1H2H3H4H5LLM Judge
H11.0000.4280.6340.7390.6980.677
H20.4281.0000.3830.4450.4710.355
H30.6340.3831.0000.6050.6490.648
H40.7390.4450.6051.0000.6360.735
H50.6980.4710.6490.6361.0000.629
LLM Judge0.6770.3550.6480.7350.6291.000

For Overall, the mean correlation between the LLM judge and individual experts is 0.609, compared with a mean correlation of 0.569 between human experts, indicating that LLM–human agreement is comparable to inter-human agreement. These results support the reliability and validity of the LLM judge in our evaluation setting. The complete human evaluation setup, correlations by evaluation setting, and agreement matrices for the three scoring dimensions and Overall have been added to Appendix C.6 (Human Validation), on page 28 of the revised manuscript.

References

[1] Zhengxuan Wu et al. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. 2025.

[2] Zikang Ding et al. FaithSteer-BENCH: A Deployment-Aligned Stress-Testing Benchmark for Inference-Time Steering. 2026.

[3] Aashiq Muhamed et al. SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs. 2025.

[4] Kartik Sharma and Rakshit S. Trivedi. COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics. 2026.

[5] James Wedgwood et al. DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment. 2026.

[6] Mengru Wang et al. Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atoms. 2025.

[7] Avni Mittal et al. VIAS: Value-Isolating Activation Steering for Multi-Value Alignment and Compliance for Organisations. 2026.

Dear Reviewer JQCg,

Thank you again for your constructive feedback. In response to your concerns, we have tested a milder prompting strategy, added Qwen3-8B evaluations, clarified the purpose of our sample-efficiency evaluation, and added human validation of the LLM judge. All changes in the revised manuscript are highlighted in blue.

As the discussion period is ending soon, we look forward to your feedback on whether our revisions and responses have addressed your concerns.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to the Area Chair and All Reviewers

We sincerely thank the Area Chair and Reviewers JQCg, M29a, and 9NsY for their constructive feedback and recognition of SteerScope’s evaluation coverage and trade-off analysis. In response, we have added experiments and expanded our analysis of the findings and their implications for method design. All updates in the revised manuscript are highlighted in blue. The major revisions are summarized below.

1. Extending validation across model families and intervention layers (W2 of JQCg; W3 and Q2 of M29a; W1, Q1, and Q2 of 9NsY).
We added evaluations on Gemma-2-2B at L10, Gemma-2-9B at L31, and Qwen3-8B, using full steering-strength grids. We also compared shared-layer and validation-based per-method layer selection on Qwen3-8B. Per-method selection improves some methods, while the Overall ranking remains largely unchanged. Across these additional settings, no evaluated activation steering method achieves both greater concept gain and lower composite side effects than Prompt Steering.

2. Testing a milder prompting strategy (W1 of JQCg; W2 of M29a).
We evaluated a prompting strategy requiring the model to express the target concept while remaining coherent and directly answering the question. We regenerated the contrastive training data and steering prompts, and evaluated 20 methods plus Random on 10 concepts using Gemma-2-9B at L20. The main trade-off finding continues to hold under this milder strategy. The templates and results are reported in Appendix D.10.3.

3. Expanding the analysis of method differences (W1 and Q1 of M29a).
We expanded the Discussion to analyze adaptive and fine-grained steering methods, focusing on design factors including training objectives, intervention construction, state-dependent strength allocation, and joint learning across layers and concepts. We discuss how these factors relate to concept expression and side effects, providing insights for designing future steering methods that better balance efficacy and side effects.

4. Adding human validation (W4 of JQCg).
Five experts independently rated 222 responses on Concept Expression, Instruction Relevance, and Fluency. For Overall, the mean correlation between the LLM judge and individual experts is 0.609, compared with 0.569 between human experts, indicating that LLM–human agreement is comparable to inter-human agreement. These results support the reliability and validity of the LLM judge in our evaluation setting. The protocol and agreement analyses are reported in Appendix C.6.

5. Clarifying the purpose of sample-efficiency evaluation and strengthening SFT comparability (W3 of JQCg; W4 and Q3 of M29a).
We clarified that Sample Efficiency measures retention of steering effectiveness at a fixed, full-data-selected operating point as the data budget changes. We also compared all 23 methods and Random on the same 20 concepts used by SFT for each Gemma model, retaining full steering-strength grids. This comparison remains consistent with the main trade-off finding; results are reported in Appendix D.10.4.

Detailed experimental settings, results, and responses to individual concerns are provided in our point-by-point replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.