- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SteerScope, a multidimensional evaluation suite for LLM steering that jointly assesses target efficacy, unintended side effects, cross-lingual generalization, and dependence on training-data size and composition through 15 metrics. The authors evaluate 23 methods spanning activation steering, prompting, and parameter fine-tuning on Gemma-2-2B-it and Gemma-2-9B-it, using steering-strength sweeps to characterize efficacy–side-effect trade-offs. Under the evaluated configurations and default aggregation, no activation steering method achieves both higher efficacy and lower composite side effects than Prompt Steering. The experiments further show that stronger steering generally incurs greater side effects, cross-lingual shifts can preserve target efficacy while worsening response quality, and methods differ substantially in their data dependence.
Critical strengths
-
The paper addresses a meaningful gap in steering evaluation: achieving the target behavior is only useful if the model remains capable and reliable. Bringing capability, safety, and response-quality measurements into the same evaluation makes it possible to see costs that existing steering scores can miss. This is the strongest reason I see value in the benchmark.
-
The strength sweeps make the comparison more informative than a ranking at a single operating point. Evaluating a broad set of methods this way shows where stronger control comes at the expense of other behaviors. The metric-weighting analysis also lends support to the finding that Prompt Steering remains a competitive baseline, rather than leaving that conclusion entirely dependent on equal weighting.
Critical weaknesses
My main concern is that the breadth of the method comparison is not matched by the breadth of the model evaluation. Both backbones belong to the Gemma-2 family, and single-site interventions are fixed at layer 20, which corresponds to quite different relative depths in the two models. This provides a controlled comparison, but leaves open whether the observed trade-offs reflect the methods themselves or their suitability for these particular configurations. Since a central finding concerns activation steering’s competitiveness with Prompt Steering, a small validation on another model family, with reasonable intervention-layer selection, would substantially strengthen the evidence. As it stands, I find the benchmark useful, but the scope of its empirical conclusions limited.
Questions for authors
- Clarification: What motivated fixing single-site interventions at layer 20 in both models, and what evidence suggests this choice does not disadvantage particular methods?
- Additional validation: Can you test Prompt Steering and a few representative activation steering methods on another model family, with intervention layers selected on held-out data, to assess whether the main trade-off finding holds?
4 comments
Q2: Can the main trade-off finding be tested on another model family with layers selected on separate validation data?
(a) Validation and evaluation protocol.
On Qwen3-8B, we include 11 methods plus a Random control. Six methods undergo per-method layer selection: S-PSR, RePS, ReFT-r1, DiffMean, Linear Probe, and ODESteer. Random follows the same layer-selection protocol to compare concept-related interventions with random perturbations. The other five methods are Prompt Steering, Simple Prompt Steering, A-PSR, LoRA, and LoReFT.
We select each method's intervention layer on a validation set, with L8, L14, L20, and L28 as candidates. At each candidate layer, we sweep the method's 14 nonzero steering strengths and select the layer based on Overall. Specifically, we first take the highest mean validation Overall score across the strength grid at each layer, then select the layer with the highest score.
The shared layer is also determined on the validation set. At each candidate layer, we average the best mean Overall scores across the methods participating in layer selection and select L14, which has the highest average, as the shared intervention layer.
After layer selection, we evaluate the full strength grids on a separate evaluation set covering 10 concepts, comparing efficacy–side-effect trade-offs under shared L14 and per-method layer selection. The other five methods retain identical configurations in both settings.
(b) Per-method layer-selection results.
The table below reports each method's validation performance at the four candidate layers. Each cell reports the highest mean Overall score attained by the method over its strength grid at that layer. The selected layer is bolded. Methods are ordered by their validation Overall score at the selected layer, from highest to lowest.
| Method | L8 | L14 | L20 | L28 |
|---|---|---|---|---|
| S-PSR | 0.838 | 1.060 | 0.726 | 0.074 |
| RePS | 0.834 | 0.874 | 0.798 | 0.490 |
| ReFT-r1 | 0.282 | 0.542 | 0.220 | 0.384 |
| DiffMean | 0.240 | 0.216 | 0.288 | 0.234 |
| Linear Probe | 0.156 | 0.128 | 0.060 | 0.024 |
| ODESteer | 0.024 | 0.072 | 0.096 | 0.048 |
| Random | 0.030 | 0.020 | 0.024 | 0.024 |
S-PSR, RePS, and ReFT-r1, which have higher validation Overall scores, all select L14. This may suggest that the advantage of an intervention layer can generalize to some extent across methods.
(c) Trade-offs under shared-layer and per-method layer selection.
The table below compares evaluation results under shared L14 and per-method layer selection. For each setting, we report the steering factor with the highest mean Overall score and its corresponding Overall, Concept Gain, and Composite scores. Methods are ordered by Overall under the shared-layer setting, from highest to lowest. Entries for methods that do not undergo per-method layer selection are marked with “—”.
| Method | Shared factor | Shared Overall ↑ | Shared Concept Gain ↑ | Shared Composite ↓ | Per-method layer | Per-method factor | Per-method Overall ↑ | Per-method Concept Gain ↑ | Per-method Composite ↓ |
|---|---|---|---|---|---|---|---|---|---|
| A-PSR | 2 | 1.233 | 1.310 | 0.0731 | — | — | — | — | — |
| Simple Prompt Steering | 1 | 1.169 | 1.420 | 0.0401 | — | — | — | — | — |
| S-PSR | 3 | 1.125 | 1.190 | 0.1243 | L14 | 3 | 1.125 | 1.190 | 0.1243 |
| Prompt Steering | 1 | 1.095 | 1.070 | 0.0241 | — | — | — | — | — |
| RePS | 7.5 | 0.940 | 1.180 | 0.2563 | L14 | 7.5 | 0.940 | 1.180 | 0.2563 |
| LoRA | 1 | 0.705 | 0.890 | 0.2044 | — | — | — | — | — |
| LoReFT | 1 | 0.682 | 0.740 | 0.0735 | — | — | — | — | — |
| ReFT-r1 | 1 | 0.492 | 0.610 | 0.1040 | L14 | 1 | 0.492 | 0.610 | 0.1040 |
| DiffMean | 1.5 | 0.364 | 0.440 | 0.2226 | L20 | 2 | 0.475 | 0.670 | 0.2047 |
| Linear Probe | 7.5 | 0.148 | 0.220 | 0.3013 | L8 | 5 | 0.149 | 0.140 | 0.2034 |
| ODESteer | 54 | 0.130 | 0.080 | 0.2394 | L20 | 72 | 0.201 | 0.130 | 0.0832 |
| Random | 0.01 | 0.100 | 0.010 | −0.0056 | L8 | 0.01 | 0.087 | 0.010 | 0.0054 |
Per-method layer selection improves some methods, while the Overall ranking remains largely unchanged: only ODESteer and Linear Probe exchange positions. For example, DiffMean's Overall increases from 0.364 to 0.475, and ODESteer's from 0.130 to 0.201, with lower Composite scores at the corresponding strengths for both methods. S-PSR, RePS, and ReFT-r1 all select L14, the layer used in the shared-layer setting.
Comparisons over the full strength grids show that both shared-layer and per-method layer selection support the paper's main observation: no evaluated activation steering method achieves both a larger gain in Concept Expression and a lower equally weighted composite side-effect score than Prompt Steering.
References
[1] Zhengxuan Wu et al. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. 2025.
[2] Jiuding Sun et al. HyperSteer: Activation Steering at Scale with Hypernetworks. 2025.
[3] Geert Heyman and Frederik Vandeputte. Steer Like the LLM: Activation Steering that Mimics Prompting. 2026.
[4] Zehao Jin et al. Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention. 2026.
[5] Ziwen Xu et al. EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Models. 2025.
[6] Ziwen Xu et al. How Controllable Are Large Language Models? A Unified Evaluation across Behavioral Granularities. 2026.
[7] Hongjue Zhao et al. ODESteer: A Unified ODE-based Steering Framework for LLM Alignment. 2026.
[8] Shawn Im and Sharon Li. A Unified Understanding and Evaluation of Steering Methods. 2025.
[9] Zikang Ding et al. FaithSteer-BENCH: A Deployment-Aligned Stress-Testing Benchmark for Inference-Time Steering. 2026.
Dear Reviewer 9NsY,
Thank you again for your constructive feedback. In response to your concerns, we have added evaluations on Gemma-2-2B at L10 and Gemma-2-9B at L31, and compared shared-layer and validation-based per-method layer selection on Qwen3-8B. These additional evaluations continue to support our main trade-off finding. All changes in the revised manuscript are highlighted in blue.
As the discussion period is ending soon, we look forward to your feedback on whether our revisions and responses have addressed your concerns.
Best regards,
The Authors
Thank you for recognizing the value of SteerScope's joint evaluation of concept control, model capabilities, safety, and response quality, as well as its characterization of trade-offs through steering-strength sweeps. We agree that extending evaluation to other model families and examining the effects of intervention layer selection help clarify the scope of the main findings. Below, we explain the rationale for using L20 and a shared intervention layer, and report evaluations at different Gemma layers and a comparison of fixed-layer and per-method layer selection on Qwen3-8B.
Q1: Why use L20 in both models? Could this disadvantage some methods?
This question concerns two choices: using L20 in both Gemma models and sharing an intervention layer across methods. To examine the former, we added evaluations on Gemma-2-2B at L10 and Gemma-2-9B at L31 to study method performance at different relative depths. To examine the latter, we compare a fixed L14 with per-method layer selection on Qwen3-8B to assess whether method rankings and the main trade-off finding change.
(a) Rationale for L20 and evaluations at different intervention layers.
Within each evaluation configuration, AxBench [1] fixes a common intervention layer for different single-layer intervention methods. It evaluates four configurations: L10 and L20 of Gemma-2-2B, and L20 and L31 of Gemma-2-9B. Among these configurations, works including HyperSteer [2], Steer Like the LLM [3], FLAS [4], EasyEdit2 [5], and SteerEval [6] use L20 in their respective main Gemma evaluations. We follow this configuration for comparability with prior work. In addition, SteerScope covers many methods, full steering-strength grids, and multiple evaluation dimensions; extending this to an exhaustive layer sweep for every method would substantially increase computational cost. We therefore use a common configuration for the main experiments.
We agree that L20 corresponds to different relative depths in the two models. To examine method performance at different relative depths, we added experiments on Gemma-2-2B at L10 and Gemma-2-9B at L31. Each configuration covers 20 methods plus a Random control on 10 concepts, with each method's full steering-factor grid. Both configurations support the paper's main observation: no evaluated activation steering method achieves both a larger gain in Concept Expression and a lower equally weighted composite side-effect score than Prompt Steering. Complete results have been added to Appendix D.10.1 (Alternative Intervention Layers), on page 62 of the revised manuscript.
(b) Rationale for a shared intervention layer and the per-method layer-selection experiment.
Sharing an intervention layer is a widely adopted evaluation setup in steering research. Examples include method studies such as ODESteer [7] and A Unified Understanding and Evaluation of Steering Methods [8], benchmarks such as AxBench [1], SteerEval [6], and FaithSteer-BENCH [9], and the AxBench evaluation in EasyEdit2 [5]. Following this controlled comparison approach, our choice reflects three considerations:
(1) Consistent comparison conditions.
Comparing methods at a common representation site controls for the effect of intervention location and focuses the comparison on the intervention methods themselves; steering strengths are still swept separately for each method.
(2) Alignment between SAE features and concept data.
SAE feature dictionaries are layer-specific. Extracting new features at different layers to define evaluation concepts would also change the concept set. We therefore maintain consistency between the intervention layer, feature dictionary, and concept data to compare methods on the same evaluation targets.
(3) Balancing evaluation coverage and computational budget.
In an evaluation covering many methods, full strength grids, and multiple metrics, sharing a layer allows us to allocate the computational budget to broader evaluation coverage.
To examine how a shared layer affects different methods, we conducted a per-method layer-selection experiment on Qwen3-8B with separate validation data and compared it with the shared-layer setting, as detailed in Q2.