- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces RealContolBench, a benchmark to evaluate activation steering across five domains (response formatting, tool-use, math reasoning, reasoning-length and safety). The benchmark enables the comparison of steering methods for a given model, along with a prompting baseline: the paper reports on three different models. Overall, the benchmark aims to study whether steering induces the desired behaviour on downstream tasks while preserving overall model capabilities. The results show some gains from steering with no method leading across all tasks while prompting largely remains a strong baseline. The paper highlights trade-offs between behavioural control, task success and generalization.
Critical strengths
- The paper is clearly framed and well motivated, considering behavioral control alongside task performance rather than behavioral change alone. It is well written and covers three model families and a broad range of steering methods.
- A very valuable benchmark for future work on steering methods.
- Each component of RealControlBench (IF, Tool Use, Reasoning, Long2Short, and Safety) is derived from existing, established benchmarks that were not originally designed to evaluate steering methods.
Critical weaknesses
- For the IF component of the benchmark, the steering vectors are extracted from 100 separate questions, but the layer and steering strength are selected to maximize scores on the test set itself (cf. Section C.1 Table 12). This creates a risk of overfitting to the test set.
- The paper measures performance degradation in some settings, particularly Long2Short, but lacks a systematic analysis of how steering affects other model capabilities and how this trade-off varies with intervention strength.
Questions for authors
- Could you please select the IF steering hyperparameters using a development set that is separate from the test set?
- For tool use, Long2Short, and safety, could you provide more details on how the steering configurations were selected? Please report the selected layers, strengths, selection rules, and results across the tested strengths. Math improvement already uses a separate development set.
- Could you explain in more detail why the steering methods fail? The main results report overall scores. Examples of outputs before and after steering would help illustrate incorrect decisions, failed execution, mathematical mistakes, missing final answers, and other failure modes.
10 comments
Steering configuration selection for Tool Use, Long2Short, and Safety (part1)
2. Steering Configuration Selection Across Domains
To further clarify our evaluation procedures, we provide the configuration-selection objectives, search spaces, and selected configurations for Tool Use and Long2Short, followed by the setup and strength-dependent results of our additional Safety execution study.
2.1 Tool Use
Search spaces. For CAA, PCA, LAT, and Linear Probe, we search the following layers:
-
Qwen3-8B: {9, 18, 27}
-
Llama-3.1-8B: {8, 16, 24}
-
Gemma-2-9B: {11, 21, 32}
For these methods, α ∈ {0.5, 1, 1.5, 2, 2.5}. Gemma SAE uses layers {9, 20, 31} and the same strength grid. ITI selects 48 attention heads across layers and searches α ∈ {1, 5, 10, 15, 20}, without a separate layer sweep.
Layer indices are zero-based. The parameter α controls steering strength. For residual-stream methods and SAE, α specifies perturbation magnitude relative to the residual norm; for ITI, it scales head-wise activation standard deviations.
Single-turn selection and evaluation. We designate the original 200 examples per target as development data and evaluate the selected configurations on 200 new test examples per target. Selection is performed independently for each model, method, and target:
-
Decision: configurations maximize development initial-response success.
-
FollowUp: configurations maximize the joint success of the initial response and subsequent tool call.
Consequently, Decision and FollowUp may select different configurations for the same model, method, and target, reflecting their different evaluation objectives.
Multi-turn selection and evaluation. We use a fixed, scenario-grouped split of the previously evaluated 200 conversations per target into 100 development and 100 test conversations. Configurations maximize development joint success, combining BFCL execution and irrelevance checks with target-decision correctness for clarification and abstention. We select configurations using development results and retrieve their corresponding test results from the existing outputs, without additional generation or judging. This multi-turn analysis is a retrospective reaggregation of previously evaluated conversations, rather than a newly generated independent test evaluation. Across all three settings, ties are resolved by lower α, then lower layer where applicable, then configuration ID. Selected configurations are fixed for test reporting.
In Tables 2–4, configurations are written as layer/α, with zero-indexed layers. H48/α denotes ITI's global 48-head profile and corresponding strength. Semicolons separate the targets in the order stated in each caption.
Table 2. Single-turn Decision — development-selected configurations. Entries are ordered Call; Clarify; Abstain.
| Method | Qwen3-8B | Llama-3.1-8B | Gemma-2-9B |
|---|---|---|---|
| CAA | 27/0.5; 9/2; 27/1 | 16/0.5; 8/2; 8/1 | 32/0.5; 21/1; 11/1 |
| PCA | 27/0.5; 18/1; 9/1 | 16/0.5; 8/2.5; 24/1 | 32/0.5; 21/1; 11/0.5 |
| LAT | 27/0.5; 9/1.5; 9/1 | 16/0.5; 8/2.5; 8/0.5 | 32/0.5; 21/1; 11/1 |
| Linear Probe | 18/0.5; 18/1; 9/1 | 8/0.5; 8/2.5; 16/0.5 | 11/0.5; 21/0.5; 11/1 |
| ITI | H48/5; H48/1; H48/5 | H48/1; H48/1; H48/1 | H48/1; H48/10; H48/20 |
| SAE | — | — | 9/0.5; 20/2; 9/0.5 |
Table 3. Single-turn FollowUp — development-selected configurations. Entries are ordered Clarify; Abstain.
| Method | Qwen3-8B | Llama-3.1-8B | Gemma-2-9B |
|---|---|---|---|
| CAA | 18/0.5; 27/1 | 24/0.5; 8/0.5 | 21/0.5; 11/0.5 |
| PCA | 18/0.5; 9/0.5 | 8/1.5; 24/1 | 21/0.5; 11/0.5 |
| LAT | 18/0.5; 27/1 | 24/0.5; 8/0.5 | 21/0.5; 11/0.5 |
| Linear Probe | 18/1; 27/1 | 24/0.5; 8/0.5 | 21/0.5; 11/1 |
| ITI | H48/1; H48/5 | H48/1; H48/1 | H48/5; H48/1 |
| SAE | — | — | 20/0.5; 9/0.5 |
Table 4. Multi-turn — development-selected configurations. Entries are ordered Call; Clarify; Abstain.
| Method | Qwen3-8B | Llama-3.1-8B | Gemma-2-9B |
|---|---|---|---|
| CAA | 27/0.5; 9/0.5; 9/0.5 | 16/1; 8/0.5; 8/0.5 | 11/0.5; 11/0.5; 11/0.5 |
| PCA | 18/0.5; 9/0.5; 9/0.5 | 16/0.5; 8/0.5; 8/0.5 | 11/0.5; 11/0.5; 11/0.5 |
| LAT | 27/0.5; 9/1; 9/0.5 | 16/0.5; 8/0.5; 8/0.5 | 11/0.5; 11/0.5; 11/0.5 |
| Linear Probe | 27/0.5; 18/0.5; 9/0.5 | 16/1; 8/1; 8/0.5 | 21/0.5; 11/0.5; 11/0.5 |
| ITI | H48/5; H48/1; H48/1 | H48/1; H48/1; H48/1 | H48/5; H48/1; H48/1 |
| SAE | — | — | 9/0.5; 9/0.5; 9/0.5 |
Steering configuration selection for Tool Use, Long2Short, and Safety (part2)
Table 5. Corresponding tool-use test results (%).
| Method | Decision: Qwen | Decision: Llama | Decision: Gemma | FollowUp: Qwen | FollowUp: Llama | FollowUp: Gemma | Multi-turn: Qwen | Multi-turn: Llama | Multi-turn: Gemma |
|---|---|---|---|---|---|---|---|---|---|
| Unsteered | 32.67 | 24.83 | 39.33 | 14.00 | 4.00 | 17.50 | 9.00 | 1.33 | 1.00 |
| Prompted | 41.67 | 44.33 | 65.50 | 23.00 | 25.50 | 49.50 | 10.67 | 3.33 | 1.67 |
| CAA | 45.50 | 23.67 | 47.17 | 21.25 | 3.00 | 24.25 | 9.33 | 6.00 | 1.33 |
| PCA | 42.67 | 28.67 | 46.83 | 18.50 | 5.50 | 24.50 | 8.67 | 4.00 | 0.67 |
| LAT | 41.83 | 25.00 | 47.67 | 22.25 | 3.25 | 24.25 | 9.33 | 4.00 | 0.67 |
| Linear Probe | 43.33 | 24.67 | 38.17 | 22.50 | 3.25 | 20.25 | 9.33 | 6.33 | 0.33 |
| ITI | 32.50 | 25.00 | 43.83 | 14.50 | 4.75 | 19.25 | 7.67 | 1.33 | 0.33 |
| SAE | — | — | 46.17 | — | — | 21.50 | — | — | 1.33 |
Decision and FollowUp refer to the single-turn track and use their respective development-selected configurations. Decision and multi-turn scores average equally over Call, Clarify, and Abstain; FollowUp averages over Clarify and Abstain. The model abbreviations refer to Qwen3-8B, Llama-3.1-8B, and Gemma-2-9B. Unsteered and Prompted require no intervention-layer or strength selection.
Strength sensitivity and downstream performance. To further examine how intervention strength interacts with task completion, we compare two strengths while holding the model, method, layer, and 200 clarification examples fixed.
Table 6. Strength comparison for Gemma-2-9B with CAA (%).
| α | Initial decision accuracy | Joint follow-up success |
|---|---|---|
| 0.5 | 20.5 | 16.5 |
| 1.0 | 23.0 | 1.5 |
Increasing α improves initial decision accuracy by 2.5 percentage points while reducing joint follow-up success by 15.0 percentage points. Optimizing the initial behavioral decision can favor configurations that perform poorly when subsequent tool execution is required.
This comparison is a descriptive analysis of the historical evaluation grid; it does not isolate the causal effect of applying steering during the second turn. It provides additional evidence for the importance of evaluating initial behavioral control alongside downstream task performance.
2.2 Reasoning-Length Control (Long2Short)
For Long2Short, configurations are selected using a development set of 64 MATH questions, with the objective of minimizing generated output length in tokens. The layer and strength grids follow those specified above for the corresponding methods; SAE is evaluated only on Gemma, and ITI uses the global 48-head profile. We report the selected configurations together with output-length change, answer accuracy, and generation-collapse diagnostics.
Table 7. Development-selected Long2Short configurations and corresponding evaluation results.
| Model | Method | Layer | α | Compression Rate | Acc | Collapse Rate |
|---|---|---|---|---|---|---|
| Qwen3-8B | No Steering | — | — | — | 0.806 | 0.070 |
| Prompted | — | — | -0.415 | 0.770 | 0.036 | |
| CAA | 18 | 1.0 | -0.032 | 0.818 | 0.058 | |
| PCA | 27 | 2.5 | -0.020 | 0.806 | 0.062 | |
| LAT | 27 | 1.0 | -0.027 | 0.806 | 0.070 | |
| Linear Probe | 9 | 2.5 | -0.016 | 0.812 | 0.066 | |
| ITI | Top-48 heads | 1.0 | +0.015 | 0.800 | 0.084 | |
| No Steering | — | — | — | 0.484 | 0.006 | |
| Prompted | — | — | -0.384 | 0.438 | 0.072 | |
| CAA | 32 | 1.5 | -0.010 | 0.484 | 0.006 | |
| PCA | 21 | 0.5 | -0.017 | 0.494 | 0.004 | |
| LAT | 21 | 1.5 | -0.014 | 0.492 | 0.004 | |
| Linear Probe | 32 | 0.5 | -0.023 | 0.484 | 0.004 | |
| SAE | 9 | 2.5 | -0.010 | 0.486 | 0.004 | |
| ITI | Top-48 heads | 1.0 | -0.021 | 0.460 | 0.006 | |
| Llama-3.1-8B-Instruct | No Steering | — | — | — | 0.506 | 0.206 |
| Prompted | — | — | -0.296 | 0.472 | 0.180 | |
| CAA | 8 | 2.5 | -0.193 | 0.306 | 0.188 | |
| PCA | 24 | 1.5 | -0.030 | 0.490 | 0.196 | |
| LAT | 8 | 2.5 | -0.200 | 0.306 | 0.194 | |
| Linear Probe | 8 | 2.5 | -0.228 | 0.338 | 0.166 | |
| ITI | Top-48 heads | 1.0 | +0.021 | 0.518 | 0.218 |
Collapse is defined as any of:
- empty: Output is entirely whitespace.
- truncated: Hit the max_new_tokens limit (3072) without finishing.
- repetition: same contiguous 50-token segment appears 3 or more times
- short_reasoning: The reasoning section is shorter than 32 tokens.
- answer_only: No reasoning; only the answer was given.
Length changes are measured relative to the corresponding unsteered outputs; negative values indicate shorter generations. These results report the configurations selected from the development search.
The results illustrate that the relationship between reasoning compression and task accuracy depends on both the model and steering method. For instance, on Llama-3.1-8B, CAA and LAT reduce output length by approximately 20% while accuracy drops from 50.6% to 30.6%. In contrast, PCA achieves a smaller reduction of 2.97% while preserving 49.0% accuracy. Gemma-2-9B largely maintains accuracy under the evaluated learned-steering configurations.
Response to Reviewer
Thank you for your thoughtful review and positive assessment of RealControlBench. We are grateful for your recognition of the paper’s motivation, its broad coverage of models and steering methods, and its potential value for future research. Your acknowledgment of the importance of evaluating behavioral control alongside task performance is particularly encouraging, as this is central to our work. Your constructive comments also highlight several aspects of our evaluation that merit further clarification, particularly configuration selection, the relationship between intervention strength and task performance, and the interpretation of failure cases. We welcome this opportunity to elaborate on our methodological choices, clarify the evaluation procedures, and present additional evidence supporting our findings.
Below, we address your questions in the order raised, providing the relevant clarifications, experimental results, and analyses.
1. Development–Test Separation for Instruction Following
Q1. Could you please select the IF steering hyperparameters using a development set that is separate from the test set?
Thank you for highlighting the distinction between using separate data for vector extraction and using separate data for configuration selection. We appreciate the opportunity to make this distinction explicit and provide additional evidence regarding the robustness of our IF findings. Following your suggestion, we adopt the following procedure:
We verified that the extraction, development, and test prompts are disjoint. Configuration selection maximizes development IF success for the corresponding model, method, and constraint.
Table 1. IF success on the new test set (%).
Each entry reports IF success rate (%), averaged over five constraints. Success requires satisfying the target constraint while remaining coherent and adequately answering the question. Each model–method pair is evaluated on the same 200 test questions under all five constraints, yielding 1,000 question–constraint evaluations. A dash indicates an unevaluated combination.
Across the 16 internal-steering model–method combinations, the new test scores differ from the historical development scores by −2.0 to +1.2 percentage points. Our central IF conclusion remains unchanged under development-based configuration selection and evaluation on previously unused questions: prompting achieves 96.3%, 87.7%, and 88.3% on Qwen3-8B, Llama-3.1-8B, and Gemma-2-9B, respectively, while the strongest evaluated internal-steering result for each model is 39.2%, 26.6%, and 33.0%.
These additional results demonstrate that the principal comparison between prompting and internal steering remains consistent under an independent evaluation protocol. We will update the reported IF results and revise Section C.1, D.1, and Table 12 to explicitly distinguish development-based configuration selection from independent test evaluation.