acceptodds
Under review as a conference paper at ICLR 2027

Steer Only When Necessary: Factorized Intervention Necessity for LLM Safety

Abstract

Activation steering provides a lightweight approach to improving large language model safety at inference time, but unnecessary intervention can cause over-refusal and degrade generation quality. These costs extend beyond benign requests: a model may already respond safely to a harmful request, making additional steering unnecessary. This motivates identifying such cases before generation while accounting for the risk of cancelling necessary interventions. To address this challenge, we introduce **Factorized Intervention Necessity (FIN)**, which separates semantic eligibility, based on hazard relevance and harmful intent, from untreated vulnerability, indicating whether the target model would respond unsafely without steering. FIN estimates vulnerability from prompt representations using offline supervision from unsteered responses. Eligible requests receive steering by default; low predicted vulnerability permits cancellation only under a rule certified on an independent held-out set. Certification assesses the probability of cancellation among eligible requests with unsafe unsteered responses. Across three primary models and diverse safety and utility benchmarks, FIN reduces unsafe leakage relative to unsteered generation while largely preserving general capabilities. Compared with the evaluated selective and adaptive baselines, FIN achieves the highest average benign compliance. Integrating FIN with multiple steering methods further reduces average quality degradation on harmful requests that the unsteered models already handle safely.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Safety, Activation Steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t6JtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for determining when activation steering is necessary for LLM safety. The key insight is that harmful requests do not always require additional intervention, as the target model may already produce safe responses without steering. FIN addresses this by separating semantic eligibility from model-specific untreated vulnerability, using internal representations to predict whether intervention is needed before generation. A statistical risk-control mechanism is further introduced to limit unsafe intervention cancellation.

Critical strengths

The paper addresses a practical problem, highlighting that unnecessary steering can degrade response quality even for harmful requests that models already handle safely. The proposed framework is well structured, and the incorporation of statistical risk control adds methodological rigor to the intervention decision. The experimental evaluation is relatively comprehensive, covering different model families, safety benchmarks, general capabilities, and compatibility with existing steering methods.

Critical weaknesses

The main limitation is that the methodological novelty is somewhat incremental, as several components rely on established representation-based classifiers and statistical calibration techniques. Although the factorized formulation is intuitive, the extent to which this decomposition provides benefits beyond simpler vulnerability-based intervention strategies remains insufficiently established. Additionally, the statistical guarantee applies specifically to intervention cancellation among eligible requests with unsafe unsteered responses, rather than providing an end-to-end safety guarantee.

Questions for authors

Could the authors further clarify the necessity of explicitly separating semantic eligibility and untreated vulnerability, particularly compared with a simpler approach that directly predicts whether the unsteered model will produce an unsafe response? How sensitive is FIN to changes in decoding strategies or model alignment levels, and would the same intervention-selection mechanism remain effective for larger, more strongly aligned LLMs?

3 comments

We thank the reviewer for recognizing the practical motivation and structure of FIN. We address the questions about factorization, decoding, and model settings below.

Q1. Factorization improves behavior preservation and intervention selectivity. Please refer to our general response on the value of FIN’s factorized design, which reports comparisons with direct vulnerability probes.

Q2. The safety–preservation trade-off persists under stochastic decoding. We evaluate LLaMA3-8B-SFT with stochastic decoding (T=0.7T=0.7, top-p=0.9, top-k=0) across three generation seeds (42/43/44), keeping the FIN gate and steering configuration fixed.

DecodingMethodCR ↑SR ↑UL ↓QDR ↓IR
GreedyUnsteered97.551.348.70.00.0
GreedyAlways-On69.270.528.517.9100.0
GreedyFIN95.467.732.28.322.2
SamplingUnsteered96.4 ± 0.546.2 ± 0.652.8 ± 0.70.0 ± 0.00.0
SamplingAlways-On74.9 ± 0.963.8 ± 0.832.7 ± 0.620.5 ± 1.7100.0
SamplingFIN94.5 ± 0.560.4 ± 0.537.4 ± 0.911.4 ± 0.722.2

Metrics are percentages averaged equally across XSTest, OR-Bench, and WildGuardTest. Across both decoding settings, FIN reduces UL relative to unsteered generation while preserving higher CR and lower QDR than Always-On. The observed trade-off persists without retuning the selection rule.

Q3. FIN supports selective intervention across alignment levels and on larger models. We compare LLaMA3-8B Base, SFT, and Instruct to examine different alignment settings, and further evaluate Qwen2.5-14B. FIN uses the same decision structure with model-specific components, allowing intervention selection to reflect each target model’s unsteered behavior.

ModelMethodCR ↑SR ↑UL ↓QDR ↓IR
LLaMA3-8B BaseUnsteered97.015.359.60.00.0
LLaMA3-8B BaseAlways-On86.234.329.941.2100.0
LLaMA3-8B BaseFIN95.725.146.314.520.7
LLaMA3-8B-SFTUnsteered97.551.348.70.00.0
LLaMA3-8B-SFTAlways-On69.270.528.517.9100.0
LLaMA3-8B-SFTFIN95.467.732.28.322.2
LLaMA3-8B InstructUnsteered90.893.95.50.00.0
LLaMA3-8B InstructAlways-On88.995.94.00.5100.0
LLaMA3-8B InstructFIN90.695.04.80.130.7
Qwen2.5-14B BaseUnsteered92.954.542.40.00.0
Qwen2.5-14B BaseAlways-On85.266.431.82.4100.0
Qwen2.5-14B BaseFIN92.858.638.31.019.0

Across LLaMA3-Base, LLaMA3-SFT, and Qwen2.5-14B, FIN improves safety over unsteered generation while preserving benign compliance and response quality better than Always-On. Notably, LLaMA3-Instruct already handles most harmful requests safely, and even Always-On CAA provides smaller additional safety gains. In this setting, FIN maintains safety and benign compliance close to the unsteered model while reducing intervention and quality degradation relative to Always-On, consistent with its goal of preserving existing safe behavior through selective intervention.

Thank you again for your thoughtful questions.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

We would like to follow up on our response and the additional experiments addressing your concerns. As the discussion period is drawing to a close, we would appreciate your feedback on whether our response addresses your main concerns. Please let us know if any questions remain or if further clarification would be helpful.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Follow-up on our response

With the discussion period ending on November 18, we wanted to follow up on our response to your review. We would appreciate your feedback on whether we have addressed your concerns and would be glad to clarify any remaining points. Thank you for your time.

Reviewer_6ZHYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper addresses unnecessary safety steering when an LLM would already respond safely. FIN combines semantic eligibility screening to identify intervention candidates with an untreated vulnerability predictor to estimate their response safety without steering. A calibrated decision rule then selectively cancels steering while controlling the risk of missing necessary interventions within the eligible set.

Critical strengths
  • "Steer Only When Necessary" is a meaningful research question.

  • The paper is exceptionally easy to follow. The introduction moves smoothly from the background and existing approaches to the authors’ specific concern, making the motivation clear without requiring readers to reconstruct the argument themselves. Among the papers in my current review batch, this has been the most pleasant to read.

Critical weaknesses
  1. What does untreated vulnerability actually establish? The paper recognizes that a harmful prompt does not necessarily produce a harmful response. Yet the vulnerability predictor maps prompt representations to safety labels obtained from unsteered responses. My concern is that this mainly learns statistical associations between prompts and a particular model’s behavior. What makes these associations sufficient to determine when intervention is needed?

  2. Should “when to steer” be a token-level decision? A response can begin harmlessly and become harmful later; that transition is not necessarily evident before the first output token. This raises the anticipation problem also faced by streaming safety guardrails such as Qwen3Guard-Stream. In my view, the more useful interpretation of “intervening before generation” is intervening before the tokens that make the response harmful are emitted—not necessarily before any response exists. Why reduce “when” to a single decision at the end of the prompt?

  3. Why are the additional semantic components necessary? If untreated vulnerability reliably predicts unsafe responses under the same safety criterion, why is a separate semantic eligibility gate needed? Why not train a single safety predictor on all requests and use it directly? The current ablations do not establish that this decomposition and its additional components is necessary or preferable to that simpler alternative.

  4. Is the risk calibration applied fairly across methods? Stage III appears to use a learn-then-test-style risk-control procedure, related in spirit to conformal calibration: select a cancellation threshold, then independently certify it using a one-sided Clopper–Pearson bound at specified risk and confidence levels. Were the baselines given comparable calibration data, threshold optimization, and risk constraints? If not, the comparison seems unfair: improvements could reflect the additional calibration procedure rather than the proposed architecture. Comparisons at matched end-to-end unsafe-response rates would help separate these effects.

Questions for authors

see the weaknesses above.

10 comments

We thank the reviewer for the thoughtful questions about the behavioral target and the timing of intervention.

Q1. Vulnerability prediction targets safety when intervention is omitted. FIN asks whether unnecessary intervention can be omitted while preserving the target model's existing safe behavior. If steering is omitted, the resulting response is the model's unsteered response. When that response would already be safe, additional steering is unnecessary for obtaining a safe outcome. The model-specific association learned from actual unsteered responses thus directly targets the behavior relevant to cancellation. Prior work shows that prompt representations contain predictive information about subsequent behavior [1] and separately encode harmfulness and refusal [2], supporting estimation before generation. In FIN, our vulnerability estimator identifies cancellation candidates, and independent certification controls the resulting cancellation miss risk.

Q2. Prompt-level selection addresses the initial intervention decision. We decide whether to enable steering before generation so that the model's existing safe behavior can proceed without unnecessary modification. Although harmfulness may only become apparent as a response unfolds, prompt representations can already provide signals about the model's likely behavior [1,3]. We think the two granularities are complementary: FIN can preserve the model’s existing safe behavior by avoiding unnecessary steering from the outset, while token-level monitoring such as Qwen3Guard-Stream [4] could use the generated prefix to reassess safety as the response unfolds.

Q3. Factorization improves behavior preservation and intervention selectivity. Please refer to our general response on the value of FIN’s factorized design, which reports comparisons with direct vulnerability probes.

Q4. Comparing FIN with baselines using end-to-end UL targets. We select FIN’s cancellation thresholds to minimize the UL gap to frozen CAST/AdaSteer baselines evaluated on the same development requests, without applying the CP criterion. All thresholds are frozen before testing.

ModelMethodCR ↑SR ↑UL ↓QDR ↓IR
LLaMA3-SFTCAST88.260.938.55.864.4
LLaMA3-SFTFIN (vs. CAST)95.267.332.411.427.8
LLaMA3-SFTAdaSteer82.682.117.24.8100.0
LLaMA3-SFTFIN (vs. AdaSteer)95.267.332.411.427.8
MistralCAST88.555.644.113.368.5
MistralFIN (vs. CAST)98.664.135.90.63.2
MistralAdaSteer95.066.833.06.4100.0
MistralFIN (vs. AdaSteer)98.176.823.26.431.1
Qwen2.5-7BCAST72.283.116.03.476.6
Qwen2.5-7BFIN (vs. CAST)86.487.712.00.523.1
Qwen2.5-7BAdaSteer72.981.218.02.6100.0
Qwen2.5-7BFIN (vs. AdaSteer)87.484.814.90.516.3

All test metrics are percentages averaged equally across XSTest, OR-Bench, and WildGuardTest. For each comparison, FIN’s threshold is selected to minimize the development UL gap to the corresponding baseline. Mistral–CAST and both Qwen comparisons meet the 0.5 pp development tolerance. Overall, FIN achieves higher benign compliance with fewer interventions under empirical-UL selection, with model-dependent safety and quality trade-offs.

Thank you again for your thoughtful questions and suggestions.

[1] https://arxiv.org/abs/2502.13329

[2] https://arxiv.org/abs/2507.11878

[3] https://arxiv.org/abs/2502.06258

[4] https://arxiv.org/abs/2510.14276

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

We would like to follow up on our response and the additional experiments addressing your concerns. As the discussion period is drawing to a close, we would appreciate your feedback on whether our response addresses your main concerns. Please let us know if any questions remain or if further clarification would be helpful.

Thank you again for your time and consideration.

Thank you for the thoughtful response and the additional experiments. I read your reply soon after you posted it, and I’m sorry it took me so long to respond. I’ve been torn over two points. First, the paper still does not test when to steer at the token level. The prompt can carry a useful predictive signal, but that alone does not establish that the end of the prompt is the best time to decide in practice. Second, given that streaming guards such as Qwen3Guard-Stream can assess a response as it unfolds, I remain unconvinced that we need to decide whether to intervene before the model has generated a single token.

I found your clarifications and additional experiments valuable and have raised both my Informativeness and Soundness ratings; I’d welcome a candid exchange about where you agree or disagree with my reasoning on these two points.

Reviewer_vMmLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes FIN, a gating framework that decides when to apply safety steering. FIN factorizes the decision into two parts. The first is semantic eligibility: two linear probes for hazard relevance (HRE) and harmful intent (HIE), with a covariance decorrelation penalty (CHD). The second is untreated vulnerability: a logistic-regression predictor of whether the unsteered response would be unsafe, trained on Llama-Guard-3 labels. Steering is cancelled for eligible prompts with low predicted vulnerability, under a threshold certified with a one-sided Clopper–Pearson bound. Experiments on 1.5B–8B models report better benign compliance, general capabilities and quality preservation than CAST, AdaSteer and Always-On, and show that FIN can be combined with other steering methods.

Critical strengths
  • The paper identifies a relevant problem: steering can degrade responses even on harmful requests that the model already handles safely. FIN reduces this degradation (QDR) while preserving utility, in all nine combinations with existing steering methods.
  • FIN does not depend on a particular steering method. It works as a drop-in gate over different steering techniques and recovers utility when a steering method degrades it substantially (e.g., SADI on Qwen2.5-7B).
Critical weaknesses
  • The pipeline is complex for its gains, and its components are not well ablated.** CHD has small or inconsistent effects (no change on Qwen, a higher intervention rate on Mistral; Table 4). The vulnerability feature blocks add marginal gains (Table 19), and cancellation barely helps on Qwen. There is no comparison with simpler gates (e.g., a single vulnerability probe, or Llama-Guard applied to prompts), and no sensitivity analysis for ϵ\epsilon and δ\delta.
  • The evaluation is limited.** The largest model has 8B parameters, and results come from a single run judged only by LLMs. The certification guarantee does not transfer to the test benchmarks.
  • The paper is harder to read than it needs to be: too many acronyms and heavy notations, complex naming (Untreated vulnerability, semantic eligibility) for simple concepts
Questions for authors

See weaknesses

3 comments

We thank the reviewer for the thoughtful feedback and for recognizing the value of preserving already-safe responses and FIN's compatibility with existing steering methods. We address your concerns through the clarifications and additional experiments below.

Q1. Ablations and simpler baselines. CHD is a training regularizer that encourages intent information beyond topic relevance by penalizing within-intent-class covariance between HIE and HRE scores. We include its ablation to examine how this constraint affects semantic gating across models. Its downstream effects are model-dependent: on Qwen, removing CHD leaves the reported metrics unchanged; on Mistral, retaining CHD increases intervention but improves SR from 73.8% to 76.8% and reduces QDR from 6.8% to 6.4% (Table 4). These results characterize when this regularizer affects the safety–preservation balance. For vulnerability features, Qwen's Safe+State subset already matches Full on CR, SR, and QDR, while Full further reduces harmful-request IR from 76.0% to 75.7% (Table 19). We use the same full feature composition across models, where it consistently achieves the lowest harmful-request IR with QDR matching or improving upon every reduced configuration. For cancellation, Qwen's IR decreases from 42.8% to 38.6%, while CR increases by 0.1 points, SR decreases by 0.1 points, and QDR remains unchanged (Table 4), supporting our aim of omitting interventions that provide little additional benefit. Direct-probe comparisons are in our shared response, and the added ϵ/δ\epsilon/\delta sensitivity analyses are in Appendix C.4. We also evaluate a prompt-only Llama-Guard gate that applies the same CAA to prompts labelled harmful. This baseline requires a separate guard-model pass for each prompt, but the additional inference does not yield a consistent performance advantage over FIN.

ModelMethodCR ↑SR ↑UL ↓QDR ↓IR
LLaMA3-8B-SFTLlama-Guard gate96.467.232.012.134.5
LLaMA3-8B-SFTFIN95.467.732.28.322.2
Mistral-7B-Instruct-v0.3Llama-Guard gate97.978.121.97.434.5
Mistral-7B-Instruct-v0.3FIN98.275.624.55.922.2
Qwen2.5-7BLlama-Guard gate89.989.510.40.634.5
Qwen2.5-7BFIN85.191.38.40.938.6

Q2. Evaluation breadth, robustness, and certification scope. We add experiments on Qwen2.5-14B and stochastic decoding on LLaMA3-8B-SFT across three generation seeds.

Model / decodingMethodCR ↑SR ↑UL ↓QDR ↓IR
Qwen2.5-14BUnsteered92.954.542.40.00.0
Qwen2.5-14BAlways-On85.266.431.82.4100.0
Qwen2.5-14BFIN92.858.638.31.019.0
LLaMA3-8B-SFT / samplingUnsteered96.4 ± 0.546.2 ± 0.652.8 ± 0.70.0 ± 0.00.0
LLaMA3-8B-SFT / samplingAlways-On74.9 ± 0.963.8 ± 0.832.7 ± 0.620.5 ± 1.7100.0
LLaMA3-8B-SFT / samplingFIN94.5 ± 0.560.4 ± 0.537.4 ± 0.911.4 ± 0.722.2

Both tables report percentages averaged equally across XSTest, OR-Bench, and WildGuardTest. Sampling uses T=0.7T=0.7, top-p=0.9, top-k=0, seeds 42/43/44. Across both settings, FIN reduces UL relative to unsteered generation while achieving higher CR and lower QDR than Always-On. These results extend the observed safety–preservation trade-off to a larger model and stochastic generation. Appendix C.6 further reports consistent trends with GLM-5.3-Flash and DeepSeek-V4-Flash, supporting the robustness of FIN’s safety–preservation findings across evaluators. Regarding certification, the guarantee controls conditional cancellation risk on the certification distribution, while cross-benchmark behavior is evaluated empirically. Appendix C.5 shows that unsafe outputs on the cancellation path account for 0–3.0% of all harmful requests across nine model–benchmark settings, providing direct evidence of cancellation behavior on the test distributions.

Q3. Clarity of presentation. We will streamline the method presentation around two questions: whether a request concerns a hazard and expresses harmful intent, and whether the target model would respond unsafely without steering. We will also reduce acronym use and repeated notation in explanatory passages, making the distinction between request semantics and model behavior easier to follow.

Thank you again for your feedback. We welcome any further questions.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

We would like to follow up on our response and the additional experiments addressing your concerns. As the discussion period is drawing to a close, we would appreciate your feedback on whether our response addresses your main concerns. Please let us know if any questions remain or if further clarification would be helpful.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Follow-up on our response

With the discussion period ending on November 18, we wanted to follow up on our response to your review. We would appreciate your feedback on whether we have addressed your concerns and would be glad to clarify any remaining points. Thank you for your time.

Comments on the paper

1 comment

Response to All Reviewers: On FIN’s Factorized Design

We compare FIN with two direct vulnerability probes to address whether a single response-safety predictor can replace the factorized design. Probe-S2 uses the same training data as FIN’s Stage-II predictor. Probe-Combined additionally includes the training data used by HRE and HIE, labelled by the safety of the target model’s own unsteered responses. Both probes use the same prompt hidden-state features, and hyperparameter-selection procedure, without semantic gating. All methods use the same CAA intervention.

ModelMethodCR ↑ (%)SR ↑ (%)QDR ↓ (%)IR (%)
LLaMA3-8B-SFTProbe-S281.065.611.261.0
LLaMA3-8B-SFTProbe-Combined92.665.99.427.1
LLaMA3-8B-SFTFIN95.467.78.322.2
Mistral-7B-Instruct-v0.3Probe-S287.880.27.877.1
Mistral-7B-Instruct-v0.3Probe-Combined90.781.17.657.4
Mistral-7B-Instruct-v0.3FIN98.275.65.922.2
Qwen2.5-7BProbe-S252.996.11.775.9
Qwen2.5-7BProbe-Combined70.294.31.350.1
Qwen2.5-7BFIN85.191.30.938.6

Results are averaged equally across XSTest, OR-Bench, and WildGuardTest.

FIN achieves higher benign compliance with fewer interventions across all three models. On Mistral and Qwen, the direct probes achieve higher safe-refusal rates but also intervene more frequently and preserve less benign behavior. These results support explicitly separating the semantic scope of intervention from model-specific vulnerability to improve intervention selectivity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.