- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for determining when activation steering is necessary for LLM safety. The key insight is that harmful requests do not always require additional intervention, as the target model may already produce safe responses without steering. FIN addresses this by separating semantic eligibility from model-specific untreated vulnerability, using internal representations to predict whether intervention is needed before generation. A statistical risk-control mechanism is further introduced to limit unsafe intervention cancellation.
Critical strengths
The paper addresses a practical problem, highlighting that unnecessary steering can degrade response quality even for harmful requests that models already handle safely. The proposed framework is well structured, and the incorporation of statistical risk control adds methodological rigor to the intervention decision. The experimental evaluation is relatively comprehensive, covering different model families, safety benchmarks, general capabilities, and compatibility with existing steering methods.
Critical weaknesses
The main limitation is that the methodological novelty is somewhat incremental, as several components rely on established representation-based classifiers and statistical calibration techniques. Although the factorized formulation is intuitive, the extent to which this decomposition provides benefits beyond simpler vulnerability-based intervention strategies remains insufficiently established. Additionally, the statistical guarantee applies specifically to intervention cancellation among eligible requests with unsafe unsteered responses, rather than providing an end-to-end safety guarantee.
Questions for authors
Could the authors further clarify the necessity of explicitly separating semantic eligibility and untreated vulnerability, particularly compared with a simpler approach that directly predicts whether the unsteered model will produce an unsafe response? How sensitive is FIN to changes in decoding strategies or model alignment levels, and would the same intervention-selection mechanism remain effective for larger, more strongly aligned LLMs?
3 comments
Follow-up on our response
Dear Reviewer,
We would like to follow up on our response and the additional experiments addressing your concerns. As the discussion period is drawing to a close, we would appreciate your feedback on whether our response addresses your main concerns. Please let us know if any questions remain or if further clarification would be helpful.
Thank you again for your time and consideration.
Follow-up on our response
With the discussion period ending on November 18, we wanted to follow up on our response to your review. We would appreciate your feedback on whether we have addressed your concerns and would be glad to clarify any remaining points. Thank you for your time.
We thank the reviewer for recognizing the practical motivation and structure of FIN. We address the questions about factorization, decoding, and model settings below.
Q1. Factorization improves behavior preservation and intervention selectivity. Please refer to our general response on the value of FIN’s factorized design, which reports comparisons with direct vulnerability probes.
Q2. The safety–preservation trade-off persists under stochastic decoding. We evaluate LLaMA3-8B-SFT with stochastic decoding (T=0.7, top-p=0.9, top-k=0) across three generation seeds (42/43/44), keeping the FIN gate and steering configuration fixed.
Metrics are percentages averaged equally across XSTest, OR-Bench, and WildGuardTest. Across both decoding settings, FIN reduces UL relative to unsteered generation while preserving higher CR and lower QDR than Always-On. The observed trade-off persists without retuning the selection rule.
Q3. FIN supports selective intervention across alignment levels and on larger models. We compare LLaMA3-8B Base, SFT, and Instruct to examine different alignment settings, and further evaluate Qwen2.5-14B. FIN uses the same decision structure with model-specific components, allowing intervention selection to reflect each target model’s unsteered behavior.
Across LLaMA3-Base, LLaMA3-SFT, and Qwen2.5-14B, FIN improves safety over unsteered generation while preserving benign compliance and response quality better than Always-On. Notably, LLaMA3-Instruct already handles most harmful requests safely, and even Always-On CAA provides smaller additional safety gains. In this setting, FIN maintains safety and benign compliance close to the unsteered model while reducing intervention and quality degradation relative to Always-On, consistent with its goal of preserving existing safe behavior through selective intervention.
Thank you again for your thoughtful questions.