- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes an adaptive approach to refusal vector application, utilizing a simple, discrete-time survival analysis model to decide when to stop applying the refusal vector. Authors report similar safety gains compared to persistent application of the refusal vector, while reducing over-intervention, over-refusal, capability degradation. and inference overhead on four different LLMs.
Critical strengths
- The problem is interesting and the use of survival analysis is well motivated.
- The actual inference time mechanism is simple and efficient. I particularly like the interpretability of the parameters.
- Extensive evaluation using different models, attacks, judges, etc.
- Over refusal improvements are promising.
Critical weaknesses
- Survival target is ill-defined and the intervention may induce informative censoring (Section 3.3)
Setting a response as evaluable only if it reaches the designated width makes it so that the longest evaluable width depends on the response length, which steering itself affects. For example, a refusal ending after 10 tokens makes 16, 32, and 64 inadmissible.This induces informative censoring and makes the distribution estimated be different than "the shortest steering duration that produces a safe response". Utilizing a single noisy output per width considered and defining the survival time as a monotone closure so these times, which is sensitive to width grid coarseness, both compound the murkiness of this regression target.
Additionally, authors should consider reporting more recent survival evaluation metrics, accounting for for Inverse Probability Censoring Weights.
- Questionable practical significance of latency gains (Table 2)
Persistent steering already incurs less than 0.2% runtime overhead. Adaptive withdrawal saves less than 0.1% runtime in all settings. These savings are negligible and should not be considered a significant result, especially considering that multiple efficient activation vector inference engines exist and may cause relative gains to appear even smaller. Gains relative to other methods largely reflect architecture differences.
- MMLU evaluation in Table 1 of Appendix A could be the most significant piece of evidence for the method, but it's confounded
The MMLU results show that the adaptive approach might almost completely recover the accuracy lost under persistent steering, but the evaluation is very non-standard, employing steering during an analysis stage of the eval. The authors should score identical analyses without steering so that results are comparable. The unusually low Gemma and Llama MMLU accuracies also warrant investigation and explanation.
Questions for authors
Reproducibility: not all code needed for reproducing the paper results is present in the submission. Can this be amended?
4 comments
Official Comment Part 2
W2. Practical significance of runtime
Persistent steering already adds less than 0.2% runtime (Table 2), so the runtime saving is negligible, and the revised manuscript no longer presents it as a result. The contribution is less unnecessary intervention and less over-refusal. On the primary evaluation, adaptive withdrawal removes 38.5–72.5% of the intervention while retaining 97.2–100.0% of persistent-steering safety on complete responses, and it raises the XSTest answer rate over persistent steering by 5.8–17.6% on Gemma, Llama and Qwen3-4B (Table 1).
W3. MMLU evaluation and implementation
We take your comment on "Table 1 of Appendix A" to refer to the MMLU results in Table 1 under the protocol of Appendix A.4.2. Thank you for identifying the confound in that evaluation. We re-ran MMLU on the same 1,026 questions with two protocol changes. The model generates its full analysis with no cut-off at 96 tokens, and, as you suggested, the answer is scored without steering for every method.
| Model | Unsteered | Persistent steering | Adaptive withdrawal |
|---|---|---|---|
| Gemma-1.1-7B-IT | 46.2% | 44.2% | 45.1% |
| Llama-3.1-8B-Instruct | 57.2% | 54.7% | 56.1% |
| Qwen3-4B | 68.4% | 65.9% | 68.2% |
| Qwen3-8B | 69.4% | 67.1% | 68.2% |
Under the matched protocol the differences are small. Adaptive withdrawal is 0.9–2.3% above persistent steering on every model and within 1.2% of unsteered accuracy. These differences are modest. On 1,026 questions each accuracy has a standard error of about 1.5%, so a single-model difference of 0.9–2.3% is comparable to the sampling error of the estimates. The revised manuscript therefore presents MMLU as evidence that withdrawal does not reduce accuracy relative to persistent steering, and no longer presents it as a gain. The low Gemma and Llama accuracies came largely from the 96-token limit. With full analyses, unsteered accuracy rises to 46.2% and 57.2%.
Reproducibility
We have updated the supplementary code package to include the implementation and evaluation code. We will publicly release the complete code and experimental configurations upon acceptance.
Based on the target-robustness analyses, the weighted metrics, the binary-classifier experiment and the matched MMLU evaluation, we hope we have addressed your concerns, and we hope you will consider raising your score.
Thank you for the thorough response and the additional experiments. I appreciate the candour with which the new analyses are reported, and the revised claims are better supported than the original ones.
I am keeping my score, since the revisions narrows contributions, and the method itself needs more work.
Response to Reviewer CJSm's follow-up comment
Thank you for the follow-up, and for noting that the revised claims are better supported. We believe the strengths of the revised paper outweigh its remaining limitations, which the paper states openly. We summarise them below and hope you will consider them in your final assessment.
The core result is unchanged. Adaptive withdrawal removes 38.5–72.5% of the intervention while retaining 97.2–100.0% of persistent-steering safety on complete responses, and all four models pass this comparison under both HarmBench and Qwen2.5 (Table 7). It also raises the XSTest answer rate over persistent steering by 5.8–17.6% on Gemma, Llama and Qwen3-4B (Table 1), which is the over-refusal improvement you described as promising.
Your comments also made the paper more rigorous. The binary-classifier experiment passes the complete-response criterion on all four models without using the duration target, so the finding does not depend on the survival formulation. The weighted metrics stay close to the originals (Table 28).
The strengths in your review remain after revision, namely a simple and interpretable inference-time rule and an extensive evaluation across models, attacks and judges. We believe these results outweigh the remaining limitations, and we hope you will weigh them in your final assessment.
Official Comment Part 1
Thank you for recognising the practical motivation, the interpretable predictor, and the over-refusal results. We tested each concern you raised. The main complete-response result is supported by an experiment that does not use the duration target at all, and the revised manuscript states the remaining limitations of that target. All modifications are marked in red.
W1. Withdrawal targets and incomplete observations
On censoring, we re-analysed every tested duration for the 120 calibration prompts per model. The 37 cases with no identified safe withdrawal time all remained evaluable at 64 tokens, so none was censored by an early ending. Counting responses that end safely before the withdrawal point as safe changes none of the calibration targets or fitted policies. Steering still affects response length, so censoring may not be random, and the revised Section 3.3 says so.
On target noise, our new analysis supports your concern. Across five sampled runs, 35–42 of the 120 calibration prompts per model change their target, and the revised manuscript describes the target as one realisation of a noisy quantity. The policies fitted on these targets still meet the complete-response criterion in the primary held-out validation for every model (Table 7).
On the grid and the monotone closure, a finer duration grid does not consistently improve performance. The closure does matter. Replacing it with a first-safe target changes the selected policies and can reduce safety, for example to 94.00% on Gemma against 99.75% for persistent steering, which fails our criterion. The closure is therefore the conservative choice, and we report this sensitivity.
As independent evidence that does not use the duration target at all, we trained a binary classifier that chooses between withdrawal at two tokens and persistent steering. On the 400 held-out prompts per model it passes the multiplicity-adjusted complete-response criterion on all four models and removes 37.7–62.4% of the intervention. Its continuation comparisons pass on Gemma and Qwen3-8B only. Useful early withdrawal on complete responses therefore does not depend on the survival formulation.
Following your suggestion, we recalculated the prediction metrics with inverse-probability-of-censoring weights. Original values are from Table 12.
The weighted values stay close to the originals. The revised manuscript specifies the weighting assumptions and makes clear that we predict the tested withdrawal targets, not safety probabilities across repeated generations.
(continued in Part 2)