- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether a model can be steered away from an undesirable behavior using only examples of that behavior, without constructing desirable responses. COUNTERSTEER-GRAD averages normalized activation gradients of the undesirable-response loss and applies the opposite of the fitting direction. COUNTERSTEER-LORA trains a temporary adapter on the same type of data, then reverses its mean activation shift. The theory establishes local suppression of GRAD's matched teacher-forced score and gives an alignment condition for LORA. Experiments examine over-refusal and sycophancy across several model families.
Critical strengths
- The asymmetric-supervision setting is compelling: unwanted responses are often easier to identify than prompt-specific good answers. This gives the paper a clear, practical research question.
- COUNTERSTEER-GRAD turns that question into a simple, training-free vector construction. Its direct use of activation gradients makes the method easy to understand and analyze.
- The paper explores both a local gradient route and an adaptation route, while clearly limiting its formal claims to local likelihood behavior. This careful boundary strengthens the presentation.
Critical weaknesses
- The motivation needs a sharper success criterion. Negative examples identify a behavior to avoid, but do not say which of the many alternative responses would count as useful control.
- The method assumes averaged gradients from unwanted examples form a stable behavioral direction. The paper does not explain why that direction should capture refusal or sycophancy rather than shared wording or prompt content, or when it should transfer.
- The framework's common principle remains loose. GRAD has a local identity for its matched score, whereas LORA needs a separate alignment condition; the shared insight beyond prior one-sided steering is therefore unclear.
Questions for authors
- Under what assumptions does the average negative-only gradient isolate a shared behavior rather than incidental lexical or prompt features, and transfer to new contexts? Is the intended contribution limited to suppressing target completions, or is there a stronger claim about shifting the model toward appropriate alternatives? The answer would change how I assess the paper's central contribution.
- What falsifiable condition, shared across GRAD and LORA, predicts successful behavioral steering from negative examples? For example, can the authors relate the constructed vectors to a task-relevant score gradient on unseen inputs and explain when the LORA alignment condition holds? This would clarify whether COUNTERSTEER is a general mechanism or two related heuristics.
4 comments
Rebuttal (2/3): Weakness 2 and Question 1
- The method assumes averaged gradients from unwanted examples form a stable behavioral direction. The paper does not explain why that direction should capture refusal or sycophancy rather than shared wording or prompt content, or when it should transfer.
Reply:
This reply also addresses Question 1.
We agree that transfer should be tested rather than inferred from averaging. We now separate transfer across prompts from transfer across target formulations, using the same 182 held-out Qwen2.5-7B prompts in both conditions.
Matched-template prompt transfer. With fixed construction refusal templates on unseen prompts, both Grad and LoRA have positive unit-direction NLL margins on every example. Mean margins and item-bootstrap 95% intervals are:
| Targets | Grad | LoRA |
|---|---|---|
| Construction templates, n=512 | 0.13716 [0.13378, 0.14078] | 0.05125 [0.05007, 0.05252] |
| Held-out prompts + construction templates, n=182 | 0.08282 [0.07766, 0.08808] | 0.03136 [0.02948, 0.03331] |
The effect also persists at finite strength: mean NLL increases at every evaluated positive strength up to the frozen diagnostic endpoint. Endpoint ΔNLL is +2.561/+1.273 for Grad/LoRA on construction targets and +1.797/+0.965 on held-out matched-template targets. This demonstrates transfer beyond the construction prompts and beyond an infinitesimal step.
Target-form sensitivity. On those identical held-out prompts paired with naturally generated baseline refusal responses, average margins become negative (−0.00655 Grad; −0.00133 LoRA). Therefore, prompt transfer under matched targets does not establish independence from shared refusal wording. We will describe the direction as a target-conditioned construction rather than a universally isolated “pure behavior” feature.
The template-diversity experiment independently supports this interpretation: replacing ten templates with one retains approximately 38–124% of the reference refusal-reduction effect, depending on wording. With five templates, direction similarity to the ten-template reference reaches 0.94 on Qwen7B and 0.97 on Qwen3B. The practical implication is to include multiple expressions of the unwanted behavior.
Appropriate alternatives. Our theoretical result concerns local suppression of fixed undesirable targets. Whether generation moves toward a useful alternative is evaluated through correctness, safety, and capability, as defined in our reply to Weakness 1. Pushback provides a particularly clear test: final accuracy counts both preservation of correct answers and recovery from incorrect ones.
| Model | Original accuracy after pushback | Grad | LoRA |
|---|---|---|---|
| Qwen2.5-3B | 56.0 | 60.0 | 64.7 |
| Qwen2.5-7B | 67.0 | 75.7 | 70.3 |
| Llama-3.1-8B | 32.3 | 37.3 | 36.0 |
All entries use the same denominator of 300 examples. On Llama, fix rates remain near or above baseline (44.2% Grad / 50.5% LoRA versus 46.7%), alongside better final accuracy. On Qwen, Grad's lower fix rates (4.5%/5.4% on 3B/7B) indicate reduced answer revision; LoRA retains more revision (32.5%/21.7%). These results separate useful preservation from correction and show why both must be measured.
The NLL measurements use separate frozen diagnostic configurations: Grad L23, α=16 with unit additive steering, and historical LoRA L23, α=0.1 with activation-norm-scaled steering. They are not the main-table operating points. Overall, the evidence supports matched-target prompt transfer and improved behavioral outcomes, while identifying target formulation as an explicit condition for transfer.
Rebuttal (3/3): Weakness 3 and Question 2
- The framework's common principle remains loose. GRAD has a local identity for its matched score, whereas LORA needs a separate alignment condition; the shared insight beyond prior one-sided steering is therefore unclear.
Reply:
This reply also addresses Question 2.
The common principle is alignment between the constructed reversal direction and the undesirable-target score gradient under the specified intervention. This is directly measurable for either construction and can succeed or fail on unseen inputs.
For a frozen reverse unit direction and evaluation distribution , define
Here is response-only NLL of the fixed undesirable target. predicts an initial increase in undesirable-target NLL, and therefore local suppression of that target score. This supplies one testable condition shared by Grad and LoRA.
Why the constructions differ. Grad computes the exact derivative of an offset tied across the intervention sites:
For the matched construction objective with weights fixed at the base model, , so the NLL derivative along is whenever . Over-refusal uses ; sycophancy uses reciprocal gradient norms for nonzero examples and zero otherwise. Thus Grad's construction alignment follows from its definition.
LoRA instead constructs a candidate direction from finite adapted-minus-base prompt-final shifts. Successful adapter fitting does not by itself establish its alignment after reversal and aggregation. Our extraction-transfer analysis makes that requirement explicit:
The raw reversed-mean margin is bounded by the hypothetical position-specific reversal margin , minus a term combining score-gradient magnitude and extraction mismatch . Small extraction mismatch together with a sufficiently positive local margin provides a sufficient condition. Final unit normalization divides both the margin and its bound by the raw vector norm, preserving the positivity condition.
Empirical falsifiability. Our joint reply to Weakness 2 and Question 1 reports both outcomes on the same held-out prompts. Matched refusal templates give positive mean alignment for Grad and LoRA (0.08282 and 0.03136), supported by positive finite-strength NLL changes. Natural refusal targets yield negative mean alignment for both. The condition is therefore not automatically assumed to hold: it predicts the local target-score effect and detects failure under target-distribution shift.
This is the shared insight: construct a candidate suppression direction, characterize its relationship to the deployed intervention, and test alignment on the intended targets. Grad obtains exact construction alignment; LoRA offers a finite-adaptation route with a measurable alignment and extraction-transfer requirement. Behavioral usefulness is then assessed separately through the explicit task criteria in our reply to Weakness 1. We will frame the framework around this common testable principle and include the direct margin and transfer measurements in the revision.
Questions for Authors
- Under what assumptions does the average negative-only gradient isolate a shared behavior rather than incidental lexical or prompt features, and transfer to new contexts? Is the intended contribution limited to suppressing target completions, or is there a stronger claim about shifting the model toward appropriate alternatives? The answer would change how I assess the paper's central contribution.
Reply:
Please see our joint reply to Weakness 2 and Question 1 above for transfer diagnostics and appropriate alternatives.
- What falsifiable condition, shared across GRAD and LORA, predicts successful behavioral steering from negative examples? For example, can the authors relate the constructed vectors to a task-relevant score gradient on unseen inputs and explain when the LORA alignment condition holds? This would clarify whether COUNTERSTEER is a general mechanism or two related heuristics.
Reply:
Please see our joint reply to Weakness 3 and Question 2 above for the shared alignment criterion and held-out tests.
We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience.
References
[1] Sharma and Trivedi. COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics. ICLR 2026.
[2] Wang, Yao, and Chen. Causal-Steer: Disentangled Continuous Style Control without Parallel Corpora. ICLR 2026.
[3] Maia Polo et al. tinyBenchmarks: Evaluating LLMs with Fewer Examples. ICML 2024.
Any Questions from Reviewer T7oa on Our Rebuttal?
Dear Reviewer T7oa,
We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Paper Discussion Phase will end in less than 12 hours from this comment, i.e., 11:59 pm AoE on November 18. We are happy to answer any remaining questions you may have or provide further clarification before then, but we will no longer be able to respond once the discussion phase has concluded.
If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments!
Sincerely,
Authors
Rebuttal (1/3): Weakness 1: Success criterion
Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.
Critical Weaknesses
Reply:
Thank you for asking us to sharpen the success criterion. Useful control means reducing the unwanted behavior while preserving task correctness, safety, and relevant capabilities. Negative targets specify what to suppress; the evaluation must establish whether the resulting behavior is useful.
For over-refusal, we assess refusal on benign requests together with harmful-response rates and general capability. For sycophancy, we assess endorsement of incorrect user suggestions, correct-answer recovery, accuracy after unsupported pushback, and capability retention. These complementary outcomes distinguish useful steering from simply refusing, becoming contrarian, or never revising an answer.
Our results provide evidence on these dimensions. On Qwen2.5-7B, Grad reduces incorrect-belief agreement from 27.8% to 16.2% and increases correction from 57.3% to 64.2%, while tinyBench accuracy remains 64.3% versus 65.2% unsteered. Thus reduced agreement is accompanied by more correct answers, rather than merely rejecting the user's suggestion.
The new closely related baselines strengthen this result. On Qwen, COLD-Kernel gives 29.5% agreement / 56.5% correction, compared with Grad's 16.2% / 64.2%. LoRA improves agreement, correction, and tinyBench over the adapted single-source Causal-Steer baseline on both evaluated models:
Over-refusal success is assessed jointly with safety. For example, Grad lowers Qwen7B benign refusal on OR-Hard/XSTest from 17.1%/4.4% to 11.5%/3.2%, with HarmBench harmful behavior increasing from 27.3% to 30.3%. We report this tradeoff directly rather than equate lower refusal with universally better behavior. The two-point validation safety threshold is a selection rule, not a guarantee on held-out harmfulness.
We will make these task-specific criteria explicit in the motivation and evaluation discussion. Our claim is supported by improvements in the intended behavioral outcomes together with measured capability and safety tradeoffs; negative-target suppression alone is not the success criterion.