acceptodds
Under review as a conference paper at ICLR 2027

CounterSteer: Activation Steering from Undesirable Examples Alone

10 views

Abstract

Activation steering enables inference-time control of large language models by intervening directly on their hidden representations. Existing methods typically derive steering directions by contrasting desired and undesired behavior, which requires a carefully constructed desired dataset that can be difficult or costly to obtain. We introduce CounterSteer, a one-sided activation-steering framework that constructs steering directions using only examples of the behavior to suppress, without requiring desired counterparts or contrastive pairs. Our primary method, CounterSteer-Grad, constructs a reusable steering vector by reversing the local activation-space direction that would increase the likelihood of undesired examples. It aggregates per-sequence activation gradients of the undesired-data loss in the frozen base model, requiring neither adapter training nor parameter updates. We also introduce CounterSteer-LoRA, a variant that goes beyond this one-step probe by training a disposable low-rank adapter on undesired examples and reversing the mean same-input activation shift between the adapted and base models. The adapter is discarded before inference. Both methods apply a precomputed vector to the base model's activations. Theoretically, we establish local suppression for the fixed-weight construction score of CounterSteer-Grad and characterize the additional alignment condition required for the LoRA-based variant. Across a range of models and behaviors, CounterSteer-Grad and CounterSteer-LoRA match or outperform contrastive steering baselines despite requiring only undesired examples.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, activation steering, representation engineering, one-sided data, behavioral control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T7oaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether a model can be steered away from an undesirable behavior using only examples of that behavior, without constructing desirable responses. COUNTERSTEER-GRAD averages normalized activation gradients of the undesirable-response loss and applies the opposite of the fitting direction. COUNTERSTEER-LORA trains a temporary adapter on the same type of data, then reverses its mean activation shift. The theory establishes local suppression of GRAD's matched teacher-forced score and gives an alignment condition for LORA. Experiments examine over-refusal and sycophancy across several model families.

Critical strengths
  1. The asymmetric-supervision setting is compelling: unwanted responses are often easier to identify than prompt-specific good answers. This gives the paper a clear, practical research question.
  2. COUNTERSTEER-GRAD turns that question into a simple, training-free vector construction. Its direct use of activation gradients makes the method easy to understand and analyze.
  3. The paper explores both a local gradient route and an adaptation route, while clearly limiting its formal claims to local likelihood behavior. This careful boundary strengthens the presentation.
Critical weaknesses
  1. The motivation needs a sharper success criterion. Negative examples identify a behavior to avoid, but do not say which of the many alternative responses would count as useful control.
  2. The method assumes averaged gradients from unwanted examples form a stable behavioral direction. The paper does not explain why that direction should capture refusal or sycophancy rather than shared wording or prompt content, or when it should transfer.
  3. The framework's common principle remains loose. GRAD has a local identity for its matched score, whereas LORA needs a separate alignment condition; the shared insight beyond prior one-sided steering is therefore unclear.
Questions for authors
  1. Under what assumptions does the average negative-only gradient isolate a shared behavior rather than incidental lexical or prompt features, and transfer to new contexts? Is the intended contribution limited to suppressing target completions, or is there a stronger claim about shifting the model toward appropriate alternatives? The answer would change how I assess the paper's central contribution.
  2. What falsifiable condition, shared across GRAD and LORA, predicts successful behavioral steering from negative examples? For example, can the authors relate the constructed vectors to a task-relevant score gradient on unseen inputs and explain when the LORA alignment condition holds? This would clarify whether COUNTERSTEER is a general mechanism or two related heuristics.
4 comments
AuthorsOpenReview ↗

Rebuttal (1/3): Weakness 1: Success criterion

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.

Critical Weaknesses

  1. The motivation needs a sharper success criterion. Negative examples identify a behavior to avoid, but do not say which of the many alternative responses would count as useful control.

Reply:

Thank you for asking us to sharpen the success criterion. Useful control means reducing the unwanted behavior while preserving task correctness, safety, and relevant capabilities. Negative targets specify what to suppress; the evaluation must establish whether the resulting behavior is useful.

For over-refusal, we assess refusal on benign requests together with harmful-response rates and general capability. For sycophancy, we assess endorsement of incorrect user suggestions, correct-answer recovery, accuracy after unsupported pushback, and capability retention. These complementary outcomes distinguish useful steering from simply refusing, becoming contrarian, or never revising an answer.

Our results provide evidence on these dimensions. On Qwen2.5-7B, Grad reduces incorrect-belief agreement from 27.8% to 16.2% and increases correction from 57.3% to 64.2%, while tinyBench accuracy remains 64.3% versus 65.2% unsteered. Thus reduced agreement is accompanied by more correct answers, rather than merely rejecting the user's suggestion.

The new closely related baselines strengthen this result. On Qwen, COLD-Kernel gives 29.5% agreement / 56.5% correction, compared with Grad's 16.2% / 64.2%. LoRA improves agreement, correction, and tinyBench over the adapted single-source Causal-Steer baseline on both evaluated models:

ModelMethodAgreement ↓Correction ↑tinyBench ↑
Qwen2.5-7BCausal26.858.963.5
LoRA20.959.964.7
Llama-3.1-8BCausal15.171.960.8
LoRA13.972.761.7

Over-refusal success is assessed jointly with safety. For example, Grad lowers Qwen7B benign refusal on OR-Hard/XSTest from 17.1%/4.4% to 11.5%/3.2%, with HarmBench harmful behavior increasing from 27.3% to 30.3%. We report this tradeoff directly rather than equate lower refusal with universally better behavior. The two-point validation safety threshold is a selection rule, not a guarantee on held-out harmfulness.

We will make these task-specific criteria explicit in the motivation and evaluation discussion. Our claim is supported by improvements in the intended behavioral outcomes together with measured capability and safety tradeoffs; negative-target suppression alone is not the success criterion.

AuthorsOpenReview ↗

Rebuttal (2/3): Weakness 2 and Question 1

  1. The method assumes averaged gradients from unwanted examples form a stable behavioral direction. The paper does not explain why that direction should capture refusal or sycophancy rather than shared wording or prompt content, or when it should transfer.

Reply:

This reply also addresses Question 1.

We agree that transfer should be tested rather than inferred from averaging. We now separate transfer across prompts from transfer across target formulations, using the same 182 held-out Qwen2.5-7B prompts in both conditions.

Matched-template prompt transfer. With fixed construction refusal templates on unseen prompts, both Grad and LoRA have positive unit-direction NLL margins on every example. Mean margins and item-bootstrap 95% intervals are:

TargetsGradLoRA
Construction templates, n=5120.13716 [0.13378, 0.14078]0.05125 [0.05007, 0.05252]
Held-out prompts + construction templates, n=1820.08282 [0.07766, 0.08808]0.03136 [0.02948, 0.03331]

The effect also persists at finite strength: mean NLL increases at every evaluated positive strength up to the frozen diagnostic endpoint. Endpoint ΔNLL is +2.561/+1.273 for Grad/LoRA on construction targets and +1.797/+0.965 on held-out matched-template targets. This demonstrates transfer beyond the construction prompts and beyond an infinitesimal step.

Target-form sensitivity. On those identical held-out prompts paired with naturally generated baseline refusal responses, average margins become negative (−0.00655 Grad; −0.00133 LoRA). Therefore, prompt transfer under matched targets does not establish independence from shared refusal wording. We will describe the direction as a target-conditioned construction rather than a universally isolated “pure behavior” feature.

The template-diversity experiment independently supports this interpretation: replacing ten templates with one retains approximately 38–124% of the reference refusal-reduction effect, depending on wording. With five templates, direction similarity to the ten-template reference reaches 0.94 on Qwen7B and 0.97 on Qwen3B. The practical implication is to include multiple expressions of the unwanted behavior.

Appropriate alternatives. Our theoretical result concerns local suppression of fixed undesirable targets. Whether generation moves toward a useful alternative is evaluated through correctness, safety, and capability, as defined in our reply to Weakness 1. Pushback provides a particularly clear test: final accuracy counts both preservation of correct answers and recovery from incorrect ones.

ModelOriginal accuracy after pushbackGradLoRA
Qwen2.5-3B56.060.064.7
Qwen2.5-7B67.075.770.3
Llama-3.1-8B32.337.336.0

All entries use the same denominator of 300 examples. On Llama, fix rates remain near or above baseline (44.2% Grad / 50.5% LoRA versus 46.7%), alongside better final accuracy. On Qwen, Grad's lower fix rates (4.5%/5.4% on 3B/7B) indicate reduced answer revision; LoRA retains more revision (32.5%/21.7%). These results separate useful preservation from correction and show why both must be measured.

The NLL measurements use separate frozen diagnostic configurations: Grad L23, α=16 with unit additive steering, and historical LoRA L23, α=0.1 with activation-norm-scaled steering. They are not the main-table operating points. Overall, the evidence supports matched-target prompt transfer and improved behavioral outcomes, while identifying target formulation as an explicit condition for transfer.

AuthorsOpenReview ↗

Rebuttal (3/3): Weakness 3 and Question 2

  1. The framework's common principle remains loose. GRAD has a local identity for its matched score, whereas LORA needs a separate alignment condition; the shared insight beyond prior one-sided steering is therefore unclear.

Reply:

This reply also addresses Question 2.

The common principle is alignment between the constructed reversal direction and the undesirable-target score gradient under the specified intervention. This is directly measurable for either construction and can succeed or fail on unseen inputs.

For a frozen reverse unit direction dd and evaluation distribution DD, define

AD(d)=Ez∼D[∇uLz(0)⊤d].A_D(d)=\mathbb E_{z\sim D}[\nabla_uL_z(0)^\top d].

Here LzL_z is response-only NLL of the fixed undesirable target. AD(d)>0A_D(d)>0 predicts an initial increase in undesirable-target NLL, and therefore local suppression of that target score. This supplies one testable condition shared by Grad and LoRA.

Why the constructions differ. Grad computes the exact derivative of an offset tied across the intervention sites:

qi=∑k∈Ki∇hi,kLi,v=N−1∑iwiqi,d=v/∥v∥2.q_i=\sum_{k\in K_i}\nabla_{h_{i,k}}L_i,\quad v=N^{-1}\sum_iw_iq_i,\quad d=v/\|v\|_2.

For the matched construction objective with weights fixed at the base model, v=∇uLw(0)v=\nabla_uL_w(0), so the NLL derivative along dd is ∥v∥2>0\|v\|_2>0 whenever v≠0v\ne0. Over-refusal uses wi=1w_i=1; sycophancy uses reciprocal gradient norms for nonzero examples and zero otherwise. Thus Grad's construction alignment follows from its definition.

LoRA instead constructs a candidate direction from finite adapted-minus-base prompt-final shifts. Successful adapter fitting does not by itself establish its alignment after reversal and aggregation. Our extraction-transfer analysis makes that requirement explicit:

m≥mloc−GE.m\ge m_{\mathrm{loc}}-GE.

The raw reversed-mean margin mm is bounded by the hypothetical position-specific reversal margin mlocm_{\mathrm{loc}}, minus a term combining score-gradient magnitude GG and extraction mismatch EE. Small extraction mismatch together with a sufficiently positive local margin provides a sufficient condition. Final unit normalization divides both the margin and its bound by the raw vector norm, preserving the positivity condition.

Empirical falsifiability. Our joint reply to Weakness 2 and Question 1 reports both outcomes on the same held-out prompts. Matched refusal templates give positive mean alignment for Grad and LoRA (0.08282 and 0.03136), supported by positive finite-strength NLL changes. Natural refusal targets yield negative mean alignment for both. The condition is therefore not automatically assumed to hold: it predicts the local target-score effect and detects failure under target-distribution shift.

This is the shared insight: construct a candidate suppression direction, characterize its relationship to the deployed intervention, and test alignment on the intended targets. Grad obtains exact construction alignment; LoRA offers a finite-adaptation route with a measurable alignment and extraction-transfer requirement. Behavioral usefulness is then assessed separately through the explicit task criteria in our reply to Weakness 1. We will frame the framework around this common testable principle and include the direct margin and transfer measurements in the revision.

Questions for Authors

  1. Under what assumptions does the average negative-only gradient isolate a shared behavior rather than incidental lexical or prompt features, and transfer to new contexts? Is the intended contribution limited to suppressing target completions, or is there a stronger claim about shifting the model toward appropriate alternatives? The answer would change how I assess the paper's central contribution.

Reply:

Please see our joint reply to Weakness 2 and Question 1 above for transfer diagnostics and appropriate alternatives.

  1. What falsifiable condition, shared across GRAD and LORA, predicts successful behavioral steering from negative examples? For example, can the authors relate the constructed vectors to a task-relevant score gradient on unseen inputs and explain when the LORA alignment condition holds? This would clarify whether COUNTERSTEER is a general mechanism or two related heuristics.

Reply:

Please see our joint reply to Weakness 3 and Question 2 above for the shared alignment criterion and held-out tests.


We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience.

References

[1] Sharma and Trivedi. COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics. ICLR 2026.

[2] Wang, Yao, and Chen. Causal-Steer: Disentangled Continuous Style Control without Parallel Corpora. ICLR 2026.

[3] Maia Polo et al. tinyBenchmarks: Evaluating LLMs with Fewer Examples. ICML 2024.

AuthorsOpenReview ↗

Any Questions from Reviewer T7oa on Our Rebuttal?

Dear Reviewer T7oa,

We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Paper Discussion Phase will end in less than 12 hours from this comment, i.e., 11:59 pm AoE on November 18. We are happy to answer any remaining questions you may have or provide further clarification before then, but we will no longer be able to respond once the discussion phase has concluded.

If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments!

Sincerely,

Authors

Reviewer_ZLFpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CounterSteer, a steering method that uses only bad examples. It finds the direction that makes the model more likely to produce the bad behavior and steers the opposite way, using either activation gradients (Grad) or a temporary LoRA (LoRA). Experiments on over-refusal and sycophancy show results similar to or slightly better than contrastive baselines like DiM.

Critical strengths

(1) The problem is practical. In many cases, bad responses are easy to collect, but good responses need prompt-specific work. A method that only needs bad examples could be useful in real settings.

(2) The method is simple and cheap. CounterSteer-Grad needs only one backward pass and no training, and inference overhead is close to zero. This makes it easy to adopt, which matters if the gains hold up.

Critical weaknesses

(1) CounterSteer-Grad is very close to COLD-Kernel in COLD-Steer [1]. Both average normalized activation gradients of the loss over examples, and COLD-Steer already covers the one-sided setting and reverse steering. The LoRA version is close to CAUSAL-STEER [2], with the main difference being the sign flip. COLD-Steer is not discussed or compared, so it is hard to see what is really new.

(2) The paper says contrastive methods need hard-to-get good responses. But in the over-refusal experiments, DiM and LAT-PCA use harmful vs. benign prompts (App. D.2) and need no good responses either. Also, the positive targets in the target ablation were generated by Qwen2.5-14B with few fixes, so good responses do not seem that hard to get. Layer and strength selection still uses labeled benign and harmful data, so “one-sided” only applies to vector construction.

(3) Corollary 1 basically says a small step along the gradient lowers the loss. It only holds on the construction set, under teacher forcing, with small α. The experiments use free generation on held-out data with large α (8 to 32, and 128 for Gemma). There is no analysis linking the two, and the LoRA alignment condition is never checked.

(4) Many differences from DiM are within 1 point, and Grad is worse than DiM on Gemma (39.2 vs. 38.3). All results are single runs with no error bars. In sycophancy, gains on Llama are small, and DiM makes agreement worse on Qwen2.5-7B (27.8 to 35.3), which suggests the baselines may not be well tuned. The flip rate uses different denominators across methods, so it is hard to compare.

(5) Harmful responses go up a lot. HarmBench on Gemma rises from 7.0 to 14.0/18.7, and on Qwen3.6-27B from 1.3 to 10.3. The 2-point limit set on validation does not hold on test. The macro/micro averages in Table 1 mix refusal and harmful rates, which hides this. The trade-off curve is shown for only one model and one layer.

References: [1] COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics. ICLR 2026. [2] Causal-Steer: Disentangled Continuous Style Control without Parallel Corpora. ICLR 2026.

Questions for authors

(1) The flip rate is computed only on questions each method first got right, so the denominators differ. Could you also give overall accuracy after pushback? Otherwise it is hard to tell what the drop from 37.5 to 4.4 on Qwen2.5-3B really means.

(2) I would like to see how refusal NLL changes as α grows, on the construction set and on held-out data. And is the LoRA margin m positive in practice, or was this never checked?

9 comments
AuthorsOpenReview ↗

Rebuttal (1/5): Weakness 1: Grad, COLD-Steer, and aggregation ablation

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.

Critical Weaknesses

(1) CounterSteer-Grad is very close to COLD-Kernel in COLD-Steer [1]. Both average normalized activation gradients of the loss over examples, and COLD-Steer already covers the one-sided setting and reverse steering. COLD-Steer is not discussed or compared, so it is hard to see what is really new.

Reply:

Thank you for identifying these relevant connections. Our contribution is the derivative used to construct the activation vector, its exact relationship to the deployed intervention, and the resulting behavioral evidence.

Grad is not equivalent to the inspected COLD-Kernel operator. CounterSteer differentiates unwanted-response NLL with respect to one offset tied across all intervention sites; COLD-Kernel selects a single token's derivative before averaging. Writing gi,kg_{i,k} for the derivative at site kk,

qi=∑k∈Kigi,k=∇uLi(0),v=N−1∑iwiqi,d=v/∥v∥2.q_i=\sum_{k\in K_i}g_{i,k}=\nabla_uL_i(0),\quad v=N^{-1}\sum_iw_iq_i,\quad d=v/\|v\|_2.

Here wi=1w_i=1 for over-refusal; sycophancy uses reciprocal norms for nonzero sequence gradients and zero otherwise. These weights are fixed at the base model. For the matched weighted NLL LwL_w,

v=∇uLw(0),ddα[−Lw(αd)]∣0=−∥v∥2<0.v=\nabla_uL_w(0),\qquad \left.\frac{d}{d\alpha}[-L_w(\alpha d)]\right|_0=-\|v\|_2<0.

Thus dd is the steepest first-order suppression direction among unit shared offsets. Selected-token broadcasting does not generally have this property. Consider

L(h1,h2)=log⁡(1+exp⁡(2h1,1−4h2,1+2h2,2)).L(h_1,h_2)=\log(1+\exp(2h_{1,1}-4h_{2,1}+2h_{2,2})).

At zero, g1=(1,0)g_1=(1,0), g2=(−2,1)g_2=(-2,1) and q=(−1,1)q=(-1,1). Broadcasting g1g_1 gives initial NLL derivative q⊤g1=−1q^\top g_1=-1, whereas broadcasting qq gives +2+2. Positive normalization preserves both signs; the vectors are non-collinear, so strength or sign tuning cannot make them equivalent. This single-example counterexample isolates the extraction rule from example weighting.

Controlled position-aggregation ablation. To isolate position aggregation from example weighting, we compare single-position, prompt-position, and all-position gradients on Qwen2.5-7B with identical per-example normalization and otherwise controlled construction and evaluation.

Gradient positionsAgreement ↓Correction ↑
Single-position20.960.8
Prompt-position15.464.6
All-position18.263.5

All-position aggregation reduces agreement by 2.7 percentage points and increases correction by 2.7 points; prompt-position aggregation performs even better. This isolates the benefit of aggregation from normalization and supports the practical relevance of the shared-offset construction (Section 5, Table 6).

Direct comparison with COLD-Kernel. We evaluate a matched-data/runtime adaptation at its validation-selected operating point. Results appear in Section 4.3, Table 4, with implementation details in Appendix D.4.

Over-refusal (%; all columns ↓). OR = OR-Hard refusal; XS = XSTest benign refusal; HB = HarmBench harmful behavior.

ModelMethodORXSHB
Qwen2.5-7BCOLD-Kernel11.53.631.3
CounterSteer-Grad11.53.230.3
Llama-3.1-8BCOLD-Kernel22.26.032.3
CounterSteer-Grad18.46.032.3
Gemma-2-9BCOLD-Kernel38.710.019.0
CounterSteer-Grad39.28.014.0

Grad matches Qwen OR refusal while improving XS refusal and HB harmful behavior. On Llama it lowers OR refusal by 3.8 points at matching XS/HB rates. On Gemma, comparable OR refusal accompanies 2.0-point lower XS refusal and 5.0-point lower HB harmful behavior.

Sycophancy (%). Agreement ↓ is incorrect-belief endorsement; Correction ↑ is correct-answer recovery; tinyBench ↑ measures capability retention.

ModelMethodAgreement ↓Correction ↑tinyBench ↑
Qwen2.5-7BCOLD-Kernel29.556.565.0
CounterSteer-Grad16.264.264.3
Llama-3.1-8BCOLD-Kernel15.072.461.8
CounterSteer-Grad15.371.560.7

On Qwen, Grad reduces agreement by 13.3 points and improves correction by 7.7 points, with a 0.7-point tinyBench difference. Llama results favor COLD-Kernel. Together with the controlled ablation, these results establish a distinct construction with practical behavioral advantages and model-dependent tradeoffs.

AuthorsOpenReview ↗

Rebuttal (2/5): Weaknesses 1–2: LoRA, Causal-Steer, and motivation

The LoRA version is close to CAUSAL-STEER [2], with the main difference being the sign flip.

Reply:

LoRA differs beyond a sign flip. Causal-Steer supports single-source adaptation and reversal, but primarily studies continuous stylistic control rather than over-refusal and sycophancy correction. CounterSteer reads adapted-minus-base prompt-final block shifts before the current target response; Causal's default reads response-conditioned MLP shifts. At a fixed adapter and prompt, changing the unused current response leaves our extraction unchanged, but generally changes a response-conditioned extractor.

Our analysis further identifies the alignment requirement and bounds extraction-transfer error:

m≥mloc−GE.m\ge m_{\mathrm{loc}}-GE.

Here mm is the raw reversed-mean suppression margin, mlocm_{\mathrm{loc}} the hypothetical position-specific margin, GG a gradient bound, and EE extraction mismatch. Unit normalization rescales both sides without changing positivity. Successful adapter fitting alone does not establish either favorable alignment or small extraction error; we directly measure alignment in our joint reply to W3 and Q2.

Direct comparison with single-source Causal-Steer. We use the paper-formula construction with PCA and geometric-median aggregation; the inspected released implementation instead uses a projected mean. Single-layer deployment is adapted to our runtime. Each method uses its own validation-selected operating point; these are not shared-adapter ablations.

Over-refusal (%; all columns ↓). OR = OR-Hard refusal; XS = XSTest benign refusal; HB = HarmBench harmful behavior.

ModelMethodORXSHB
Qwen2.5-7BCausal-Steer15.23.230.7
CounterSteer-LoRA9.02.831.3
Llama-3.1-8BCausal-Steer24.07.633.7
CounterSteer-LoRA20.96.034.3

LoRA achieves stronger benign-refusal reduction on both models: OR refusal falls by 6.2 points on Qwen and 3.1 points on Llama, alongside lower XS refusal. HB harmful behavior increases by 0.6 points on each model.

Sycophancy (%).

ModelMethodAgreement ↓Correction ↑tinyBench ↑
Qwen2.5-7BCausal-Steer26.858.963.5
CounterSteer-LoRA20.959.964.7
Llama-3.1-8BCausal-Steer15.171.960.8
CounterSteer-LoRA13.972.761.7

LoRA improves all three reported sycophancy metrics on both models, with a simple prompt-final mean readout without PCA or geometric-median aggregation. These results are included in Section 4.3, Table 4, with implementation details in Appendix D.4.

For both baseline comparisons, Agreement and Correction use 740 incorrect-belief examples, and tinyBench averages six capability tasks. The additional belief judge has conditional acceptance pending independent held-out labeled validation.

(2) The paper says contrastive methods need hard-to-get good responses. But in the over-refusal experiments, DiM and LAT-PCA use harmful vs. benign prompts (App. D.2) and need no good responses either. Also, the positive targets in the target ablation were generated by Qwen2.5-14B with few fixes, so good responses do not seem that hard to get. Layer and strength selection still uses labeled benign and harmful data, so “one-sided” only applies to vector construction.

Reply:

Scope of one-sidedness. We will clarify that over-refusal DiM/LAT contrast harmful and benign prompts and require no helpful completions. “One-sided” concerns direction construction; validation still uses labeled task and safety data. Sycophancy more directly tests the motivation: contrastive baselines use both response types, while CounterSteer requires only the undesired side. Generating helpful targets was feasible for our target ablation, but this is a resource available in that experiment, not a requirement of CounterSteer. Holding 102 prompts fixed, refusal-target reversal outperforms helpful-target steering across most evaluated Qwen layers and every evaluated Llama layer for Grad. This tests which supervision yields effective control, rather than merely reversing one fixed vector's sign.

AuthorsOpenReview ↗

Rebuttal (3/5): Weakness 3 and Question 2

(3) Corollary 1 basically says a small step along the gradient lowers the loss. It only holds on the construction set, under teacher forcing, with small α. The experiments use free generation on held-out data with large α (8 to 32, and 128 for Gemma). There is no analysis linking the two, and the LoRA alignment condition is never checked.

Reply:

This reply also addresses Question 2.

We now directly test the two missing links: whether LoRA's suppression margin is positive and whether the effect persists beyond an infinitesimal construction-set step. The new Qwen2.5-7B measurements support both under matched target templates.

For a frozen reverse unit direction dd, we measure

munit=ddαL‾(αd)∣α=0.m_{\mathrm{unit}}=\left.\frac{d}{d\alpha}\overline{\mathcal L}(\alpha d)\right|_{\alpha=0}.

Positive values mean the fixed undesirable targets become locally less likely. We compute response-only mean token NLL per example, then average examples; automatic derivatives are checked against central finite differences.

Direct LoRA alignment check.

Evaluation targetsGrad margin [95% CI]LoRA margin [95% CI]
Construction templates, n=5120.13716 [0.13378, 0.14078]0.05125 [0.05007, 0.05252]
Held-out prompts + construction templates, n=1820.08282 [0.07766, 0.08808]0.03136 [0.02948, 0.03331]

Both methods have positive margins on every example in these two conditions. LoRA's alignment is therefore empirically verified here, rather than inferred from successful adapter fitting. The held-out prompts are disjoint from construction prompts and are the benign held-out cases whose baseline outputs were classified as refusals.

Finite-strength behavior. We evaluate fixed prompt–response targets as strength increases from zero to each frozen diagnostic operating point. Mean NLL increases at every evaluated positive strength and continues increasing to the endpoint:

Evaluation targetsGrad ΔNLLLoRA ΔNLL
Construction templates+2.561+1.273
Held-out prompts + construction templates+1.797+0.965

The full NLL–α curves, including near-zero strengths and all three target conditions, are provided in Appendix H.5. These results establish matched-template transfer to unseen prompts and persistence at finite strength, beyond the construction-set local identity.

These are separate frozen diagnostic configurations: Grad L23, α=16 with unit additive steering; historical LoRA L23, α=0.1 with activation-norm-scaled steering. They are not the selected configurations in the paper's main result table. The unit-direction margin and the derivative along historical LoRA's norm-scaled deployment path are distinct measurements.

What transfers. On those same 182 prompts paired with naturally generated baseline refusal responses, the mean unit margins are −0.00655 for Grad and −0.00133 for LoRA, with endpoint ΔNLL −0.07055 and −0.01121. This identifies target-response distribution as an important transfer condition. Our measured result is strong prompt transfer under matched templates; free-generation refusal, harmfulness, and capability are assessed directly by the behavioral benchmarks. We will make this separation explicit and include the NLL curves and all three target conditions in the revision.

Reviewer_z9RVOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies activation steering when only examples of the behavior to suppress are available, without desired counterparts or contrastive pairs. It proposes two ways to build a steering vector from such one-sided data. CounterSteer-Grad takes the gradient of the undesired responses' negative log-likelihood with respect to a shared additive offset at one layer of the frozen model, normalizes and averages it across examples, and adds the resulting vector at inference so that the model moves away from the undesired behavior. CounterSteer-LoRA fine-tunes a temporary LoRA adapter on the undesired examples, measures the average activation shift it induces, and steers the base model in the opposite direction. Experiments on over-refusal and sycophancy across Qwen, Llama, and Gemma models compare the two variants with contrastive steering baselines, and report reduced target behavior with roughly preserved general capability.

Critical strengths

A well-motivated problem setting. The asymmetry argument is convincing: a generic refusal or sycophantic reply is easy to recognise and reuse, while the correct replacement is prompt-specific and often costly to obtain. Steering when effective supervision is one-sided is a practical and under-studied question, and this framing is the main reason I find the submission relevant. The target ablation is informative. On identical prompts and budget, reversing the refusal target yields higher non-refusal than steering toward helpful targets across most layers on Qwen2.5-7B (Fig. 3) and at every layer for Grad on Llama-3.1-8B (Fig. 6), and Grad helps on Llama despite near-zero cosine with DiM. This is the most direct evidence in the paper that reversing undesired targets can be more effective than following helpful targets, and it suggests that the Grad direction is not simply a rediscovery of the known refusal direction.

Critical weaknesses

W1. Limited methodological novelty

The motivation is sound, but the methodological novelty is limited relative to the question the paper raises. The central question is how to steer a model effectively when the available supervision is restricted to undesired examples. CounterSteer addresses it by estimating the direction in which the undesired examples would move the model's representations and steering in the opposite direction. Grad obtains this direction from loss gradients, and LoRA obtains it from actual fine-tuning. This idea is similar to that of influence functions[1]. This method also estimates the effect of training examples through their loss gradients. The paper does not discuss this relation or clarify how its construction differs from these methods. Both variants serve mainly as implementation vehicles for this reversal idea. Beyond this idea, the paper offers limited insight into how to steer more effectively when desired responses are unavailable.

W2. Missing comparison with closely related methods

CounterSteer-LoRA follows a fine-tune-and-negate procedure, which the authors relate to task arithmetic[2] and to the single-source variant of Causal-Steer[3]. Similarly, CounterSteer-Grad is closely related to steering vectors optimized on undesired targets[4]. The authors distinguish the two mainly by Grad's use of a single averaged gradient at the base model. Neither variant includes a component that specifically addresses the scarcity of effective data, such as robustness to fewer, noisier, or less diverse undesired examples. The distinction from these methods therefore lies mainly in construction details. However, the experiments compare only against DiM, LAT-PCA and, for sycophancy, a random direction, and none of the closely related methods is evaluated. Without such comparisons, the empirical advantage of CounterSteer over methods that use the same one-sided data cannot be assessed. Also, DiM and LAT-PCA are computed from prompt-only activations at the final prompt token, contrasting 128 harmful AdvBench prompts with 128 benign OR-Bench prompts. CounterSteer instead uses 512 benign OR-Bench prompts paired with refusal responses and backpropagates through the response tokens. The two settings differ in data source, sample size, use of responses, and readout position, so the reported gains cannot be attributed to the one-sided construction alone.

[1]: Understanding Black-box Predictions via Influence Functions

[2]: Editing Models with Task Arithmetic

[3]: Causal-Steer: Disentangled Continuous Style Control without Parallel Corpora

[4]: One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMs

Questions for authors
  1. How does CounterSteer-Grad differ from influence functions applied to the undesired examples? For instance, would incorporating curvature information, as influence functions do via the inverse Hessian H−1H^{-1}, yield a different direction or result?

  2. Apart from computing a gradient or an adapter shift, which component of the method is designed for the setting in which effective data are scarce? How does performance change with fewer or noisier undesired examples?

  3. How do the over-refusal results change when DiM is built from the same prompt-response data as CounterSteer?

I would be happy to raise the score if the author could provide more evidence to answer my question.

8 comments
AuthorsOpenReview ↗

Rebuttal (1/5): Weakness 1 and Question 1

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.

Critical Weaknesses

W1. Limited methodological novelty The motivation is sound, but the methodological novelty is limited relative to the question the paper raises. The central question is how to steer a model effectively when the available supervision is restricted to undesired examples. CounterSteer addresses it by estimating the direction in which the undesired examples would move the model's representations and steering in the opposite direction. Grad obtains this direction from loss gradients, and LoRA obtains it from actual fine-tuning. This idea is similar to that of influence functions[1]. This method also estimates the effect of training examples through their loss gradients. The paper does not discuss this relation or clarify how its construction differs from these methods. Both variants serve mainly as implementation vehicles for this reversal idea. Beyond this idea, the paper offers limited insight into how to steer more effectively when desired responses are unavailable.

Reply:

This reply also addresses Question 1.

Thank you for highlighting this connection. Both methods use loss derivatives, but they differentiate different objects and answer different counterfactual questions. Classical influence functions estimate how a fitted parameter optimum changes when a training example is reweighted, using an inverse parameter-space Hessian. CounterSteer-Grad instead asks: with model parameters fixed, which shared activation offset most strongly suppresses the undesirable targets locally?

For response-only NLL LiL_i, we construct

qi=∇uLi(0)=∑k∈Ki∇hi,kLi,v=N−1∑iwiqi,d=v/∥v∥2.q_i=\nabla_uL_i(0)=\sum_{k\in K_i}\nabla_{h_{i,k}}L_i,\quad v=N^{-1}\sum_iw_iq_i,\quad d=v/\|v\|_2.

The sum captures the joint sensitivity of all intervention positions. The weights are wi=1w_i=1 for over-refusal and reciprocal gradient norms for nonzero sycophancy examples (zero otherwise), followed by final unit normalization. We will correct the manuscript's overly broad description of per-example normalization.

For the matched NLL LwL_w, with weights fixed at the base model, v=∇uLw(0)v=\nabla_uL_w(0). Hence, for nonzero vv,

ddα[−Lw(αd)]∣0=−∥v∥2<0.\left.\frac{d}{d\alpha}[-L_w(\alpha d)]\right|_{0}=-\|v\|_2<0.

Thus our unit direction maximizes the initial increase in the undesirable targets' matched NLL among unit shared offsets. This is a direct characterization of the deployed activation intervention, rather than an estimate of retraining or data-removal effects.

Curvature. A fixed positive-definite activation-space preconditioner PP would generally change the direction. Applying PvPv gives derivative −v⊤Pv<0-v^\top Pv<0 for the matched log-likelihood, preserving local suppression. This differs from classical parameter-space influence functions; whether it improves behavioral control is a useful extension.

Contribution beyond reversal. Task arithmetic negates parameter changes; iterative optimized steering vectors use a different optimization procedure and token-unlikelihood objective. CounterSteer explicitly connects its constructed activation vector to the intervention it controls. Grad supplies the exact shared-offset derivative; LoRA supplies a finite-adaptation route whose success is characterized through alignment and extraction-transfer analysis. The additional comparisons and controlled aggregation ablation below provide empirical evidence for these constructions.

AuthorsOpenReview ↗

Rebuttal (2/5): Weakness 2: COLD-Steer comparison

W2. Missing comparison with closely related methods CounterSteer-LoRA follows a fine-tune-and-negate procedure, which the authors relate to task arithmetic[2] and to the single-source variant of Causal-Steer[3]. Similarly, CounterSteer-Grad is closely related to steering vectors optimized on undesired targets[4]. The authors distinguish the two mainly by Grad's use of a single averaged gradient at the base model. Neither variant includes a component that specifically addresses the scarcity of effective data, such as robustness to fewer, noisier, or less diverse undesired examples. The distinction from these methods therefore lies mainly in construction details.

[1]: Understanding Black-box Predictions via Influence Functions [2]: Editing Models with Task Arithmetic [3]: Causal-Steer: Disentangled Continuous Style Control without Parallel Corpora [4]: One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMs

However, the experiments compare only against DiM, LAT-PCA and, for sycophancy, a random direction, and none of the closely related methods is evaluated. Without such comparisons, the empirical advantage of CounterSteer over methods that use the same one-sided data cannot be assessed. Also, DiM and LAT-PCA are computed from prompt-only activations at the final prompt token, contrasting 128 harmful AdvBench prompts with 128 benign OR-Bench prompts. CounterSteer instead uses 512 benign OR-Bench prompts paired with refusal responses and backpropagates through the response tokens. The two settings differ in data source, sample size, use of responses, and readout position, so the reported gains cannot be attributed to the one-sided construction alone.

Reply:

We added direct comparisons with COLD-Kernel and single-source Causal-Steer. CounterSteer achieves stronger results on key behavioral metrics, with the tradeoffs reported below.

COLD-Kernel versus CounterSteer-Grad. COLD-Kernel reads a selected-position gradient; Grad sums gradients across intervention positions. These are different operators, not merely different strengths. For example, site gradients g1=(1,0)g_1=(1,0) and g2=(−2,1)g_2=(-2,1) yield shared-offset gradient q=(−1,1)q=(-1,1). Broadcasting g1g_1 gives initial NLL derivative q⊤g1=−1q^\top g_1=-1, while broadcasting qq gives q⊤q=2q^\top q=2. Positive normalization preserves both signs: single-position NLL ascent need not suppress the target when broadcast. We evaluate COLD-Kernel with matched negative data and inference intervention.

Over-refusal (%; all columns ↓). OR = OR-Hard refusal; XS = XSTest benign refusal; HB = HarmBench harmful behavior.

ModelMethodORXSHB
Qwen2.5-7BCOLD11.53.631.3
Grad11.53.230.3
Llama-3.1-8BCOLD22.26.032.3
Grad18.46.032.3
Gemma-2-9BCOLD38.710.019.0
Grad39.28.014.0

Grad improves Qwen's refusal–safety balance and lowers Llama OR refusal by 3.8 points with matching XS/HB rates. On Gemma, OR refusal is comparable, while XS refusal and HB harmful behavior fall by 2.0 and 5.0 points.

Sycophancy (%). Agr = incorrect-belief agreement ↓; Corr = correction ↑; tiny = tinyBench accuracy ↑.

ModelMethodAgrCorrtiny
Qwen2.5-7BCOLD29.556.565.0
Grad16.264.264.3
Llama-3.1-8BCOLD15.072.461.8
Grad15.371.560.7

On Qwen, Grad reduces agreement by 13.3 points and improves correction by 7.7 points, with a 0.7-point capability difference. Llama results favor COLD.

AuthorsOpenReview ↗

Rebuttal (3/5): Weakness 2: Causal-Steer comparison and aggregation ablation

Causal-Steer versus CounterSteer-LoRA. Causal-Steer primarily studies continuous stylistic control, whereas CounterSteer targets undesired behavioral tendencies such as over-refusal and sycophancy. Its single-source variant establishes bidirectional style control, but does not establish effectiveness on these behavioral correction tasks. Our contribution combines distinct activation constructions and their analysis with direct evidence on behavioral correction, safety, and capability retention. Both methods support single-source adaptation and reversal; the extraction differs: LoRA averages prompt-final block shifts before the current target response, whereas Causal uses response-conditioned MLP shifts with PCA and geometric-median aggregation. Our Causal baseline follows the paper formula (the released implementation uses a projected mean) with single-layer deployment adapted to our runtime.

Over-refusal (%; all columns ↓).

ModelMethodORXSHB
Qwen2.5-7BCausal15.23.230.7
LoRA9.02.831.3
Llama-3.1-8BCausal24.07.633.7
LoRA20.96.034.3

LoRA lowers OR refusal by 6.2 points on Qwen and 3.1 on Llama, also improving XS refusal, with a 0.6-point HB increase on each model.

Sycophancy (%; same definitions as above).

ModelMethodAgr ↓Corr ↑tiny ↑
Qwen2.5-7BCausal26.858.963.5
LoRA20.959.964.7
Llama-3.1-8BCausal15.171.960.8
LoRA13.972.761.7

LoRA improves all three reported sycophancy metrics on both models, despite using a simple prompt-final mean without PCA or geometric-median aggregation.

Agreement and Correction are USER/GOLD verdict rates on 740 incorrect-belief examples; tinyBench averages six capability tasks. These are operating-point comparisons, not shared-adapter ablations. The additional belief judge has conditional acceptance pending independent held-out labeled validation.

Position aggregation. To isolate aggregation from example weighting, we compare single-position, prompt-position, and all-position gradients on Qwen with identical per-example normalization and otherwise controlled construction and evaluation.

Gradient positionsAgreement ↓Correction ↑
Single20.960.8
Prompt15.464.6
All18.263.5

Aggregating beyond one position improves both behavioral metrics: all-position aggregation reduces agreement by 2.7 points and increases correction by 2.7 points. Prompt aggregation performs strongly as well. This supports the value of position aggregation, rather than attributing the COLD–Grad difference solely to normalization.

The data-comparability concern is addressed further in our reply to Question 3 below; the scarcity concern is addressed in Question 2.

Questions for Authors

  1. How does CounterSteer-Grad differ from influence functions applied to the undesired examples? For instance, would incorporating curvature information, as influence functions do via the inverse Hessian , yield a different direction or result?

Reply:

Please see our joint reply to Weakness 1 and Question 1 above, which distinguishes the differentiated objects and addresses activation-space curvature.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

Dear AC and reviewers,

Thank you for your thoughtful reviews and valuable comments, which have helped us strengthen the paper. We are encouraged by the reviewers’ recognition that: (1) the asymmetric-supervision setting is practical and compelling, since unwanted responses are often easier to identify than prompt-specific desired answers (z9RV, ZLFp, T7oa); (2) CounterSteer-Grad offers a simple, training-free construction that is easy to understand and analyze, with minimal inference overhead (ZLFp, T7oa); (3) the fixed-prompt target ablation provides informative evidence that reversing undesired targets can outperform steering toward helpful targets (z9RV); and (4) examining both gradient and adaptation routes while carefully delimiting the local theoretical claims strengthens the presentation (T7oa).

The shared concerns focus on the distinction from closely related methods, transfer beyond construction examples, and the criteria for useful behavioral control. We address them with the following additional evidence.

Methodological contribution and direct comparisons. CounterSteer-Grad differentiates an offset shared across intervention positions, providing an exact local characterization of the deployed intervention. CounterSteer-LoRA uses prompt-final adapted-minus-base shifts, with alignment and extraction-transfer analysis distinguishing it from response-conditioned extraction. A controlled Qwen ablation, with identical per-example normalization, shows that all-position versus single-position aggregation reduces agreement from 20.9% to 18.2% and improves correction from 60.8% to 63.5%.

The new comparisons also show behavioral advantages. On Qwen sycophancy, Grad reduces agreement by 13.3 percentage points and increases correction by 7.7 points relative to COLD-Kernel[1]. LoRA improves agreement, correction, and tinyBench accuracy[3] over the adapted single-source Causal-Steer baseline[2] on both evaluated models. Over-refusal results likewise show competitive refusal–safety tradeoffs, reported separately for each model.

Alignment, transfer, and data robustness. We directly verify positive Grad and LoRA margins on construction and held-out matched-template examples, with NLL increases persisting at finite strengths. Natural-target diagnostics identify target formulation as an important transfer condition. Additional experiments show that Grad retains most of its effect with 32 examples, both variants tolerate substantial contamination in the tested settings, and formulation diversity matters.

Useful control and evaluation reliability. We assess unwanted-behavior reduction alongside correctness, capability, and safety. Both variants improve accuracy after pushback on all three evaluated models. We add confidence intervals, three LoRA training seeds, and fix rates, and distinguish the validation safety-selection rule from held-out harmfulness.

Together, these additions strengthen the evidence for useful control from undesired-only direction construction.

We hope that our rebuttal has helped to clear concerns about our work. We are glad to answer any further questions you have on our submission and we would appreciate it if we could get your further feedback at your earliest convenience.

References

[1] Sharma and Trivedi. COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics. ICLR 2026.

[2] Wang, Yao, and Chen. Causal-Steer: Disentangled Continuous Style Control without Parallel Corpora. ICLR 2026.

[3] Maia Polo et al. tinyBenchmarks: Evaluating LLMs with Fewer Examples. ICML 2024.

AuthorsOpenReview ↗

Summary of Revisions

Incorporating the reviewers’ comments and suggestions, we have made the following revisions. Changes are highlighted in blue.

  • Added closely related baseline comparisons (Section 4.3, Table 4; Appendix D.4). We compare COLD-Kernel and single-source Causal-Steer with CounterSteer on over-refusal and sycophancy, reporting behavioral performance alongside safety and capability tradeoffs. Related Work now distinguishes their constructions.

  • Added a controlled position-aggregation ablation (Section 5, Table 6). With per-example normalization controlled, both prompt-position and all-position aggregation improve agreement and correction over single-position extraction.

  • Clarified normalization and the theoretical scope (Sections 2.3 and 3; Appendices C, D.5, and F). We distinguish task-specific example weighting from final vector normalization, correct the algorithm references, and explain how the matched-score analysis covers both raw-gradient averaging and per-example normalization.

  • Added direct alignment and transfer diagnostics (Appendix H.5). We report Grad and LoRA margins and finite-strength NLL curves on construction and held-out examples, separating prompt transfer from sensitivity to target formulation.

  • Expanded baseline and data robustness experiments (Appendices H.3–H.4). We added larger-data DiM and response-readout comparisons, together with experiments varying the sample count, contamination, and template diversity of undesired examples.

  • Expanded baseline and data robustness experiments (Appendices H.3–H.4). We added larger-data DiM and response-readout comparisons, together with experiments varying the sample count, contamination, and template diversity of undesired examples. The sample-count ablation reports both OR-Hard refusal and OR-Toxic harmful-response rates to assess the accompanying safety tradeoffs.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.