acceptodds
Under review as a conference paper at ICLR 2027

CANS: Capability-Preserving Neuron Steering for Large Language Models

Abstract

Activation steering offers training-free and dynamic control of target behaviors in large language models, but existing methods can impair general capabilities and require domain-specific intervention designs. We introduce Capability-Preserving Neuron Steering (CANS), a unified framework that balances target-behavior control with capability preservation. To make target-behavior improvement and general-capability preservation tractable, we introduce two proxies: log-probabilities of target tokens in model generated responses, and output-distribution KL divergence on general-purpose inputs. Combining the local approximation of the two proxies yields a closed-form solution, enabling lightweight steering without iterative model training or additional inference-time computation. The resulting neuron-wise scaling coefficients achieve effective steering while limiting general-capability loss. A shared data-construction and intervention procedure supports adaptation across domains using sampled responses with target-behavior filtering and target-span annotations. Across safety, toxicity, and hallucination control on Qwen and Llama models, CANS achieves comparable or better steering while better preserving MMLU, BBH, and IFEval performance than SFT, CAA, and neuron-level baselines. On Qwen3-4B, CANS reduces HarmBench attack success from to with a -point decrease in average capability. These results support CANS as a practical approach to behavioral control across domains.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Neuron-Level Steering, Capability Preservation, Adaptive Activation Scaling, KL-Regularized Optimization, Behavioral Control

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_vMuNOpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CANS, a neuron-level steering framework that balances behavioral control with capability preservation. CANS combines target-token log-probability gradients with a diagonal Fisher approximation of output-distribution drift to derive closed-form neuron scaling coefficients. Experiments on safety, toxicity, and hallucination show favorable steering–capability trade-offs across Qwen and Llama models.

Critical strengths
  1. The method explicitly incorporates preservation costs into neuron-wise interventions, providing a coherent formulation of the steering–capability trade-off.
  2. The closed-form solution avoids iterative fine-tuning, and scaling coefficients can be folded into model weights without additional inference operations.
  3. Evaluation covers three behavioral domains and multiple capability benchmarks, with substantial improvements in several settings.
Critical weaknesses
  1. Capability preservation relies on distributional stability over reference text. Although the paper acknowledges this proxy and provides empirical support, its coverage of broader capabilities remains uncertain.
  2. Target-span likelihood may capture lexical preferences rather than the intended behavior. The reported improvements do not fully establish whether safety gains avoid over-refusal or hallucination reductions reflect more informative, correct answers.
  3. CANS-Grad-Only changes both the preservation mechanism and the intervention scheme by uniformly scaling a selected subset. Consequently, this comparison does not cleanly isolate the contribution of Fisher weighting.
Questions for authors

I rewrite the weakness above into the questions below:

  1. How robust is capability preservation on tasks substantially different from the reference corpus, such as coding and multilingual reasoning?
  2. Can the authors report benign-request refusal rates and distinguish correct, informative answers from abstentions in hallucination evaluation?
  3. Can the authors compare against scaling all neurons proportionally to their gradients, using the same clipping and tuning protocol, to isolate the benefit of Fisher weighting?
9 comments

We sincerely thank the reviewer for the constructive and thoughtful feedback. We have fully incorporated every experimental and methodological suggestion you raised across the entire revised manuscript:

  1. Expanded Capability Suite with Coding (MBPP) & Multilingual Math (MGSM) (W1 & Q1): We evaluated all methods on MBPP and MGSM and updated our overall capability metric across the paper to the 5-benchmark average (IFEval, BBH, MMLU, MBPP, and MGSM), confirming that WikiText-2 Fisher curvature robustly protects specialized coding and multilingual reasoning circuits.
  2. Evaluated truthfulness and informativeness (T&I failure) and benign refusal (XSTest) (W2 & Q2). Throughout the paper, we now report T&I failure (%) as 100(1−J/N)100(1-J/N), where NN is the total number of responses and JJ is the number judged both truthful and informative. Under this metric, an uninformative abstention counts as a failure. We also measured how often CANS refuses benign requests on XSTest to assess over-refusal after safety steering.
  3. Standardized CANS-Grad-Only to All-Neuron Gradient Scaling (F=I\mathbf{F}=\mathbf{I}) (W3 & Q3): We updated CANS-Grad-Only across all experiments to scale all dd neurons proportionally to their gradients (Δα=cg\Delta\alpha = c\mathbf{g}), cleanly isolating the critical role of inverse-Fisher weighting.

Response to W1 & Q1 (Robustness of Capability Preservation on Coding and Multilingual Reasoning)

We thank the reviewer for this constructive suggestion. To rigorously test whether distributional stability on our general-purpose reference corpus (WikiText-2) preserves capabilities on tasks substantially different from encyclopedic English prose, we expanded our evaluation suite in the revised paper with two additional benchmarks:

  1. MBPP for code generation, and
  2. MGSM for multilingual mathematical reasoning.

We re-evaluated all methods across all models and domains on MBPP and MGSM, and updated the aggregated capability score across the entire revised manuscript to the equal-weight mean of all five benchmarks (IFEval, BBH, MMLU, MBPP, and MGSM; see updated Figure 2, Figure 3, and Appendix B). We report detailed results for MGSM and MBPP below, alongside domain-specific metrics for all six model–domain settings. All values are expressed as percentages. The corresponding plots appear in Figures 10 and 11 of Appendix B (p. 22-23).

Llama Family (Llama-3-8B-Instruct Safety / Llama-2-7B Toxicity / Llama-3.1-8B-Instruct Hallucination):

MethodSafety: ASR ↓Safety: MGSM ↑Safety: MBPP ↑Toxicity: Toxic Rate ↓Toxicity: MGSM ↑Toxicity: MBPP ↑Hallu.: T&I Failure ↓Hallu.: MGSM ↑Hallu.: MBPP ↑
Unmodified14.0051.8257.2037.3312.3619.2058.5068.7359.40
SFT10.0054.5557.2017.8311.6420.0037.7569.2758.40
CAA10.5027.6436.6034.5010.9119.2059.2568.3658.20
AurA0.000.000.0021.5010.9118.8052.5065.4557.60
SN-Tune8.7550.9157.6033.6711.4519.2047.2556.3659.00
DAP12.2550.7356.8035.6711.6420.2064.2570.0059.60
H-Neurons7.5023.2735.8032.339.0918.6056.5069.4559.00
CANS-Grad-Only8.7530.7353.2019.333.646.6046.5069.0959.20
CANS (Ours)8.7554.3658.2018.8311.6419.4046.5069.4559.20

Qwen3-4B (Safety / Toxicity / Hallucination):

MethodSafety: ASR ↓Safety: MGSM ↑Safety: MBPP ↑Toxicity: Toxic Rate ↓Toxicity: MGSM ↑Toxicity: MBPP ↑Hallu.: T&I Failure ↓Hallu.: MGSM ↑Hallu.: MBPP ↑
Unmodified15.5074.5565.0019.1774.5565.0039.5074.5565.00
SFT8.0073.8264.404.5073.4563.8037.0073.6465.60
CAA14.0049.4553.8019.5073.8265.6038.0073.8265.20
AurA0.000.000.0010.8372.9165.2039.0068.0061.00
SN-Tune15.7572.9165.2020.0074.1864.6037.7573.4565.20
DAP16.5073.4565.8017.8373.4564.6038.5074.1865.00
H-Neurons16.5074.0065.0016.8374.7365.8037.0074.7365.40
CANS-Grad-Only7.5073.4565.404.6771.0962.2035.2572.5564.60
CANS (Ours)8.0075.2765.404.1773.6463.2034.2571.6466.20

Key Takeaways:

  1. Robust Preservation on Coding and Multilingual Math: Even though our Fisher diagonal is estimated purely on unlabeled English prose (WikiText-2), CANS robustly preserves both MBPP (code generation) and MGSM (multilingual math) across all six settings—frequently matching or slightly exceeding the unmodified model (e.g., on Llama-3-8B Safety, CANS reaches 54.36% on MGSM and 58.20% on MBPP vs. 51.82%51.82\% and 57.20%57.20\% unmodified; on Qwen3-4B Safety, CANS reaches 75.27% on MGSM and 65.40% on MBPP vs. 74.55%74.55\% and 65.00%65.00\% unmodified).
  2. Sharp Contrast with Unregularized and Heuristic Baselines: Without Fisher curvature protection, methods that achieve strong steering often suffer severe collapses on out-of-distribution capabilities. For instance, on Llama-3-8B Safety, H-Neurons and CAA drop MGSM from 51.82%51.82\% to 23.27% and 27.64%, and MBPP from 57.20%57.20\% to 35.80% and 36.60%. Similarly, CANS-Grad-Only (F=I\mathbf{F}=\mathbf{I}) drops MGSM to 30.73% on Llama-3-8B Safety (vs. 54.36% for CANS) and collapses on Llama-2-7B Toxicity (dropping MGSM from 12.36%12.36\% to 3.64% and MBPP from 19.20%19.20\% to 6.60%, whereas CANS maintains 11.64% and 19.40%).
  3. Mechanistic Insight: These results demonstrate that specialized tasks such as code synthesis and multilingual reasoning rely on shared foundational FFN circuits (e.g., logical deduction, state tracking, and syntactic composition) that are already active during general text continuation. By penalizing neurons with high Fisher curvature on reference continuations, CANS effectively protects these shared computational backbones across diverse downstream tasks.
Reviewer_Nn81OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper targets on constructing steering approach that can be more generalised and not reduced the other behaviour of the Large Language Models (LLMs). These are important challenges in lightweight steering behaviour of the model. The author proposed to use proxies (the log-prob and the KL divergence) to measure and construct steering objectives for neuron steering. The experiments were conducted to steer various behaviours of LLMs and to test the general capability on different datasets.

Critical strengths
  • The approach is well-presented with clear structure in section 3.
  • The experiments are conducted to illustrate the approach is suitable for steering different behaviours of LLMs, including safety, toxicity, hallucination. It shows that the CANS can achieve high performance of steering, while preserving the ability on general capability task on instruction following, reasoning, problem solving and knowledge, due to the utilisation of two proxies (Figure 2). The target models are in different families.
  • Figure 3 shows the ability of controlling behaviour with different level based on controlling the factor ϵq\epsilon_q, while the general capability is not changed.
  • The paper also analyse different effects of two proxies to two steering and general capability (Figure 4 and 5).
Critical weaknesses
  • The paper can be more comprehensive if the effect of location of neuron in the model is discussed or analysed (for example layer wise). However, with the current approach and analysis, the paper already shows effective approach in intervention with analyses on the aspects that the paper introduced.
Questions for authors

Could the author discuss whether there are effects of location of neuron to the effects?

2 comments

We sincerely thank the reviewer for the positive assessment of our paper and for recognizing the effectiveness of our framework and analyses!


Response to W1 & Q1 (Layer-Wise Analysis of Neuron Locations)

Following your suggestion, we conducted a comprehensive layer-wise analysis of the CANS scaling perturbations across all models and domains, which we have added to Appendix H (Figure 14 and Figure 15) on page 34 of the revised manuscript. Specifically, we examine the layer-wise mean absolute scaling change, mean(∣Δαj∣)=mean(∣αj−1∣)\text{mean}(\lvert\Delta\alpha_j\rvert) = \text{mean}(\lvert\alpha_j - 1\rvert), across all FFN intermediate neurons in each layer, as well as the separate layer-wise distributions for amplified (Δαj>0\Delta\alpha_j > 0) and attenuated (Δαj<0\Delta\alpha_j < 0) neurons.

Below, we illustrate the layer-wise profile for Llama-3-8B-Instruct on Safety (32 layers, 14,336 neurons per layer):

Mean |delta alpha| across layers (Llama-3-8B-Instruct, Safety)
0.012318 |
         |                         ******
0.010265 |                      ***......****
         |                   ***.............***
         |                ***...................**              *
0.008212 |             ***........................**          **.
         |           **.............................***     **...
         |         **..................................** **.....
0.006159 |       **......................................*.......
         |     **................................................
0.004106 |  ***..................................................
         |**.....................................................
0.002053 |.......................................................
0.000000 +-------------------------------------------------------
          0        5        10       15       20       25      30
                          Layer Index (0 to 31)

Key Empirical Findings Across Models and Domains:

  1. Dominance of Middle Layers: Across models and domains, CANS consistently allocates the largest scaling magnitudes to the middle layers. For Llama-3-8B-Instruct (Safety), mean(∣Δα∣)\text{mean}(\lvert\Delta\alpha\rvert) rises steadily from early layers to a primary peak at Layer 13 (0.01100.0110), gradually recedes to a local trough at Layer 26 (0.00570.0057), and exhibits a moderate rebound at the final layer (Layer 31, 0.00850.0085). This aligns with prior mechanistic interpretability findings that middle FFN layers are primary loci for semantic and behavioral feature composition, whereas early layers process shallow token representations with high downstream KL sensitivity.
  2. Symmetric Co-Localization of Amplification and Attenuation: When decomposing perturbations into amplified neurons (αj>1\alpha_j > 1) and attenuated neurons (αj<1\alpha_j < 1), we find that both groups follow a highly consistent layer-wise trajectory. Layers that undergo the strongest average amplification also exhibit the strongest average attenuation, indicating that CANS performs coordinated push-pull behavioral modulation within the same semantically active middle layers.

We thank the reviewer again for championing our work and suggesting this revealing mechanistic analysis, which has significantly enriched the interpretability discussion in our revised manuscript. Please let us know if you have any further questions!

Thank the authors for your effort to give further and analysis about the layer-wise effects. My current comments are addressed, and I will maintain the positive evaluation.

Reviewer_KehvOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CANS, a neuron-level steering method that aims to change specific LLM behaviors while preserving general capabilities. It uses target-token log-probabilities to measure steering effects and KL divergence on general-purpose text to estimate capability loss. By combining a first-order approximation of the steering objective with a diagonal Fisher approximation of the KL constraint, the authors derive closed-form scaling coefficients for each neuron. These coefficients can be folded into the model weights without adding inference-time operations.

The method is evaluated on safety, toxicity, and hallucination control using Qwen and Llama models. Compared with several fine-tuning and steering baselines, CANS generally achieves a better trade-off between reducing undesirable behaviors and preserving performance on MMLU, BBH, and IFEval. The paper also includes ablations and experiments examining steering strength, reference data, and the accuracy of the proposed approximations.

Critical strengths
  • CANS provides a clear way to balance behavioral steering with preserving general capabilities. The closed-form solution assigns different scaling strengths to neurons based on their steering gradients and Fisher estimates. Folding these coefficients into the weights avoids additional inference operations. The results also show a favorable trade-off compared with gradient-ranked uniform scaling, although this comparison does not isolate the Fisher weighting alone.

  • The paper evaluates three target behaviors across Qwen and Llama models, with comparisons against several steering baselines. I appreciate the experiments examining the steering budget, reference corpus, and local approximations. In particular, the controlled approximation tests provide useful evidence for the proposed optimization, although they do not directly validate the deployed coefficient vectors.

Critical weaknesses
  • The core optimization in CANS closely resembles a damped natural-gradient step, using a diagonal Fisher matrix to constrain changes in model behavior. Similar Fisher-based regularization ideas have also been explored in continual learning. The paper would benefit from a clearer discussion of these connections. In particular, it is unclear which part of the closed-form optimization is methodologically new, beyond applying it to neuron scaling with a target-span objective.

  • The method relies on first-order and quadratic approximations, but Figure 6 validates them using uniform scaling of selected neurons rather than the adaptive coefficients produced by CANS. Appendix H shows that some coefficients can deviate substantially from 1 before clipping, which raises questions about the accuracy of the local approximations. Although Appendix I reports measured KL and proxy changes across steering budgets, it does not directly compare these values with the local predictions at the deployed coefficients. Clipping also changes the closed-form solution, so its effect on the claimed optimality and observed performance deserves further analysis.

Questions for authors
  • How does CANS differ from a single damped natural-gradient step on the neuron scaling coefficients? Would multiple updates offer a better steering-capability trade-off?

  • How accurate are the first-order and Fisher approximations when evaluated at the actual CANS coefficients, both before and after clipping? How sensitive are the results to the clipping threshold?

  • The gradient-only baseline changes both neuron selection and scaling strength. Could the authors compare CANS against an all-neuron gradient-scaling baseline with the Fisher matrix replaced by the identity?

8 comments

We sincerely thank the reviewer for the rigorous and insightful feedback, which helped us sharpen our methodological positioning, theoretical exactness, and empirical evaluation. In response, we have made the following key revisions:

  1. Clarified Methodological Novelty & Proved Why Diagonal Fisher Constraint is More Favorable (W1 & Q1): We expanded Sections 2, 3.2 & 3.3 to show that CANS improves finite-sample natural gradient optimization by decoupling the trust-region constraint metric (damped diagonal F\mathbf{F}) from the local KL curvature estimator (full Monte Carlo Fisher estimate FF). When d≫nd \gg n (which holds in our setting), using the sample Fisher FF with weak damping can favor its sample nullspace (at least d−nd-n-dimensional), producing updates with small sample-based quadratic cost but substantial actual KL drift. In contrast, our diagonal constraint mitigates the influence of the sample nullspace by removing cross-neuron cancellation from the optimization cost. Also, we empirically observe accurate full-Fisher quadratic KL predictions along the tested CANS updates and performance matching damped full Fisher matching the Woodbury-solved damped full Fisher at 1/4,000th of the memory cost. In Point 2 of the reply to W1&Q1, we further clarify two conceptual distinctions between CANS and natural gradient/continual learning.
  2. Verified Approximation Accuracy at Δα⋆\Delta\boldsymbol{\alpha}^{\star} & Removed Post-Hoc Clipping Completely (W2 & Q2): We updated Figure 6 to evaluate local approximations directly at the deployed coefficients α⋆\boldsymbol{\alpha}^{\star}, showing that both first-order and Fisher quadratic estimates closely track measured changes along the CANS trajectory. Furthermore, because the inverse Fisher metric naturally regularizes updates, we removed post-hoc clipping entirely and re-ran all experiments—preserving the exact optimality of Theorem 1 with all empirical conclusions unchanged.
  3. Added Target-Span, Multi-Step, and CANS-Grad-Only Ablations (W1, Q1, Q3): We added direct ablations against whole-response objectives and multi-step iterative updates with K∈{2,4}K\in\{2,4\} steps, and also redefined CANS-Grad-Only as all-neuron gradient scaling (F=I\mathbf{F}=\mathbf{I}) under the identical protocol.

Response to W1 & Q1 (Connection to Damped NGD / EWC, Why Diagonal Fisher Constraint is Methodologically More Favorable, Target-Span Objective, and Multi-Step Updates)

We thank the reviewer for highlighting this important connection. We agree that our constrained optimization shares the foundational local KL–Fisher geometry with damped natural gradient descent (NGD; Amari, 1998) and Fisher-based continual learning (e.g., EWC; Kirkpatrick et al., 2017). We have expanded Section 2 (Related Work) and Section 3.3 to explicitly discuss these connections and clarify what is methodologically new in CANS.

1. Methodological Novelty: Decoupling the Optimization Constraint Metric from KL Curvature Estimation (d≫nd \gg n Regime):

Our methodological insight is to distinguish two roles of the Monte Carlo Fisher estimate: estimating local KL curvature and constraining an adaptively selected update. CANS uses a diagonal constraint to mitigate finite-sample nullspace effects while retaining the full Fisher for KL estimation.

Let F=S⊤S/nF=S^\top S/n be the full Monte Carlo Fisher estimate (sample Fisher for short), with SS stacking nn sampled sequence-score vectors. CANS uses the damped diagonal constraint F=DF+λI\mathbf{F}=D_F+\lambda I, where DF=diag⁡(F11,…,Fdd)D_F=\operatorname{diag}(F_{11},\ldots,F_{dd}) and λ>0\lambda>0.

  • Mitigating Sample Nullspace Effects: A damped full-Fisher step for the same steering objective solves max⁡Δα  g⊤Δα s.t. 12Δα⊤(F+λI)Δα≤ϵq\max_{\Delta\boldsymbol{\alpha}}\;\mathbf{g}^\top\Delta\boldsymbol{\alpha}\ \text{s.t.}\ \frac12\Delta\boldsymbol{\alpha}^\top(F+\lambda I)\Delta\boldsymbol{\alpha}\le\epsilon_{\mathrm{q}}, where ϵq>0\epsilon_{\mathrm{q}}>0 is the quadratic budget. Its direction is (F+λI)−1g(F+\lambda I)^{-1}\mathbf{g}, whereas the undamped quadratic Q(Δα)=12Δα⊤FΔα\mathcal{Q}(\Delta\boldsymbol{\alpha})=\frac12\Delta\boldsymbol{\alpha}^\top F\Delta\boldsymbol{\alpha} estimates KL. Because dd far exceeds nn (e.g., d=350,208d=350,208 and n=4,096n=4,096 for Qwen3-4B), rank⁡(F)≤n\operatorname{rank}(F)\le n and dim⁡ker⁡F≥d−n\dim\ker F\ge d-n. In this sample nullspace, neuron-wise score contributions cancel within each sampled continuation, leaving only the damping cost λ∥Δα∥22/2\lambda\|\Delta\boldsymbol{\alpha}\|_2^2/2. If g\mathbf{g} has a nonzero nullspace component, sufficiently small damping can make that component dominate the update.

  • Decoupled Constraint for Statistical Stability & Empirical KL Validation: Unlike the damped full Fisher, which charges sample-nullspace updates only through damping, our diagonal constraint retains the individual Fisher costs of the participating neurons. This removes cross-neuron cancellation from the optimization cost and mitigates the influence of the sample nullspace on update selection. Proposition 3 further motivates this choice: under independent, zero-mean test updates, the diagonal Fisher minimizes mean squared error to the full-Fisher quadratic among additive neuron-wise costs. We assess prediction accuracy separately by comparing 12(Δα⋆)⊤FΔα⋆\frac{1}{2}(\Delta\boldsymbol{\alpha}^{\star})^\top F\Delta\boldsymbol{\alpha}^{\star} with measured KL along the resulting updates:

Role in OptimizationDamped Full-Fisher StepCANS (Decoupled Geometry)
Optimization ConstraintFull sample curvature: F+λIF+\lambda IAdditive neuron-wise costs: F=DF+λI\mathbf{F}=D_F+\lambda I
Update Direction and Computation(F+λI)−1g(F+\lambda I)^{-1}\mathbf{g}; coupled linear-system solveF−1g\mathbf{F}^{-1}\mathbf{g}; efficient element-wise scaling in O(d)O(d)
Cost of Sample-Nullspace UpdatesDamping only; sampled Fisher contributions cancelIndividual Fisher costs plus damping; no cross-neuron cancellation
Full-Fisher KL Prediction: Empirical ResultsSevere underestimation of measured KL (accuracy evaluation below)Accurate predictions at actual CANS coefficients (Figure 6)
  • Empirical Verification of Nullspace Exploitation vs. Diagonal Regularization:

Approximation accuracy: Across both models, over 99.998% of the update norm lies in the sample nullspace, and the full-Fisher quadratic severely underestimates measured KL, as shown below.

The table reports the nullspace norm ratio ∥Δα0∥2/∥Δαfull∥2\lVert \Delta\boldsymbol{\alpha}_0 \rVert_2 / \lVert \Delta\boldsymbol{\alpha}_{\mathrm{full}} \rVert_2, where Δαfull\Delta\boldsymbol{\alpha}_{\mathrm{full}} is the damped full-Fisher update and Δα0\Delta\boldsymbol{\alpha}_0 is its orthogonal projection onto ker⁡F\ker F. For the same update, we report the signed relative error as 100(Q−K)/K100(\mathcal{Q}-\mathcal{K})/\mathcal{K}, where Q\mathcal{Q} is the full-Fisher quadratic prediction and K\mathcal{K} is the measured KL divergence. A negative error means that the quadratic prediction underestimates the measured KL divergence.

ModelDamping λ\lambdaBudget ϵq\epsilon_{\mathrm{q}}Full-Fisher Predicted KLMeasured KLSigned Relative Error (%)Nullspace Norm Ratio (%)
Qwen3-4B1.07×10−31.07 \times 10^{-3}0.0485.99×10−55.99 \times 10^{-5}0.884−99.993%-99.993\%99.9995%
Llama-2-7B1.55×10−41.55 \times 10^{-4}0.0191.56×10−51.56 \times 10^{-5}0.752−99.998%-99.998\%99.9985%

Downstream performance: The damped full Fisher solved using the Woodbury identity offers no empirical advantage over CANS on Qwen3-4B while incurring ∼4,000×\sim 4,000\times higher storage (5.34 GiB5.34\text{ GiB} vs. 1.34 MiB1.34\text{ MiB}) in the curvature representation.

We use the widely adopted Woodbury identity to solve the damped full-Fisher system; regularization comes from damping. The direct implementation requires O(n2d+n3)O(n^2d+n^3) for Gram-matrix construction and factorization.

Domain (Qwen3-4B)CANS (Diag.) Domain ↓\downarrowDamped Full Fisher (Woodbury) Domain ↓\downarrowDomain Δ\Delta (pp)CANS 5-Bench Avg. ↑\uparrowDamped Full Fisher (Woodbury) 5-Bench Avg. ↑\uparrow5-Bench Δ\Delta (pp)
Safety (ASR ↓\downarrow)8.008.000.0072.3572.26−0.09-0.09
Toxicity (RTP ↓\downarrow)4.174.170.0072.5372.34−0.19-0.19
Hallucination (T&I failure ↓\downarrow)34.2535.00+0.75+0.7572.1272.27+0.15+0.15

2. Additional Conceptual Distinctions from NGD and EWC:

  • Single-Step Weight Folding vs. Iterative Parameter Training: NGD is commonly used for iterative parameter optimization, while EWC regularizes parameter changes during continual learning. Grounded in the hypothesis that FFN neurons encode localized functional features, CANS re-allocates activation magnitudes across existing functional units in a single closed-form step (α∈Rd\boldsymbol{\alpha} \in \mathbb{R}^d) and folds them directly into FFN down-projection weights with zero inference overhead.
  • Decoupled Task-Agnostic Reference Curvature: In common likelihood-based applications of NGD, the gradient and Fisher are computed using the same task distribution. CANS computes the behavioral gradient on the steering data (Dsteer\mathcal{D}_{\mathrm{steer}}) and estimates the Fisher diagonal on the reference data (Dref\mathcal{D}_{\mathrm{ref}}). For a fixed model and reference protocol, the same Fisher diagonal can be reused across steering domains.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Author Responses

We sincerely thank the Area Chair and all three reviewers (Reviewer Kehv, Reviewer Nn81, and Reviewer vMuN) for their constructive, insightful, and rigorous feedback. We are encouraged that the reviewers unanimously found our paper clear and easy to follow (Presentation: 3/3 across all reviewers) and recognized CANS as a coherent, well-structured, and zero-overhead neuron-level steering framework that achieves favorable trade-offs between behavioral control and capability preservation across model families and domains.

In response to the reviewers' valuable suggestions, we have conducted extensive new theoretical analyses, diagnostic evaluations, and benchmark experiments, and have thoroughly updated the manuscript. Below is an executive summary of our major revisions and new findings:

1. Clarifying Methodological Novelty: Decoupled Fisher Trust-Region Geometry in the d≫nd \gg n Regime (Reviewer Kehv: W1, Q1): We expanded Sections 2 and 3.3 to explicitly connect CANS with damped natural gradient descent (NGD) and EWC, and to formalize our core methodological insight: under finite Monte Carlo reference samples (n≪dn \ll d, e.g., n=4,096≪d=350,208n=4,096 \ll d=350,208), decoupling the trust-region constraint metric (damped diagonal F=DF+λI\mathbf{F}=D_F+\lambda I) from the local KL curvature estimator (full Monte Carlo Fisher estimate FF) mitigates sample nullspace exploitation. Optimizing directly against rank-deficient FF routes >99.998%>99.998\% of the update norm into its (d−n)(d-n)-dimensional sample nullspace, causing catastrophic underestimation of actual KL drift (−99.99%-99.99\% relative error). In contrast, our diagonal constraint bounds perturbations coordinate-wise, keeps the update within the well-estimated subspace where 12(Δα⋆)⊤FΔα⋆\frac{1}{2}(\Delta\boldsymbol{\alpha}^{\star})^\top F\Delta\boldsymbol{\alpha}^{\star} accurately predicts true KL drift, and matches Woodbury-regularized Full Fisher performance at ∼1/4,000\sim 1/4,000th the memory cost (1.34 MiB1.34\text{ MiB} vs. 5.34 GiB5.34\text{ GiB}). We also validated that target-span localization is critical (whole-response likelihood fails to reduce ASR) and that single-step CANS (K=1K=1) matches or outperforms iterative multi-step updates (K∈{2,4}K \in \{2, 4\}) without O(K)O(K) computational overhead.

2. Exact Unclipped Closed-Form Solution & Local Approximation Validation at Δα⋆\Delta\boldsymbol{\alpha}^{\star} (Reviewer Kehv: W2, Q2): We updated Figure 6 to evaluate the first-order target-span proxy M(Δα⋆)\mathcal{M}(\Delta\boldsymbol{\alpha}^{\star}) and second-order Fisher quadratic estimate Q(Δα⋆)\mathcal{Q}(\Delta\boldsymbol{\alpha}^{\star}) directly along the actual deployed CANS scaling trajectory Δα⋆\Delta\boldsymbol{\alpha}^{\star}, confirming high approximation fidelity across the operating budget regime. Furthermore, our clipping ablation revealed that while unregularized gradient scaling collapses without clipping, CANS is naturally regularized by inverse-Fisher weighting (Fj+λ)−1(F_j + \lambda)^{-1} and achieves its best trade-off with no clipping at all. Consequently, we removed post-hoc clipping entirely across the paper and re-ran all CANS experiments—ensuring the deployed coefficients are the exact closed-form optimum of Theorem 1 with all empirical conclusions intact.

3. Isolating Fisher Weighting via All-Neuron Gradient Scaling (F=I\mathbf{F} = \mathbf{I}) (Reviewer Kehv: Q3; Reviewer vMuN: W3, Q3): To cleanly isolate the contribution of Fisher curvature weighting from neuron selection, we redefined CANS-Grad-Only across the entire manuscript as all-neuron proportional gradient scaling (Δα=cg\Delta\boldsymbol{\alpha} = c\mathbf{g}, i.e., replacing F\mathbf{F} with I\mathbf{I}) under the exact same protocol. Across all six model-domain settings, CANS consistently preserves substantially higher general capability than CANS-Grad-Only (F=I\mathbf{F}=\mathbf{I}) at matched or stronger steering (e.g., +7.94 pp capability on Llama-3-8B Safety and +13.61 pp on Llama-2-7B Toxicity).

4. Expanding General Capability Evaluation to Coding (MBPP) & Multilingual Math (MGSM) (Reviewer vMuN: W1, Q1): To test whether preserving distributional stability on general English prose (WikiText-2) generalizes to substantially different downstream tasks, we expanded our capability suite from 3 to 5 benchmarks across the entire paper by adding MBPP (code generation) and MGSM (multilingual mathematical reasoning). Across all settings, CANS robustly preserves both coding and multilingual reasoning (often matching or slightly exceeding the unmodified model), whereas baselines without Fisher protection (CANS-Grad-Only, CAA, H-Neurons) suffer severe out-of-distribution capability drops.

5. Evaluating Informative Truthfulness (T&I Failure) & Benign Refusal (XSTest) (Reviewer vMuN: W2, Q2):

To verify that target-span steering induces genuine behavioral improvements rather than trivial abstentions or lexical over-refusal:

  1. Throughout the paper, we now report T&I failure (%) ↓ as 100(1−J/N)100(1-J/N), where NN is the total number of responses and JJ is the number judged both truthful and informative. CANS reduces T&I failure by 12.00 percentage points on Llama-3.1-8B. On Qwen3-4B, CANS achieves the lowest T&I failure among the evaluated methods (34.25%), with a five-benchmark average of 72.12% compared with 72.60% for the unmodified model.
  2. We evaluated benign-request refusal on XSTest (Qwen3-4B Safety), showing that while AurA collapses to a 100%100\% benign refusal rate, CANS halves HarmBench ASR (15.50%→8.00%15.50\% \rightarrow 8.00\%) while maintaining an 86.0% non-refusal rate on XSTest.

6. Mechanistic Layer-Wise Analysis of Neuron Perturbations (Reviewer Nn81: W1 and Q1). We analyzed the mean absolute change in neuron scaling, mean⁡(∣Δαj∣)\operatorname{mean}(|\Delta\alpha_j|), within each FFN layer. Across models and domains, scaling magnitudes consistently peak in intermediate layers. Amplified (Δαj>0\Delta\alpha_j > 0) and attenuated (Δαj<0\Delta\alpha_j < 0) neurons show similar distributions across layers, with both groups concentrated in the same semantically active middle layers. This pattern indicates that CANS concentrates its push–pull modulation at intermediate depths.

We have provided detailed data tables and point-by-point responses in the individual comments to each reviewer. We thank the Area Chair and reviewers again for their time and invaluable feedback, which have substantially strengthened our work!

AuthorsOpenReview ↗

Final Summary for the Area Chair and Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all three reviewers (Reviewer Kehv, Reviewer Nn81, and Reviewer vMuN) for your time, thoughtful engagement, and constructive feedback throughout the review and discussion process. Your suggestions have substantially strengthened both the clarity and empirical rigor of our paper. Below is a concise summary of our work, the reviewers' assessments, and how our revisions address all raised concerns.

1. Paper Overview

  • Background & Motivation: Large language models (LLMs) deployed in high-stakes settings remain prone to unsafe or toxic outputs and factual hallucinations. While fine-tuning methods are effective, they are costly and lack dynamic controllability. Lightweight inference-time neuron steering offers a promising alternative, but existing methods face two major bottlenecks: (1) capability degradation, because polysemantic neurons encode both target behaviors and general skills; and (2) poor cross-domain transfer, due to domain-specific heuristics.
  • Methodological Novelty: We propose Capability-Preserving Neuron Steering (CANS), a unified framework that jointly optimizes target-behavior control (using target-span likelihoods) and general-capability preservation (via output-distribution stability on general reference text) and generalizes across domains effectively.
    • By decoupling the optimization constraint (a damped diagonal Fisher metric) from local distribution-shift estimation (given by the full Fisher Matrix), CANS prevents finite-sample nullspace exploitation and yields a closed-form scaling coefficient for every FFN neuron.
    • These coefficients fold directly into model weights in a single step with zero inference overhead, and a shared data protocol makes adapting to new domains straightforward.
  • Key Results: Evaluated across Qwen3-4B and Llama-2/3/3.1 models on three domains and five capability benchmarks (IFEval, BBH, MMLU, MBPP, MGSM), CANS achieves strong steering with minimal capability loss:
    • Safety (HarmBench): Halves Qwen3-4B attack success rate (15.50% to 8.00%) with only a 0.25-point capability drop and 86.0% benign non-refusal on XSTest; reduces Llama-3-8B ASR (14.00% to 8.75%) with no loss in capability.
    • Toxicity (RealToxicityPrompts): Cuts Llama-2-7B toxic rate from 37.33% to 18.83% (with no capability loss) and Qwen3-4B from 19.17% to 4.17% (-0.07 pp capability).
    • Hallucination (TruthfulQA): Reduces T&I failure by 12.00 pp on Llama-3.1-8B (58.50% to 46.50%, +0.36 pp capability) and achieves the lowest failure rate on Qwen3-4B (34.25%).

2. Reviewer Recognition & Discussion Outcome

All three reviewers unanimously rated Presentation: 3/3 (Clear and easy to follow) and praised CANS for its coherent, zero-overhead closed-form formulation, continuous budget controllability, and strong steering–capability trade-offs across diverse models and behavioral domains. Following the rebuttal:

  • Reviewer Nn81 maintained Clear Acceptance (4).
  • Reviewer Kehv raised their score to Weak Acceptance (3), noting that the additional experiments addressed their main concerns.
  • Reviewer vMuN maintained a positive score (Weak Acceptance, 3), confirming that most questions and concerns were addressed.

3. Summary of Rebuttal Revisions

  1. Clarified Novelty & Design Choices (Kehv W1, Q1): Contrasted CANS with natural gradient descent and continual learning, showing why our decoupled diagonal constraint avoids finite-sample nullspace drift and matches full-matrix performance at a fraction of the memory cost. Added ablations confirming both target-span localization and single-step updates are essential.
  2. Validated Approximations & Removed Coefficient Clipping (Kehv W2, Q2): Verified local approximation accuracy directly at the deployed coefficients and removed post-hoc clipping paper-wide after showing inverse-Fisher weighting naturally regularizes updates and performs best unclipped.
  3. Isolated Fisher Weighting in the baseline CANS-Grad-Only (Kehv Q3; vMuN W3, Q3): Redefined the gradient-only baseline as all-neuron proportional gradient scaling, confirming CANS preserves substantially higher capability (+7.94 pp on Llama-3 Safety; +13.61 pp on Llama-2 Toxicity) at matched or stronger steering.
  4. Expanded to Coding & Multilingual Math (vMuN W1, Q1): Added MBPP and MGSM to form a 5-benchmark capability suite, showing CANS protects specialized circuits where unregularized baselines collapse.
  5. Verified Informative Truthfulness & Benign Refusal (vMuN W2, Q2): Adopted strict T&I Failure (penalizing trivial abstentions) and evaluated XSTest, confirming CANS avoids abstentions and over-refusal.
  6. Added Layer-Wise Mechanistic Analysis (Nn81 W1, Q1): Added Appendix H, showing CANS concentrates coordinated push–pull modulation in semantically active middle FFN layers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.