acceptodds
Under review as a conference paper at ICLR 2027

First-Order Steering: Translating Weight Adaptation into Activation Steering

Abstract

Activation steering exploits interpretable directions in the residual stream to enable inference-time manipulation of model behavior. Composing steering vectors to apply multiple target behaviors simultaneously is important in various fields—including AI alignment and safety—but remains a challenge for existing activation steering methods. In contrast, prior work in model merging shows that target behaviors represented by learned weight adaptations can be combined with high accuracy. A method that translates weight adaptations into activation steering vectors could therefore extend prior work in model merging to generate composable steering vectors that better enable simultaneous inference-time behavioral control. For this, we introduce first-order steering, a formulation of activation steering as a first-order approximation of weight update matrices parameterized by a vector , and establish theoretical bounds on the approximation error of first-order steering. We then develop a novel model merging procedure, HeRD-Merging, which minimizes the first-order approximation error terms to enable higher first-order steering accuracy. Together, our method produces steering vectors that control both individual and composed behaviors more accurately than existing activation steering methods. Furthermore, HeRD-Merging matches the performance of conventional model-merging baselines, while producing weight adaptations that admit more accurate first-order steering vectors.

Primary area:
interpretability and explainable AI
Keywords:
Activation Steering, Representation Engineering, LLM Alignment, LLM Finetuning, Mechanistic Interpretability, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QZpDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether several trained LoRAs can be converted into activation-space intervention vectors that control multiple behaviors at once: it writes model merging as a weight update parameterized by a behavior-strength vector ss, expands a layer's hidden state in ss around zero, and averages the resulting Jacobians over inputs to obtain an input-independent steering vector, with a Taylor remainder bound and a variance bound in terms of a mixed-Jacobian norm QQ. The proposed HeRD-Merging adds low-rank interaction correction terms weighted by products of behavior strengths to the linear adapter sum, training the correction matrices by distribution matching and then optimizing their coefficients against distribution matching, curvature, cross-input variation, and magnitude. On Qwen2.5-VL-3B, Mistral-7B-Instruct-v0.3, and Gemma-4-E2B with three target behaviors (analogies, bullet points, exaggerated complex language), single-layer first-order steering averages 46.4% behavioral adherence with HeRD versus 42.6% for task arithmetic and 30.3% for the strongest non-first-order baseline, Linear-AcT, while HeRD weight merging averages 79.9% against 78.5% for DARE task arithmetic.

Critical strengths

The shift in perspective from weight adaptation to activation intervention is of research value. By mapping existing weight updates into activation space through the Jacobian, a concrete mathematical connection is established between model merging and control during inference. There is also a clear least-squares interpretation for averaging the input-dependent Jacobian. The fundamental structure provides a feasible and verifiable starting point for research and is worth further development.

Critical weaknesses

The analysis and the method do not operate on the same object. The error bounds in the main text are derived for a first-order expansion of the hidden state in the behavior-strength vector, but HeRD's correction terms are products of at least two behavior strengths, so both their values and first derivatives vanish at zero and they cannot contribute to the first-order term; as the paper acknowledges, the resulting steering vectors are then equivalent to task arithmetic. To let the corrections matter, the appendix switches to a linearization in the weights instead, which changes what is being approximated: the original analysis controls the curvature of the layer output as a function of the strengths, whereas the accuracy of the generalized intervention described in Appendix A.2 depends on the error incurred by linearizing in the weights, as well as the error from averaging over inputs. The two are not interchangeable, because the chain rule allows the curvature contributions to cancel, so small curvature in strength space does not imply a small weight-linearization error. The assertion that the generalization leaves the Hessian-regularization results unchanged is therefore unsupported, and no condition under which it would hold is stated. This is consequential because the paper's theoretical contribution is precisely the link from Hessian regularization to steering accuracy: as written, the stated bounds do not directly establish an error guarantee for that generalized intervention, and no experiment reports the actual approximation error or ablates the curvature and cross-input variation penalties that are supposed to drive the reported gains.

Questions for authors
  1. Please confirm whether the HeRD steering experiments use JWh(W0)[△W(s)]J_Wh(W_0)[\triangle W(s)] from Appendix A.2 and specify the full intervention, including how it is averaged over inputs. If this expression is used, please provide applicable bounds for both the weight-linearization error and the input-averaging error, and explain how the regularization terms defined through derivatives with respect to ss control these errors.

  2. Using the actual intervention implemented in the experiments, how do the activation approximation error and behavioral adherence change when r2r_2 is removed, when r3r_3 is removed, and when both are removed? Please keep the correction-matrix parameterization, first-stage training, second-stage initialization, and optimization budget fixed, changing only the inclusion of these loss terms. Please also report the resulting r2r_2 and r3r_3 values for all variants. These matched comparisons would help distinguish the effects of regularization from those of the additional parameters and training.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Dear Reviewer QZpD,

We sincerely appreciate the thorough and constructive feedback you have provided on our paper. We have carefully incorporated your feedback and have responded to your suggestions below. Please let us know if there are any outstanding comments or concerns.

Question 1

Please confirm whether the HeRD steering experiments use JWh(W0)[ΔW(s)]J_{W} h(W_0) [\Delta W(s)] from Appendix A.2 and specify the full intervention, including how it is averaged over inputs.

Yes, the HeRD steering experiments use JWℓh(x,Wbase)[ΔW(s)]J_{W_\ell} h(x, W^{\textrm{base}}) [\Delta W(s)], and we now explained this with more detail in Section 4.2 and Section A of our paper. Namely, we clarify that the approximation used to motivate first-order steering with HeRD-Merging is

hℓ′(x,s)≈hℓ(x)+JWℓhℓ′(x,0)[ΔW(s)].h_{\ell}'(x, s) \approx h_{\ell}(x) + J_{W_\ell} h_{\ell}'(x, 0) [\Delta W(s)].

The averaging over inputs is done by generating a steering function (which generalizes the steering matrix A∗A^* from Section 3) defined by

A∗(s)=1M∑m=1MJWℓhℓ′(xm,0)[ΔWℓ(s)]A^*(s) = \frac{1}{M}\sum_{m=1}^M J_{W_\ell} h_\ell'(x_m, 0)[\Delta W_\ell(s)]

so that the final intervention is

hℓ′(x,s)=hℓ(x)+A∗(s).h_{\ell}'(x, s) = h_{\ell}(x) + A^*(s).

This information is now clearly outlined in Section 4.2 and Appendix A of our updated paper.

If this expression is used, please provide applicable bounds for both the weight-linearization error and the input-averaging error, and explain how the regularization terms defined through derivatives with respect to ss control these errors.

Because we previously clarified the intervention from first-order steering in the appendix rather than in the main paper, we did not use this notation to describe the residuals. Following your comments, we realized that this is ambiguous. We have therefore corrected this in Section 4.4 by replacing the residuals as follows.

Let s∗s^{\ast} be a unit vector constructed by sampling uniformly from {−1,1}n\{-1, 1\}^n and then scaling by 1/n1/\sqrt{n}. We define D∗=ΔWℓ(s∗)D^{\ast} = \Delta W_\ell(s^{\ast}) so that

r2=∥JWℓ(JWℓhℓ′(x,Wℓbase)[D∗])[D∗]∥∥JWℓhℓ′(x,Wℓbase)[D∗]∥r_2 = \frac{\|J_{W_\ell} (J_{W_\ell} h_\ell' (x, W_\ell^{\mathrm{base}}) [D^{\ast}]) [D^{\ast}]\|}{\|J_{W_\ell} h_\ell' (x, W_\ell^{\mathrm{base}}) [D^{\ast}]\|}

and

r3=∥Covx∼D[JWℓhℓ′(x,Wℓbase)[D∗]]∥FE[∥JWℓhℓ′(x,Wℓbase)[D∗]∥2].r_3 = \frac{\lVert\mathrm{Cov}_{x \sim \mathcal{D}}[J_{W_\ell} h_\ell' (x, W_\ell^{\mathrm{base}}) [D^{\ast}]]\rVert_F}{\mathbb{E}[\lVert J_{W_\ell} h_\ell' (x, W_\ell^{\mathrm{base}}) [D^{\ast}]\rVert^2]}.

Importantly, the residuals used for fitting the reported HeRD-Merging adapters do not use derivatives with respect to ss. The applicable error bounds which motivate these residuals are now shown in Section 4.2 and Section A (Proposition 4).

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Question 2

Using the actual intervention implemented in the experiments, how do the activation approximation error and behavioral adherence change when r2r_2 is removed, when r3r_3 is removed, and when both are removed?

We have now introduced Table 9, which provides the results of the ablation study that you requested. We summarize the results below by taking averages.

MethodAnalogy UseBulletpoint UseSophisticated LanguageAverage
First-Order Steering (HeRD-Merging)19.865.953.646.4
First-Order Steering (Task Arithmetic)20.263.544.242.6
Without r2r_217.363.947.442.9
Without r3r_316.462.352.443.7
Without r2r_2 and r3r_317.463.947.442.9

Notably, removal of one or both of the residuals r2,r3r_2, r_3 noticeably reduces the performance of the method, suggesting that Hessian regularization indeed contributes to the performance of first-order steering with HeRD-Merging.

We have also computed the average approximation error and relative hessian norms (defined in Section C.2) before/after the Levenberg-Marquardt Algorithm (LMA) for each model.

QwenRelative Hessian Norm (%)Average Relative Approximation Error (%)
Before LMA35.4516.31
After LMA23.4111.66
Task arithmetic20.5710.46
MistralRelative Hessian Norm (%)Average Relative Approximation Error (%)
Before LMA72.7333.13
After LMA52.5925.27
Task arithmetic45.0021.28
GemmaRelative Hessian Norm (%)Average Relative Approximation Error (%)
Before LMA42.6023.65
After LMA39.6722.49
Task arithmetic34.5620.44

This suggests that the LMA is contributing meaningfully to reducing the first-order steering error, despite the initial error accumulated from the addition of the interaction matrices RSR_S. We have added these results to Table 10 in our updated paper.

Please also report the resulting r2r_2 and r3r_3 values for all variants.

We provide the residuals with the same ablations below. Including the regularization terms generally reduces the corresponding residuals, although the differences are small for some models.

Qwenr2r_2r3r_3
First-Order Steering (HeRD-Merging)0.65890.0329
Without r2r_20.70780.0346
Without r3r_30.68730.0342
Without r2,r3r_2,r_30.70780.0346
Mistralr2r_2r3r_3
First-Order Steering (HeRD-Merging)1.71210.0686
Without r2r_21.71250.0686
Without r3r_31.71210.0686
Without r2,r3r_2,r_31.71250.0686
Gemmar2r_2r3r_3
First-Order Steering (HeRD-Merging)3.00690.2474
Without r2r_23.11350.2498
Without r3r_33.00690.2474
Without r2,r3r_2,r_33.11350.2498

Here, we note that even if the values of some residuals r2r_2 and r3r_3 are the same between ablations (e.g. without r3r_3 vs. with all residuals), it is still possible for the overall behavioral adherence to change, as displayed in Table 9. This is because the computed values of ηS\eta_S are different between ablations, which produces different steering vectors and therefore produces different generations under greedy decoding. For reference, we include the computed values of ηS\eta_S for each variant and model below.

Mistralη12\eta_{12}η13\eta_{13}η23\eta_{23}η123\eta_{123}
All residuals0.44310.92540.14210.4232
Without r2r_20.20720.20220.22120.0000
Without r3r_30.20480.20270.22300.0000
Without r2,r3r_2,r_30.20720.20220.22120.0000
Gemmaη12\eta_{12}η13\eta_{13}η23\eta_{23}η123\eta_{123}
All residuals0.84870.83820.72170.7455
Without r2r_20.35310.81370.49570.0000
Without r3r_30.91590.50821.00000.0000
Without r2,r3r_2,r_30.35310.81360.49560.0000
Qwenη12\eta_{12}η13\eta_{13}η23\eta_{23}η123\eta_{123}
All residuals0.00000.88350.44670.0029
Without r2r_20.34490.56420.65150.3385
Without r3r_30.33080.55920.64530.3731
Without r2,r3r_2,r_30.34490.56420.65150.3386
AuthorsOpenReview ↗

Request for Follow-Up Feedback on Rebuttal

Dear Reviewer QZpD,

Thank you once again for your feedback on our paper. We have responded to the points that you have identified in our rebuttal, and we would like to ask if our responses have sufficiently addressed your concerns. Please do not hesitate to let us know if there are any additional points which remain unclear. We greatly appreciate the opportunity to improve our paper.

Reviewer_aefbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes translating weight adaptations into activation steering through a first-order approximation, together with herd-merging to improve approximation quality. The connection between weight adaptation and activation steering is interesting, and evaluating three model families across all eight combinations of three behaviours is a useful contribution. The reported improvements are promising, particularly on Qwen.

Critical strengths

The paper tests all eight combinations of three behaviors across three model families and compares against multiple steering baselines. This directly addresses an important challenge in activation steering: whether interventions can support several requested behaviors simultaneously.

Critical weaknesses
  1. The main weakness is the mismatch between the mathematical formulation and the linked implementation. Equation (50 defines steering vectors through the Jacobian with respect to behaviour strengths ss at s=0s=0. However, every HERD correction in Equation (7) has degree at least two in ss, so its first derivative vanish at the origin. With fixed original adapters and averaging inputs, HeRD therefore gives exactlyu the same steering vectors as task arithmetic under the stated formulation. It can change the nonlinear merged model and its approximation error, but cannot change the resulting steering model’s outputs. It is unclear how HeRD training improves activation steering under the current formulation. Equation (5) extracts steering vectors by averaging Jshℓ′(x,0)J_s h'_\ell(x,0), while the interaction corrections in Equation (7) have degree at least two in ss, so their first derivatives vanish at the origin. With fixed original adapters and averaging inputs, this appears to leave A∗A^* unchanged relative to task arithmetic. Although HeRD could change the merged model and reduce its discrepancy from the first-order approximation, it is unclear how this changes the extracted steering vectors or their behavioral performance. Could the authors clarify how the learned interaction corrections affect Equation (5), and how the stated formulation accounts for the reported steering improvements?

  2. The second weakness is the unclear practical benefit over directly using merged LoRAs. The method already requires LoRA training, followed by additional merging and extraction procedures. Smaller steering vectors and flexible inference-time control are plausible benefits, but no measured latency, memory, or throughput comparison establishes their value relative to applying the merged adapters directly. A controlled accuracy vs cost comparison would clarify whether the conversion offers a worthwhile trade-off. Overall, the research direction is worthwhile, but the. current formulation raises question about what actually works.

Questions for authors
  1. Equation (5) extracts Jshℓ′(x,0)J_s h'_\ell(x,0), but the HeRD corrections in Equation (7) have zero first derivatives at s=0s=0. With fixed original adapters and averaging inputs, how do these corrections change A∗A^* and account for the reported steering improvements?

  2. What advantage does conversion to activation steering offer over directly using merged LoRAs? Do you have existing measurements of memory, latency, or throughput that establish a worthwhile trade-off with behavioral accuracy?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/1)

Dear Reviewer aefb,

We sincerely appreciate the thorough and constructive feedback you have provided on our paper. We have carefully incorporated your feedback and have responded to your suggestions below. Please let us know if there are any outstanding comments or concerns.

Question 1

Equation (5) extracts Jshℓ′(x,0)J_s h'_\ell(x, 0), but the HeRD corrections in Equation (7) have zero first derivatives at s=0s = 0. With fixed original adapters and averaging inputs, how do these corrections change and account for the reported steering improvements?

It is true that under the formulation stated, the first-order steering vectors from HeRD-Merging and task arithmetic should be equal. Our previous version of the paper had attempted to clarify this in the appendix, but you are right that there was substantial ambiguity in the steering procedure for HeRD-Merging. We have now fixed this in Section 4.2 and Appendix A of the updated paper and summarize the correction below.

For a merging rule ΔW\Delta W like HeRD-Merging which satisfies

JsΔW(0)ek=ΔWk,J_s \Delta W(\mathbf 0) e_k = \Delta W_k,

we may apply the chain rule to write

Jshℓ′(x,0) s=∂hℓ′∂W(x,0)[JsW(x,0) s]=∂hℓ′∂W(x,0)[∑k=1nskΔWk].J_s h_\ell'(x, \mathbf{0})\,s = \frac{\partial h_\ell'}{\partial W}(x,\mathbf{0})\left[J_s W(x,\mathbf{0})\,s\right] = \frac{\partial h_\ell'}{\partial W}(x,\mathbf{0})\left[\sum_{k=1}^n s_k \Delta W_k\right].

The first-order steering rule used for HeRD-Merging is achieved by observing that if ΔW\Delta W is given by task arithmetic then

∂hℓ′∂W(x,0)[∑k=1nskΔWk]=∂hℓ′∂W(x,0)[ΔW(s)].\frac{\partial h_\ell'}{\partial W}(x,\mathbf{0}) \left[\sum_{k=1}^n s_k \Delta W_k\right] = \frac{\partial h_\ell'}{\partial W}(x,\mathbf{0})\left[\Delta W(s)\right].

We use this observation to define, for a merging rule ΔW\Delta W, the first-order steering update as hℓ′(x,s)≈hℓ(x)+JWℓhℓ′(x,0)[ΔW(s)].h_{\ell}'(x, s) \approx h_{\ell}(x) + J_{W_\ell} h_{\ell}'(x, \mathbf{0}) [\Delta W(s)]. Choosing ΔW\Delta W to be given by the HeRD adapter amounts to having the ordinary task arithmetic first-order steering vectors in addition to first-order steering vectors for each learned matrix RSR_S in the HeRD adapter.

These additional steering terms are why HeRD-Merging has the capacity to be more accurate than task arithmetic.

Question 2

What advantage does conversion to activation steering offer over directly using merged LoRAs?

Our paper does not propose activation steering as a cheaper alternative to merged LoRA adapters, and we do not intend to present first-order steering as a parameter-efficient finetuning (PEFT) method that competes with LoRA. Instead, we intend for first-order steering to be an advancement in mechanistic interpretability.

Much of the existing literature in mechanistic interpretability rests on the linear representation hypothesis, which suggests that high-level concepts are encoded in LLMs as directions or subspaces of their residual stream [1]. This view has numerous consequences. For instance, Testing with Concept Activation Vectors [2] uses a fixed steering direction to identify the sensitivity of a model's output relative to a certain concept, and Concept Influence [3] uses steering vectors to perform training data attribution.

Typical PEFT methods like LoRA or LoReFT [4] do not directly produce the fixed activation-space directions used in these interpretability methods. Even steering methods like Linear-AcT or ODESteer do not compute fixed vectors that can be easily combined with existing interpretability methods.

The advantage of our method is that, because it produces a fixed intervention analogous to traditional steering methods like RepE or CAA, it can be interfaced directly with prior interpretability work while exceeding the steering performance of activation-space PEFT counterparts (Table 1 and Table 6).

[1] The Linear Representation Hypothesis and the Geometry of Large Language Models

[2] Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

[3] Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution

[4] ReFT: Representation Finetuning for Language Models

Thank you for the clarification and revisions. The weight-space formulation in Eqs. (12)–(13) explains how the interaction terms can affect composed steering. This changes the formulation underlying my original concern, however. Please explicitly confirm that these equations describe the implementation used for the reported results, and update the introduction’s description of differentiation with respect to ss accordingly.

My remaining concern is the connection between the revised intervention and its error analysis. Proposition 4 bounds the error from averaging over inputs, while the curvature penalty is evaluated at the base weights. Reducing this penalty does not by itself establish control of the linearization error along the full weight update. The added ablations and diagnostics are helpful, but the paper should clearly distinguish the proven bound from the empirical justification for the regularizer.

I also appreciate the clarification that the intended benefit is obtaining fixed activation-space directions; I do not consider additional efficiency comparisons necessary.

These are requests to clarify the existing formulation and evidence, rather than run new experiments. Given the remaining uncertainty in the central error-control argument, I am keeping my rating for now.

AuthorsOpenReview ↗

Reply to Follow-Up (1/1)

Dear Reviewer aefb,

We appreciate the time you have spent in evaluating our revision. We have clarified the remaining points you have raised below.

Please explicitly confirm that these equations describe the implementation used for the reported results, and update the introduction's description of differentiation with respect to ss accordingly.

Yes, these equations and the updated Hessian regularization terms in Section 4.4 match the implementation used for all first-order steering results. Thank you for mentioning the discrepancy in the introduction section. We have updated this in Section 1 of our paper.

Proposition 4 bounds the error from averaging over inputs, while the curvature penalty is evaluated at the base weights. Reducing this penalty does not by itself establish control of the linearization error along the full weight update. The added ablations and diagnostics are helpful, but the paper should clearly distinguish the proven bound from the empirical justification for the regularizer.

We have included an explanation in Section A which distinguishes the proven bound from the regularization algorithm used in practice. We summarize this update below.

You are right that the entire line along the weight update is not handled by the LMA. By Taylor's theorem and the mean value inequality (Proposition 3), the linearization bound for generalized first-order steering is ∣R(ΔW(s))∣≤12∣ΔW(s)∣2sup⁡t∈[0,1]∣JWℓ2hℓ′(x,Wℓbase+tΔW(s))∣F.\\|R(\Delta W(s))\\| \leq \frac{1}{2} \\|\Delta W(s)\\|^2 \sup_{t \in [0, 1]} \\|J_{W_\ell}^2 h'_\ell(x, W_\ell^{\mathrm{base}} + t\Delta W(s))\\|_F. This motivates the choice of r2r_2 in Section 4.4 as a heuristic for reducing the linearization remainder, but r2r_2 does not necessarily reduce the curvature along the full weight update sup⁡t∈[0,1]∣JWℓ2hℓ′(x,Wℓbase+tΔW(s))∣\sup_{t \in [0, 1]} \\|J_{W_\ell}^2 h'_\ell(x, W_\ell^{\mathrm{base}} + t\Delta W(s))\\| since we only consider evaluation at t=0t = 0. This observation is the primary difference between r2r_2 and the theoretical bound on the entire approximation error, and we have made this explicit in Section A.

The reason we accept this limitation of r2r_2 is purely practical. By using t=0t = 0, we can reuse Jacobian-vector products computed for the denominator of r2r_2 for the computation of the numerator, and it prevents repeated JVPs for varying values of tt. Nevertheless, using r2r_2 as a proxy is still well-motivated theoretically. In particular, r2r_2 is proportional to the second-order term in the remainder if we consider a higher-order approximation at h′(x,Wbase).h'(x, W^{\textrm{base}}). This explanation is in Section A, but we have now expanded this to explicitly include r~2\widetilde r_2, an alternative formulation of r2r_2 which more closely approximates the supremum but is significantly more expensive. For convenience, we include both r2r_2 and r~2\widetilde r_2 below.

Let s∗s^{\ast} be a unit vector constructed by sampling uniformly from −1,1n\\{-1, 1\\}^n and then scaling by 1/n1/\sqrt{n}. We define D∗=ΔWℓ(s∗)D^{\ast} = \Delta W_\ell(s^{\ast}) so that r2=∣JWℓ(JWℓhℓ′(x,Wℓbase)[D∗])[D∗]∣∣JWℓhℓ′(x,Wℓbase)[D∗]∣r_2 = \frac{\\|J_{W_\ell} (J_{W_\ell} h_\ell' (x, W_\ell^{\mathrm{base}}) [D^{\ast}]) [D^{\ast}]\\|}{\\|J_{W_\ell} h_\ell' (x, W_\ell^{\mathrm{base}}) [D^{\ast}]\\|} and r~2=Et∼U([0,1])∣JWℓ(JWℓhℓ′(x,Wℓbase+tΔW(s))[D∗])[D∗]∣∣JWℓhℓ′(x,Wℓbase)[D∗]∣.\widetilde r_2 = \frac{\mathbb{E}_{t \sim U([0, 1])} \\|J_{W_\ell} (J_{W_\ell} h_\ell' (x, W_\ell^{\mathrm{base}} + t\Delta W(s)) [D^*]) [D^*]\\|}{\\|J_{W_\ell} h_\ell' (x, W_\ell^{\mathrm{base}}) [D^*]\\|}.

We appreciate your detailed feedback and the opportunity to further refine our paper.

Reviewer_eAbpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to convert weight adaptations into activation steering vectors so that several target behaviors can be composed at inference time. Task-arithmetic strengths parameterize the merged model over a set of LoRA adapters, and each hidden layer is linearized in these strengths around zero. The Jacobian columns, averaged over prompts, serve as per-behavior steering vectors, so composition reduces to vector addition. The authors bound the linearization error and the prompt dependence of the averaged Jacobian by Hessian norms of the hidden layers. Building on this, they propose HeRD-Merging, which adds learnable low-rank interaction terms to task arithmetic. These terms are first trained to match a geometric interpolation of the single-adapter output distributions, and their scales are then fitted with Levenberg–Marquardt against proxies for Hessian size and prompt variance. Experiments on three stylistic behaviors (analogy use, bullet points, and sophisticated language) with Qwen, Mistral, and Gemma models report higher behavioral adherence than contrastive and transport-based steering baselines, and merging performance comparable to task arithmetic, DARE, TIES, and KnOTS.

Critical strengths

S1. Principled bridge from merging to steering

Reading steering vectors off the Jacobian of an ss-parameterized merged model is a simple idea with a clear rationale. It carries the known composability of task arithmetic over to activation steering by construction. The paper further shows that task arithmetic is the first-order term of any differentiable merging rule whose derivative at zero recovers each adapter. Composing steering vectors for multiple behaviors is a recognized open problem, and this formulation addresses it without relying on heuristics.

S2. Explicit decomposition of steering error

The analysis separates the steering error into a Taylor remainder and a prompt-dependence term, both bounded by Hessian norms. Most existing steering methods offer no comparable characterization of when a vector can be trusted. The same construction also yields steering vectors at every adapted layer at no extra cost, and the all-layer results on Qwen show that these vectors can induce the target behaviors substantially.

Critical weaknesses

W1. Unsupported benefit of Hessian regularization

Every correction term in the HeRD adapter is a product ∏t∈Sst\prod_{t\in S} s_t with ∣S∣≥2|S|\ge 2, and its derivative with respect to any sks_k vanishes at s=0s=0. The Jacobian of the HeRD adapter at s=0s=0 therefore equals that of task arithmetic. By the chain-rule argument given in the paper, the prompt-averaged steering vectors would then coincide for the two methods, and under greedy decoding they should produce identical outputs. This is consistent with the near-identical example generations for Mistral and Gemma and the 0.2-point gap on Gemma, but not with the 9.3-point gap on Qwen. The paper presents Hessian norms as a quantifiable signal of steering accuracy, yet no experiment reports a Hessian norm, the value of either bound, the regularization residuals before and after optimization, or the actual linearization error. The remainder bound also holds for small ∥s∥\|s\|, while evaluation is carried out at the corners of [0,1]n[0,1]^n, where ∥s∥2\|s\|^2 reaches 3. The second contribution, that HeRD-Merging yields more accurate steering vectors through Hessian regularization, is therefore not supported by the current evidence, and the source of the Qwen gap is not explained.

W2. Unclear source of the steering gains

First-order steering relies on about 10k supervised examples per behavior and LoRA training, whereas the baselines are training-free or lightly fitted. Methods that also learn interventions from supervised data, such as ReFT[1], BiPO[2] and LoFiT[3], are neither discussed nor compared. Prior work has steered models with the activation difference between a fine-tuned model and its base model[4]. The corresponding finite-difference vector from the same adapters, Ex[hℓ′(x,ek)−hℓ(x)]\mathbb{E}_x[h'_{\ell}(x,e_k)-h_{\ell}(x)], is also absent as a reference. In addition, composability is the main motivation of the paper, yet adherence is pooled over all combinations that contain a behavior, and the joint success rate, the off-target rate and the unsteered rate are not reported. It therefore remains unclear whether the gains come from the first-order formulation or from the additional supervision, and to what extent multiple behaviors are composed reliably.

[1]: ReFT: Representation Finetuning for Language Models

[2]: Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization

[3]: LoFiT: Localized Fine-tuning on LLM Representations

[4]: Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences

Questions for authors

Clarifications that could affect the assessment:

  1. At which value of ss is the steering Jacobian evaluated for the HeRD and task-arithmetic variants? Given the paper's own first-order argument, what accounts for the different results on Qwen, and what is the cosine similarity between the two sets of steering vectors?
  2. Do the Hessian and variance residuals decrease after the Levenberg–Marquardt stage, and does the measured linearization error at the corners of [0,1]n[0,1]^n relate to adherence across layers, behaviors, and models?
  3. How does the first-order vector compare with the finite-difference vector Ex[hℓ′(x,ek)−hℓ(x)]\mathbb{E}_x[h'_{\ell}(x,e_k)-h_{\ell}(x)] from the same adapters, and with supervised interventions such as ReFT[1], BiPO[2] and LoFiT[3]?

Suggestions for future work:

  • Reporting the joint success rate, the off-target rate, and the unsteered rate for each behavior combination may help substantiate the composability claim.
  • The relation to linearized fine-tuning, such as task arithmetic in the tangent space[4], may be worth discussing.
  • Minor presentation points: the base term of the first-order expansion appears to be hℓ,i(x)h_{\ell,i}(x) rather than hℓ,i(x,s)h_{\ell,i}(x,s); the argument in the condition on the merging Jacobian contains a typographical error; the bold entry in the analogy column of the main steering table does not mark the highest value; all-layer steering is described as free of output degradation, while the Mistral results decrease; MiMiC appears to be attributed to a different paper than Representation Surgery.

[1]: ReFT: Representation Finetuning for Language Models

[2]: Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization

[3]: LoFiT: Localized Fine-tuning on LLM Representations

[4]: Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models

I would be happy to raise the score if the author could provide more evidence to answer my question.

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

Dear Reviewer eAbp,

We sincerely appreciate the thorough and constructive feedback you have provided on our paper. We have carefully incorporated your feedback and responded to your suggestions below. Please let us know if there are any outstanding comments or concerns.

W1: Unsupported benefit of Hessian regularization (related to Question 1)

The Jacobian of the HeRD adapter at s=0s = 0 therefore equals that of task arithmetic. By the chain-rule argument given in the paper, the prompt-averaged steering vectors would then coincide for the two methods, and under greedy decoding they should produce identical outputs.

It is true that under the formulation stated, the first-order steering vectors from HeRD-Merging and task arithmetic should be equal. Our previous version of the paper had attempted to clarify this in the appendix, but you are right that there was substantial ambiguity in the steering procedure for HeRD-Merging. We have now fixed this in Section 4.2 and Appendix A of the updated paper and summarize the correction below.

For a merging rule ΔW\Delta W like HeRD-Merging which satisfies

JsΔW(0)ek=ΔWk,J_s \Delta W(0) e_k = \Delta W_k,

we may apply the chain rule to write

Jshℓ′(x,0) s=∂hℓ′∂W(x,0)[JsW(x,0) s]=∂hℓ′∂W(x,0)[∑k=1nskΔWk].J_s h_\ell'(x, \mathbf{0})\,s = \frac{\partial h_\ell'}{\partial W}(x,\mathbf{0})\left[J_s W(x,\mathbf{0})\,s\right] = \frac{\partial h_\ell'}{\partial W}(x,\mathbf{0})\left[\sum_{k=1}^n s_k \Delta W_k\right].

The first-order steering rule used for HeRD-Merging is achieved by observing that if ΔW\Delta W is given by task arithmetic then

∂hℓ′∂W(x,0)[∑k=1nskΔWk]=∂hℓ′∂W(x,0)[ΔW(s)].\frac{\partial h_\ell'}{\partial W}(x,\mathbf{0}) \left[\sum_{k=1}^n s_k \Delta W_k\right] = \frac{\partial h_\ell'}{\partial W}(x,\mathbf{0})\left[\Delta W(s)\right].

We use this observation to define, for a merging rule ΔW\Delta W, the first-order steering update as hℓ′(x,s)≈hℓ(x)+JWℓhℓ′(x,0)[ΔW(s)].h_{\ell}'(x, s) \approx h_{\ell}(x) + J_{W_\ell} h_{\ell}'(x, \mathbf{0}) [\Delta W(s)]. Choosing ΔW\Delta W to be given by the HeRD adapter, this amounts to having the ordinary task arithmetic first-order steering vectors in addition to first-order steering vectors for each learned matrix RSR_S in the HeRD adapter.

These additional steering terms are why HeRD-Merging has the capacity to be more accurate than task arithmetic.

This is consistent with the near-identical example generations for Mistral and Gemma and the 0.2-point gap on Gemma, but not with the 9.3-point gap on Qwen.

The near-identical examples between first-order steering with HeRD-Merging vs. task arithmetic are expected, as the provided example generations are for when only one target behavior is active. Notably, based on the previous explanation, the steering terms only differ in activation space when they do in weight space, i.e. when more than one target behavior is requested.

The Qwen example generation which displayed slightly different generations is a data entry error where the HeRD-Merging generation was mistakenly taken from a different experiment. We now verified that both outputs are exactly the same for both task arithmetic and HeRD-Merging when only one behavior is active, and corrected this in Section C.5 of the paper.

What is the cosine similarity between the two sets of steering vectors?

First-order steering with HeRD-Merging generates the steering vectors of task arithmetic as well as steering vectors for each interaction matrix RSR_S. So, we report the cosine similarity between the resulting steering vectors for combinations of one, two, and three active target behaviors, taking averages whenever there is more than one possible combination.

ModelOne Behavior (avg.)Two Behaviors (avg.)Three Behaviors
Qwen2.5-VL-3B1.0000.9320.828
Gemma-4-E2B1.0000.9720.915
Mistral-7B1.0000.9570.824

As expected, the cosine similarity between first-order steering with HeRD-Merging and with task arithmetic becomes smaller as more directions are active. The steering vectors also have high cosine similarities, and this is consistent with the fact that they are meant to be semantically similar. However, it is certainly clear that that they are not the same.

We also provide the fitted values of ηS\eta_S for further reference.

Modelη12\eta_{12}η13\eta_{13}η23\eta_{23}η123\eta_{123}
Qwen2.5-VL-3B0.00000.88350.44670.0029
Gemma-4-E2B0.84870.83820.72170.7455
Mistral-7B0.44310.92540.14210.4232

We have also added these tables to our paper for reference (Tables 11 and 12).

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W1: Unsupported benefit of Hessian regularization (related to question 2)

The paper presents Hessian norms as a quantifiable signal of steering accuracy, yet no experiment reports a Hessian norm, the value of either bound, the regularization residuals before and after optimization, or the actual linearization error.

The previous version of our paper indeed does not provide data which empirically supports the utility of Hessian regularization as being a signal for first-order steering accuracy. We have corrected this by computing the average approximation error and relative hessian norms (defined in Section C.2) before/after the Levenberg-Marquardt Algorithm (LMA) for each model. Analyzing these quantities explains the higher first-order steering performance on Qwen relative to Mistral and Gemma.

QwenRelative Hessian Norm (%)Average Relative Approximation Error (%)
Before LMA35.4516.31
After LMA23.4111.66
Task arithmetic20.5710.46
MistralRelative Hessian Norm (%)Average Relative Approximation Error (%)
Before LMA72.7333.13
After LMA52.5925.27
Task arithmetic45.0021.28
GemmaRelative Hessian Norm (%)Average Relative Approximation Error (%)
Before LMA42.6023.65
After LMA39.6722.49
Task arithmetic34.5620.44

The highest performing model (Qwen) has the lowest relative Hessian norm and approximation error with HeRD-Merging (23.41, 11.66) and with task arithmetic (20.57, 10.46). Mistral and Gemma both have higher approximation errors (see table immediately above), consistent with their lower first-order steering performance relative to Qwen.

We also remark that while task arithmetic has lower Hessian norm and approximation error than HeRD-Merging, this is an expected outcome. These results demonstrate that the LMA substantially reduces the error difference between HeRD-Merging and task arithmetic, even when the inclusion of interaction matrices RSR_S increases the Hessian norm. The additional approximation error is offset by the higher behavioral adherence of HeRD-Merging when composing steering vectors.

We have also done an ablation study to understand the impact of the Hessian regularization residuals r2r_2 and r3r_3 on behavioral adherence. Consistently, removing one or both of these terms reduces the adherence rates. The details are in Table 9 of our paper, and we summarize the results below by taking averages:

MethodAnalogy UseBulletpoint UseSophisticated LanguageAverage
First-Order Steering (HeRD-Merging)19.865.953.646.4
First-Order Steering (Task Arithmetic)20.263.544.242.6
Without r2r_217.363.947.442.9
Without r3r_316.462.352.443.7
Without r2r_2 and r3r_317.463.947.442.9

W2: Unclear source of the steering gains (related to question 3)

Methods that also learn interventions from supervised data, such as ReFT, BiPO and LoFiT, are neither discussed nor compared.

In addition, composability is the main motivation of the paper, yet adherence is pooled over all combinations that contain a behavior, and the joint success rate, the off-target rate and the unsteered rate are not reported.

We have now compared our method to BiPO, LoReFT, and LoFiT. In doing this, we also decided to distinguish methods which add fixed interventions at token position—which we term static steering methods—from methods which recompute the intervention at every step—termed dynamic steering methods. Our method is a static steering method.

Results are available for static steering methods in Table 1 and dynamic steering methods in Table 6, with more details in Tables 2, 7, and 8. To evaluate the strength of our method in composing steering vectors, we now also report the joint adherence (which rewards the method for each present behavior) and exact adherence (which additionally penalizes the presence of unrequested behaviors). We summarize the results below.

Steering MethodAverageJoint AdherenceExact Adherence
First-Order (HeRD-Merging)46.436.934.6
First-Order (Task Arithmetic)42.634.532.1
LoReFT45.133.233.9
ODESteer23.210.912.7
Linear-AcT30.318.617.1
RE-Control12.96.112.5
BiPO20.212.114.0
LoFiT22.915.620.3
CAA21.712.913.3
RepE24.112.215.2

First order steering with HeRD-Merging outperforms the supervised methods BiPO, LoReFT, and LoFiT in addition to all the other baselines. The precise details of the experiments are in Section C.1 and Section 5.3.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

How does the first-order vector compare with the finite-difference vector Ex[hℓ′(x,ek)−hℓ(x)]\mathbb{E}_x[h'_{\ell}(x,e_k)-h_{\ell}(x)] from the same adapters?

We have labeled the method you describe "Adapter CAA" and take the expectation over our synthetically generated dataset. The results are summarized below with details in Table 5 of our paper.

Steering MethodAnalogy UseBulletpoint UseSophisticated LanguageAverage
First-Order (HeRD-Merging)19.865.953.646.4
First-Order (Task Arithmetic)20.263.544.242.6
Adapter CAA1.738.738.126.1

Adapter CAA performs substantially worse than first-order steering, suggesting that the first-order formulation is contributing nontrivially to behavioral adherence.

Additional Comments

The relation to linearized fine-tuning, such as task arithmetic in the tangent space[4], may be worth discussing.

[4]: Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models

The goal and formulations of our paper differ greatly from that of [4]. Task arithmetic in the tangent space uses a linearization of the model at the weights in order to encourage "weight disentanglement," which they demonstrate improves the performance of task arithmetic [4]. The primary similarity is the use of a linearization, but the purpose of this linearization and where it is taken differs greatly. There may be similarities between our work and [4] which may inform improvements to both methodologies, but it is outside the scope of this paper to study these similarities.

The remainder bound also holds for small ss, while evaluation is carried out at the corners of [0,1]n[0, 1]^n, where ∣s∣|s| reaches 3.

Our previous version of Proposition 2 used an arbitrary open neighborhood of 0∈Rn,0 \in \mathbb{R}^n, but we have modified this to apply to s∈Vs \in V for a chosen open set VV containing [0,1]n[0, 1]^n. The remainder bound (4) in our paper applies to any ss by Taylor's theorem, and is still controlled by the Hessian (with O(∥s∥2)O(\|s\|^2) error). We greatly appreciate the level of detail of this comment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary by Authors

Dear Area Chair and Reviewers,

Thank you for your thoughtful feedback on and discussion of our manuscript. This discussion has clearly improved it. As the author/reviewer discussion period comes to an end, we would like to concisely summarize the suggestions of reviewers and the subsequent modifications which we have made.

The primary reviewer concerns were with insufficient clarity in the first-order steering formulation used for HeRD-Merging, a lack of comparison with more matched steering methods, missing evidence for the practical contribution of Hessian regularization, and an unclear distinction between empirical and theoretical results.

We have since made several changes to address these concerns and briefly detail them below:

1. Clarification of First-Order Steering Methodology

  • We have added a clearer explanation in Section 4.2 and Appendix A, describing how the generalized first-order steering rule is obtained, and how it enables steering vectors produced with task arithmetic to differ from those produced with HeRD-Merging.
  • We have updated Section 4.4 to reflect the Hessian regularization terms used for our HeRD-Merging results.
  • We have added Proposition 4 to our paper, proving the Hessian regularization bound for the least-squares error in the general case of first-order steering. This is followed by an explicit discussion on how it relates to the original Hessian regularization claims (Proposition 2).

2. Clarified Scope of Theoretical Claims for Hessian Regularization

  • We have explicitly explained in Section A the limitations of using the residual r2r_2 as a proxy for the first-order error bound. In particular, we note that r2r_2 does not directly optimize the approximation error along the entire weight update, and explained the practical reasons for constructing r2r_2 as we did.
  • We explicitly noted in Section A the theoretical relevance of r2r_2 and r3r_3, stressing that we make no claims about guaranteed improvement due to optimization of these residuals, even though we have shown that the procedure is effective empirically.

3. Additional Baselines for Steering Comparison

  • We compared our model against additional steering baselines (BiPO, LoReFT, LoFiT), with results in Section 5.3 and C.1. We now show that, on average, first-order steering with HeRD-Merging also outperforms these methods in our experiments.
  • We further added a reference in Section C.1, which takes the average activation difference between our trained LoRA adapters and the original transformer. The performance of first-order steering is consistently above this baseline.
  • We updated our results for all methods with joint and exact adherence rates, more directly showing that first-order steering with HeRD-Merging outperforms other methods on average for composing behaviors.

4. Extended Ablations and Justification for Steering Results

  • We added three ablations (without r2r_2, without r3r_3, without r2r_2 and r3r_3) in Section C.2 that evaluate the contribution of Hessian regularization to steering performance. These results demonstrate that first-order steering with HeRD-Merging performs substantially worse without Hessian regularization.
  • We also added a comparison of the approximation error and Hessian norm before/after the Levenberg-Marquardt algorithm in Section C.2, demonstrating empirically that our optimization procedure meaningfully reduces the relevant first-order steering error terms.
  • We provided possible interpretations of these results, which support the idea that Hessian regularization can be a signal of steering accuracy.

We believe that these revisions address the reviewer concerns. We once again thank the reviewers and the area chair for this opportunity for discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.