acceptodds
Under review as a conference paper at ICLR 2027

Orthogonal and Parametrized Multi-Attribute Steering of LLMs

Abstract

Activation steering offers an effective and efficient way to modify an LLM's behavior at inference time. However, it remains unclear how to apply steering such that the signal is consistently strong and the steering strength has a universally interpretable meaning. We address this first by identifying the most steerable interval for each steering vector and rescaling the steering range accordingly. Building on this, we introduce a more interpretable method for steering, called parameterized steering which instead of pushing activations in a direction, it sets them at the desired trait strength. Next, with multi-trait steering, we find that naively combining vectors causes them to interfere with one another. In response, we develop a method to orthogonalize the vectors; it decouples these traits, allowing a model to exhibit one behavior without inducing the others, thus enabling multiple traits to be steered simultaneously.

Primary area:
interpretability and explainable AI
Keywords:
activation steering, multi-attribute steering, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9SVZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper improves activation steering through interval calibration, setting activations to target coordinates, and jointly controlling correlated directions in a transformed coordinate system. Experiments examine single- and multi-attribute control, including warmth and sycophancy, on Llama and Qwen models.

Critical strengths
  1. Setting target coordinates gives steering a useful operational meaning: activations can move different distances toward the same target. Joint control removes dependence on intervention order, with a simple construction and valid reconstruction guarantees.
  2. The experiments show a concrete benefit: average sensitivity for highly correlated traits falls from 1.508 to 0.827 on Llama. Open-ended generation and Qwen tests add useful evidence, including cases where the benefit is limited.
Critical weaknesses
  1. Order-independent coordinate updates do not guarantee independent behaviors: changing coordinate j still moves along the original direction u_j. The abstract should match the more careful limitations. Mixed Qwen results and Table 4 also show that orthogonalization's empirical advantage is conditional.
  2. The disclosed setup mismatch prevents a direct ranking against MAT-Steer. A clearer distinction from the cited K-Steering would help establish the method's place. These issues limit comparative claims, but do not invalidate the controlled additive/parameterized/orthogonalized comparisons.
  3. Appendix F's judge ignores correctness, completeness, and truncation. The open-ended results support trait control, but leave output quality unclear. Appendix C.2 should also avoid implying that changing a small subspace necessarily has negligible collateral effects.
  4. Endpoint coverage and slope do not guarantee a monotonic or linear interval. Clipping is therefore an empirical calibration heuristic. Likewise, zero denotes a midpoint between contrastive means, not necessarily neutral behavior; claims about universally interpretable strength need qualification.
  5. Minor corrections: Replace the Llama 2 citation “(?)” on page 3 and remove the doubled subject in “which ... it sets them” in the abstract. Theorem 1 should specify that invertibility requires retaining the residual, as the appendix does.
Questions for authors
  1. Can you state precisely which independence properties are guaranteed and which behavioral improvements are only observed empirically?
  2. What explains the differing benefit of orthogonalization across Llama, Qwen, and Table 4, and when should a user prefer it over parameterization alone?
  3. For the existing open-ended outputs, how does correctness or completeness change at comparable levels of target-trait control?
  4. How often do selected intervals violate monotonicity, and what interpretation should users give to zero and intermediate parameter values?
3 comments

Thank you for the feedback and questions. Sorry for delay, we had challenges accessing our shared GPUs.

Q1: Independence Guarantees

Can you state precisely which independence properties are guaranteed and which behavioral improvements are only observed empirically?

Independence guarantees. For linearly independent steering directions, our construction guarantees that, at the intervention layer, assigning one selected transformed coordinate leaves the other selected coordinates unchanged, and assignments to distinct coordinates commute. The procedure is idempotent. Reconstruction also preserves the activation’s component outside the span of the steering directions, provided that the residual is retained.

These are guarantees about activation coordinates, not independent behaviors. The linear representation hypothesis motivates associating the learned directions with concepts, but it does not guarantee that those directions capture the concepts perfectly or that later layers preserve behavioral separation. Whether changing one coordinate affects other measured behaviors is therefore an empirical question.

Our experiments show reduced behavioral interference in some settings, including the power–coordination example in Figure 5, but the results vary across traits and models. We will revise the abstract and Section 5 to distinguish these empirical findings from the coordinate-level guarantees.

Q2: Benefits of orthogonalization (across models)?

What explains the differing benefit of orthogonalization across Llama, Qwen, and Table 4, and when should a user prefer it over parameterization alone?

Parameterized steering sets an activation to a target coordinate rather than adding a fixed displacement. In the joint construction, assignments to distinct transformed coordinates commute and leave the other selected coordinates unchanged. By contrast, additive α\alpha-iterative steering does not gain this benefit from orthogonalization: adding a transformed steering vector and mapping back produces the same update as adding the original steering vector directly, so interference between the directions remains. These coordinate guarantees do not ensure behavioral independence, and the empirical benefit varies across settings.

As shown in Table 2 and illustrated in Figure 5 (right, top), just applying parameterized on two traits has the effect that the second steered trait partially overwrites the effect of steering the first trait.

Comparison with K-Steering. K-Steering also addresses multi-attribute control, using a trained nonlinear multi-label classifier and its gradients to construct interventions. Our approach instead starts from individual steering directions and sets targets in a joint coordinate system. This yields an explicit order-independence guarantee for the selected coordinates, while K-Steering can model nonlinear interactions through its classifier. We have not found an easy way to perform a direct empirical comparison, so we do not claim that either approach is generally more effective.

In Table 4, parameterized steering scores higher than orthogonal parameterized steering on all three benchmarks on the less stable Percent Steered measure. The comparison with MAT-Steer uses unmatched data splits and different baselines, so it does not support a direct ranking against MAT-Steer. Some of our no-steering scores start very high, leaving limited room for improvement on those datasets.

Q3: Refined LLMAJ evals

For the existing open-ended outputs, how does correctness or completeness change at comparable levels of target-trait control?

Open-ended evaluation at intermediate steering levels. To examine trait control at intermediate settings of the second trait, we extended the LLM-as-a-judge evaluation to β=−0.5\beta=-0.5 and β=+0.5\beta=+0.5. For each fixed β\beta, we measure the change in the fraction of responses judged to exhibit the α\alpha trait as α\alpha varies between −1-1 and +1+1.

The results at β=±0.5\beta=\pm0.5 are mixed: across the five trait pairs shown, the average ranges for parameterized and orthogonal parameterized steering are 2.0 and 1.6 percentage points at β=−0.5\beta=-0.5, and 1.5 and 1.2 percentage points at β=+0.5\beta=+0.5, respectively. These results concern the effectiveness of reaching the target behavior at comparable target-trait control.

Alpha-trait percent-steered range at β=±0.5\beta=\pm0.5

Assuming ΔAI=OP−AI\Delta_{\mathrm{AI}} = \mathrm{OP} - \mathrm{AI}, here is the corrected table with it included for both beta settings:

α Traitβ TraitCorr.β = −0.5: AIβ = −0.5: Pβ = −0.5: OPβ = −0.5: ΔAIβ = −0.5: ΔPβ = +0.5: AIβ = +0.5: Pβ = +0.5: OPβ = +0.5: ΔAIβ = +0.5: ΔP
coordinationpower0.8701.0%0.5%0.5%−0.5%0.0%1.0%1.0%1.0%0.0%0.0%
powercoordination0.8704.0%2.5%1.5%−2.5%−1.0%1.0%1.0%2.0%1.0%1.0%
corrigibilitycoordination0.8101.5%2.5%1.5%0.0%−1.0%2.5%3.5%1.5%−1.0%−2.0%
coordinationcorrigibility0.8100.5%0.5%3.5%3.0%3.0%0.5%0.0%0.5%0.0%0.5%
powerwealth0.7304.5%4.0%1.0%−3.5%−3.0%1.0%2.0%1.0%0.0%−1.0%
Average2.3%2.0%1.6%−0.7%−0.4%1.2%1.5%1.2%0.0%−0.3%

Our existing multiple-choice evaluations provide complementary evidence on correctness. In Table 4, orthogonal parameterized steering improves the fraction of preferred labeled answers from 63.0% to 78.5% on TruthfulQA and from 59.5% to 69.5% on BBQ relative to the unsteered baseline, although parameterized steering scores higher on both. These evaluations do not measure correctness or completeness of the open-ended responses. We will make that limitation explicit and aim to extend the LLM-as-a-judge evaluation to assess correctness and completeness in the final version.

Q4: Do intervals violate monotonicity

How often do selected intervals violate monotonicity, and what interpretation should users give to zero and intermediate parameter values?

Monotonicity and interpretation of steering strength. As discussed in our response to PUzq, we seek a generally increasing steering curve but do not enforce strict monotonicity. The interval is selected on training data, while Figure 2 shows the resulting curves on held-out data; they are increasing over the selected range. Thus, monotonicity is an empirical goal rather than a guarantee.

After rescaling, −1-1 and +1+1 represent the endpoints of the selected steering interval, and 00 is their midpoint. Zero therefore provides a reference within the calibrated interval, but it does not necessarily represent neutral behavior or the unsteered model. For some behaviors, neutrality may not be well-defined; nor would a neutral setting necessarily correspond to 50% behavior-matching answers, since that measure depends on the evaluation data distribution. Intermediate values interpolate between the endpoints, but do not guarantee a proportional change in measured behavior.

We will also correct the missing Llama 2 citation and the doubled subject in the abstract.

Thanks for your response. I have read it and kept my positive score.

Reviewer_TzDeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a way to control several LLM behaviors at one through activation steering. It motivates with making a model warmer without making it more sycophantic. The method has three parts. It first calibrate the steering range, then set a target activation coordinate, and finally separate steering coordinates. Experiments show that improvements over additive steering in several settings, although the benefits of orthogonalization are mixed.

Critical strengths

The paper addresses a useful problem for the interpretability literature: how to change one model behavior without unintentionally changing another. The warmth and sycophancy example makes the problem easy to understand.

The method is simple, and setting activation coordinates removes the dependence on the order of steering updates. The experiments across several traits and model families help show where it works and where it struggles.

Critical weaknesses
  1. I am not convinced that separating the steering coordinates is enough to separate the behaviors. The theorem shows that setting one coordinate does not change the other coordinates, and that the order of the updates does not matter. But after the activation passes through the remaining model layers, changing one coordinate could still affect several other behaviors.

  2. It is unclear to me how much orthogonalization adds to parameterized steering. In Table 4, parameterized steering performs better on all three benchmarks, and Qwen are mixed. The generation judge also ignores things like correctness, so the trait scores do not reveal whether the responses remain useful. I think it would be helpful to see both target behaviors evaluated on the same responses and answer quality.

Questions for authors
  1. What evidence shows that separating the activation coordinates also separates the behaviors in generated responses?
  2. When does orthogonalization improve on parameterized steering? Can you evaluate both target behaviors on the same responses while also checking correctness and usefulness?
5 comments

Thank you for the feedback and questions. Sorry for delay, we had challenges accessing our shared GPUs.

Q1: Why focusing on intermediate layer, coordinate independence is appropriate.

I am not convinced that separating the steering coordinates is enough to separate the behaviors. The theorem shows that setting one coordinate does not change the other coordinates, and that the order of the updates does not matter. But after the activation passes through the remaining model layers, changing one coordinate could still affect several other behaviors.

What evidence shows that separating the activation coordinates also separates the behaviors in generated responses?

Coordinate independence and behavioral independence. We agree that separating coordinates at an intermediate layer does not guarantee that the corresponding behaviors remain independent in the model’s outputs. Our theoretical result applies at the intervention layer: assigning one selected coordinate leaves the other selected coordinates unchanged, and assignments to distinct coordinates commute. Subsequent model layers can still couple the effects of these changes.

Our motivation for studying these coordinates comes from the linear representation hypothesis: the idea that some concepts are represented by directions in an activation space. This motivates looking for structure within an individual layer and studying interventions along directions learned from behavioral contrasts. Intermediate layers are a natural setting for such interventions because their representations can contain information relevant to a behavior while still influencing the model’s subsequent computation and output. This is supported by Anthropic's residual stream view of transformer activation layers, and the prior work on activation steering.
The hypothesis motivates the construction, but it does not by itself establish that the learned directions perfectly represent the intended behaviors or that their downstream effects are independent.

We therefore view behavioral separation as an empirical question. For example, Figure 5 examines how steering power affects coordination under different methods; it shows reduced interference in that setting, rather than complete behavioral independence. Our open-ended evaluations provide additional evidence that steering changes the measured traits, but they do not establish independent control of both traits on the same responses. We will revise the abstract and Section 5 to make this distinction between the coordinate-level guarantee and the empirical behavioral results explicit.

Q2: When does orthogonality improve on parameterized steering?

It is unclear to me how much orthogonalization adds to parameterized steering. In Table 4, parameterized steering performs better on all three benchmarks, and Qwen are mixed. The generation judge also ignores things like correctness, so the trait scores do not reveal whether the responses remain useful. I think it would be helpful to see both target behaviors evaluated on the same responses and answer quality.

When does orthogonalization improve on parameterized steering? Can you evaluate both target behaviors on the same responses while also checking correctness and usefulness?

When does orthogonalization help? Its geometric benefit is clearest when steering directions are correlated and we want to set multiple target coordinates: assignments to distinct coordinates are idempotent and order-independent. Without orthogonalization, setting a second coordinate can partially undo the first assignment (for parameterized steering) or combine with the first assignment (for α\alpha-iterative steering). Figure 5 illustrates these dependence challenges for power and coordination.

The behavioral benefit is conditional. In Table 2, for the highly correlated pairs, the average sensitivity is 0.975 for parameterized steering when the measured trait is set first and 0.827 for orthogonal parameterized steering. When the measured trait is set second, parameterized steering has lower sensitivity (0.355), but this ordering cannot put both traits last. The behavioral benefit varies across traits and models. Orthogonal parameterized steering is therefore useful for setting both selected coordinates to their intended targets, without one assignment partially undoing the other. Whether both corresponding behaviors are steered as intended remains an empirical question.

Correctness and usefulness. Our existing multiple-choice experiments provide some evidence on correctness. In Table 4, while jointly steering the three benchmark attributes, orthogonal parameterized steering improves the fraction of preferred labeled answers from 63.0% to 78.5% on TruthfulQA and from 59.5% to 69.5% on BBQ, relative to the unsteered baseline. Parameterized steering scores higher still on both benchmarks. These results suggest that joint steering need not reduce performance on these correctness-related evaluations.

We agree, however, that these answer-selection results do not establish the correctness, completeness, or usefulness of the open-ended responses. The current judge measures the target trait and does not assess those qualities. Nor do our reported open-ended results evaluate both target traits on the same responses. We will clarify these limitations, work towards these more expensive LLM evaluations in the final version, and make sure our claims are properly scoped.

Thank you for the response.

Regarding Q1: I am not questioning why studying coordinates at an intermediate layer (which is a sensible choice). My question is why separating the coordinates separate behaviors in the generated responses.

The response clarifies that the theorem supports coordinate independence, and behavioral independence is an empirical question. However, my reading is that the usefulness of the coordinate interdependence depends on whether it improves control of the behavior. Could you please clarify how Figure 5 helps with demonstrating the usefulness, how does establishing coordinate interdependence help? If it's hard to demonstrate this via Figure 5, any other practical benefits can coordinate benefit support?

Regarding Q2 : I now understand the proposed advantage. The clarification is helpful. But if I understand it correctly, the results do not show orthogonalization producing better behavioral control than standard steering?

Reviewer_PUzqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies interpretable activation steering and interference between jointly steered attributes. It proposes three components: identifying and rescaling an empirically effective steering interval; parameterized steering that sets an activation’s projection to a target value rather than adding a fixed displacement; and a change of coordinates that allows correlated steering directions to be controlled through separate coordinate assignments. Validation primarily uses Llama-2-7B-Chat, with multiple-choice behavioral evaluations and additional open-ended evaluations using an LLM judge. The paper also presents selected experiments on Qwen3-8B and a comparison with MAT-Steer using Llama-3.1-8B. Results generally favor the proposed methods on Llama-2, while the Qwen results are less consistent.

Critical strengths

The research questions are relevant and clearly motivated. Interpreting steering strength and controlling multiple correlated attributes are practical challenges for activation steering.

Critical weaknesses
  1. Novelty relative to existing interpretable steering methods is insufficiently established.

Angular Steering [1] and Spherical Steering [2] already motivate geometric alternatives to additive steering, while Activation Transport [3] explicitly introduces an interpretable interpolation parameter linking the intervention strength to its destination. These approaches differ from the submission, but substantially overlap with its motivation and claimed contribution concerning interpretable steering strength. Section 4 is itself an affine intervention that replaces an activation’s projection with a target coordinate. The paper does not adequately position this formulation against existing approaches or establish what additional capability or insight it provides.

  1. The steerable-interval procedure is underspecified and does not explicitly enforce its stated criteria.

Selecting A,BA,B requires first evaluating a steering curve over some domain, but the initial bounds, sampling resolution, endpoint-search algorithm, and associated evaluation cost are unclear. Moreover, the proposed loss measures endpoint coverage and a secant slope; it does not directly penalize internal nonmonotonicity or deviation from a linear fit. The method therefore appears to be an empirical calibration heuristic, without sufficient justification that it identifies an interval satisfying the advertised properties. This weakens both its reproducibility and its contribution as a principled calibration method.

  1. Orthogonal coordinates do not establish independent behavioral control.

The transformation separates coordinate assignments, but reconstruction returns interventions along the original, correlated steering directions. An equivalent intervention can also be applied directly in the original space, so any performance improvement must be attributed to the intervention rule rather than the intermediate coordinate transformation itself. The paper needs a clearer justification connecting this rule to reduced behavioral interference.

The method also assumes that no steering direction can be expressed as a linear combination of the others. If this assumption fails, the proposed inverse does not exist, and the traits cannot all be assigned independent coordinates. The paper does not explain how such cases are detected or handled.

  1. The experimental coverage is limited and uneven, and the stronger findings do not transfer consistently across models.

The main sensitivity and steering-range evaluations are concentrated on Llama-2-7B-Chat. The Qwen3-8B results show substantially weaker improvements. Llama-3.1-8B appears only in the separate MAT-Steer comparison, rather than a replication of the core protocol; that comparison also uses unmatched splits and substantially different baselines. The evaluated models are older generations, and even within this limited scope, the core findings are not consistently replicated. This weakens the broader claims of robustness and generalizability.

References

[1] H. M. Vu and T. M. Nguyen. Angular Steering: Behavior Control via Rotation in Activation Space. NeurIPS, 2025.

[2] Z. You, C. Deng, and H. Chen. Spherical Steering: Geometry-Aware Activation Rotation for Language Models. ICML, 2026.

[3] P. Rodríguez et al. Controlling Language and Diffusion Models by Transporting Activations. ICLR, 2025.

Questions for authors
  1. What is the precise methodological distinction between Section 4 and existing geometric or transport-based parameterizations [1–3] ? Which aspect of the proposed parameterization constitutes the principal new contribution ? How does the proposed method compare with those under matched experimental settings? Such comparisons would help establish its contribution beyond existing interpretable parameterizations.

  2. How are the candidate domain and resolution for the steering curve chosen, and how are A,BA,B selected algorithmically ? What ensures that the selected interval is approximately linear and monotonic, given that the loss only evaluates its endpoints ?

  3. Beyond order-independent coordinate assignment, what justifies interpreting the transformed coordinates as independently controllable behaviors ? How should the observed gains be attributed between joint coordinate setting and the orthogonal-space interpretation ? Please also clarify the method’s applicability when trait directions are linearly dependent or nearly dependent.

  4. Can the authors evaluate the core sensitivity and steering-range experiments on a wider range of recent models ? The current evaluation concentrates on a small set of older model generations, applies different experimental protocols across models, and shows weaker and mixed results on Qwen3.

7 comments

Thank you for the feedback and questions. Sorry for delay, we had challenges accessing our shared GPUs.

Q1: Compare to angular steering, spherical steering, and Linear Steering

What is the precise methodological distinction between Section 4 and existing geometric or transport-based parameterizations [1–3] ? Which aspect of the proposed parameterization constitutes the principal new contribution ? How does the proposed method compare with those under matched experimental settings? Such comparisons would help establish its contribution beyond existing interpretable parameterizations.

Relationship to geometric and transport-based steering. We agree that these methods share our motivation of exploring the space of activation augmentation functions gg of an activation beyond the additive α\alpha-iterative g(a)=a+αug(a) = a + \alpha u.

  • Angular Steering [1] g(a)=Rag(a) = R a controls behavior by rotating activations toward or away from a target direction (and RR only acts on a low rank subspace).
  • Spherical Steering [2] also uses rotation g(a)=Rag(a) = R a but acts on full subspace, and has an optional confidence-based adjustment.
  • Activation Transport [3] instead learns a full affine map g(a)=Ra+αug(a) = Ra+\alpha u, and uses an interpolation parameter to move an activation toward its transported destination.

Our parameterized intervention is a structurally different approach: it sets an activation’s coordinate to a target associated with a calibrated steering strength.

Moreover, we show parameterized steering pairs well with our orthogonalization mechanism. The multiple coordinates set in the interpretable parameterized space correspond to the projection (adjusted by calibration) onto the 1-dimensional subspaces defined by each concept U=[u1,…,uk]U=[u_1,\ldots, u_k]. This avoids order-dependence artifacts of performing such projection naively in the original space; orthogonal parameterized multi-steering is order-independent.

Moreover, for additive α\alpha-iterative steering, our orthogonal mechanism (a mapping so each fU(uj)=ejf_U(u_j)=e_j) does not separate the effect on each concept intervention. Adding δej\delta e_j in the transformed coordinates and reconstructing with the original residual gives the same activation as adding δuj\delta u_j in the original space. Thus, consistently transforming an additive intervention into the new coordinates and mapping it back does not by itself remove behavioral interference. Additive updates also commute; the issue for correlated directions is their combined effect on the activation and, potentially, on behavior.

Our joint parameterized intervention instead sets selected transformed coordinates to target values. Assignments to distinct coordinates commute and do not change the other selected coordinates at the intervention layer.

Extending a matched experimental setting. We compared α\alpha-iterative, parameterized, orthogonal parameterized, spherical, Activation Transport, and angular steering on six individual traits using percent steered. These experiments were rerun on Llama 2 7B Chat using the same general experimental settings as our earlier results. At the tested setting of α=1\alpha=1, parameterized and orthogonal parameterized steering achieve average scores of 74.5% and 74.4% (same up to small precision issues), respectively, compared with 76.9% for α\alpha-iterative, 75.7% for Activation Transport, 68.8% for angular, and 58.3% for spherical steering. Results vary by trait; for example, angular steering performs best on myopia and power.

Percent steered with α=1.0\alpha = 1.0 across steering methods

behavioralpha-iterativeparameterizedorthogonal-parameterizedsphericalActivation Transportangular
coordination83.5%86.5%86.0%64.5%77.0%78.5%
corrigibility98.5%97.5%97.5%96.0%96.0%97.0%
myopia67.5%65.0%65.0%69.0%68.0%77.0%
power77.0%79.5%79.0%66.0%77.5%82.0%
survival72.0%64.5%64.5%22.5%72.0%40.0%
wealth63.0%54.0%54.0%31.5%63.5%37.0%
average76.9%74.5%74.4%58.3%75.7%68.8%

Q2: Algorithm for choosing (A,B) in Stable Steering Region

How are the candidate domain and resolution for the steering curve chosen, and how are (A,B) selected algorithmically ? What ensures that the selected interval is approximately linear and monotonic, given that the loss only evaluates its endpoints ?

Selecting the steerable interval. For each trait, we evaluate the steering curve on the training data over [−2,2][-2,2] in increments of 0.10.1. We then perform a grid search over candidate intervals [A,B][A,B] containing zero and select the interval minimizing the loss in Section 3. The loss balances coverage of the observed minimum and maximum against a steep positive slope; in our experiments, we use γ=0.1\gamma=0.1 (which we selected among plotted options on training data in Appendix A). We rescale the selected interval to [−1,1][-1,1] for steering.

We desire a generally increasing response over the selected interval, but do not require strict monotonicity or linearity: the endpoint-based loss does not guarantee either property within the interval. Its purpose is to find a useful calibrated range while avoiding long, relatively flat regions of the steering curve.

Ultimately, the goal is not to enforce a particular curve on the training set, but to obtain a useful steering range on held-out data. Figure 2 shows the held-out curves after selecting the intervals on training data. We will clarify the search procedure and the empirical nature of the calibration in Section 3.

Q3: Independence on Trait Coordinates

Beyond order-independent coordinate assignment, what justifies interpreting the transformed coordinates as independently controllable behaviors ? How should the observed gains be attributed between joint coordinate setting and the orthogonal-space interpretation ? Please also clarify the method’s applicability when trait directions are linearly dependent or nearly dependent.

Coordinate interpretation and behavioral control. The linear representation hypothesis motivates associating learned directions with concepts in an activation space. Our construction maps the selected directions to orthogonal directions and moreover as separate coordinates, allowing us to set each coordinate without changing the others. This gives an interpretable way to specify a joint intervention, but it does not guarantee independent downstream behaviors; that remains an empirical question.

Role of the transformation. We agree that the same joint target-setting intervention can be expressed directly in the original space. The behavioral results should therefore be attributed to the intervention rule itself, not to the coordinate transformation as a separate operation. We find the transformed-space view useful because it makes the target assignments and their order independence explicit.

Dependent directions. Our construction (and Theorems 1 & 3) assumes that the selected steering directions are linearly independent. If they are exactly dependent, they cannot all be assigned independent coordinates by this construction. In such cases, one could remove or combine redundant directions.
If they are nearly dependent, the coordinate calculation could become sensitive to small changes in the directions or activations. Although this is still effective for coordination and power as illustrated in Figure 4.

More broadly, dependence between learned directions may reflect relationships between the underlying behaviors or artifacts of the datasets used to learn them. For example, as raised in the paper's discussion, warmth and sycophancy may be difficult to separate if the training examples contain elements of both. We believe our methodology helps elucidate such challenges.

Q4: Steering on newer models

Can the authors evaluate the core sensitivity and steering-range experiments on a wider range of recent models ? The current evaluation concentrates on a small set of older model generations, applies different experimental protocols across models, and shows weaker and mixed results on Qwen3.

Additional model: Gemma-4-12B-it. We ran the core multi-trait sensitivity and range evaluations on Gemma-4-12B-it at layer 26. The results are mixed and provide a further example of how the benefit of orthogonalization varies across models and traits.

For sensitivity, lower is better. Averaged across the six directed trait pairs, parameterized steering has the lowest sensitivity at 8.1%, followed by orthogonal parameterized at 10.9% and α\alpha-iterative at 12.1%.

Gemma-4-12B-it: Alpha-trait percent-steered sensitivity

α Traitβ TraitCorr.AIOPΔAI (AI − OP)PΔP (OP − P)
coordinationcorrigibility0.81112.8%12.8%0.0%14.7%−1.9%
coordinationpower0.87612.4%14.6%−2.2%14.3%0.3%
corrigibilitycoordination0.81110.8%6.7%4.1%3.7%3.0%
powercoordination0.87616.4%12.3%4.1%1.2%11.1%
powerwealth0.72912.3%9.4%2.9%10.9%−1.5%
wealthpower0.7298.0%9.8%−1.8%3.7%6.1%
Average0.80512.1%10.9%1.2%8.1%2.9%

For range, at β=+1\beta=+1, orthogonal parameterized steering improves on parameterized steering on average (35.7% versus 21.5%), although α\alpha-iterative has the largest average range at 45.3%. At β=−1\beta=-1, the averages are 23.0% for orthogonal parameterized, 21.3% for parameterized, and 44.1% for α\alpha-iterative. The size and direction of the differences also vary across trait pairs.

Gemma-4-12B-it: Alpha-trait percent-steered range

α Traitβ TraitCorr.β = −1: AIβ = −1: Pβ = −1: OPβ = −1: ΔAIβ = −1: ΔPβ = +1: AIβ = +1: Pβ = +1: OPβ = +1: ΔAIβ = +1: ΔP
coordinationpower0.87638.0%8.0%29.5%−8.5%21.5%54.5%9.0%42.5%−12.0%33.5%
powercoordination0.87662.0%48.5%30.5%−31.5%−18.0%51.0%49.5%51.0%0.0%1.5%
coordinationcorrigibility0.81157.5%18.0%26.5%−31.0%8.5%31.0%12.0%28.5%−2.5%16.5%
corrigibilitycoordination0.81129.5%17.5%10.5%−19.0%−7.0%51.0%28.0%38.0%−13.0%10.0%
powerwealth0.72950.0%33.0%40.5%−9.5%7.5%62.5%30.0%53.0%−9.5%23.0%
wealthpower0.72927.5%3.0%0.5%−27.0%−2.5%22.0%0.5%1.0%−21.0%0.5%
Average0.80544.1%21.3%23.0%−21.1%1.7%45.3%21.5%35.7%−9.7%14.2%

One possible explanation is that the numerical effects of oversteering are less pronounced on Gemma than on Llama, reducing the advantage of orthogonal parameterized steering under this metric. We have not yet been able to test this hypothesis. Evaluating open-ended generations would help determine whether the same patterns appear in model responses, and is something we will work to add in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.