acceptodds
Under review as a conference paper at ICLR 2027

Can Context Arbitrarily Reorient Linear Representations in Large Language Models?

Abstract

The so-called concept vectors in the activation space of language models are widely used to interpret and steer high-level behavioral traits represented in model outputs. These directions are typically extracted from a given pair of contrastive sample sets and treated as fixed for the model at hand thereafter. Complementary to the emerging literature on the effects of input context on linear representations, we systematically analyze how strongly context alone can change these directions while holding the model, layer, and the sample sets fixed. We find that optimizing a few shared input prefix tokens can reorient the extracted concept vector to a wide range of arbitrarily chosen directions, including the direction opposite to the original one. Further analysis reveals that reorientation of the extracted contrast vector is achieved by prefix-induced displacement in the underlying activation geometry. Preliminary observations suggest that a related geometric signature may also accompany successful adversarial prompts such as Greedy Coordinate Descent (GCG). Incorporating this geometric objective into GCG indeed yields modest gains in attack success on some models. Furthermore, we take a step toward reducing multi-concept interference under activation steering by optimizing the prefix to make the extracted contrastive vectors for different concepts orthogonal. Overall, our findings suggest that reliable interpretability should treat linear representations as context-conditioned objects rather than as specific only to the model.

Primary area:
interpretability and explainable AI
Keywords:
concept representations, language modeling, steering, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UnuVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies how input context can shape contrastive concept vectors in frozen language models. It proposes CoRA, a prefix-tuning framework that optimizes a shared soft prefix applied to source examples to align their steering vector with a prescribed direction while keeping reference examples unchanged. The paper provides necessary and sufficient local conditions for alignment and evaluates behavioral applications across diverse benchmarks, demonstrating the feasibility of the proposed method.

Critical strengths

S1. The paper addresses a clear and relevant question about how context affects concept vectors. Targeted reorientation provides a useful perspective beyond simply observing variation across contexts.

S2. The method is simple. By fixing the model, layer, original examples, and extraction rule while optimizing a shared prefix, the framework makes the intervention and its geometric effects easy to follow. The experimental setup is clean and well controlled, and the results align well with the observations.

S3. The empirical evaluation covers several model families and examines layer choice, prefix length, and alignment with random target directions, providing useful evidence about the method’s capabilities.

Critical weaknesses

W1. The learned soft prefixes lack a clear semantic interpretation. Geometric reorientation alone does not explain what semantic change the prefix induces. More behavioral analyses or comparisons with interpretable natural-language contexts would help connect the geometric findings to meaningful contextual variation.

W2. While the paper evaluates held-out prompts, cross-dataset transfer of the prefix-induced steering vector remains unclear. Testing the fixed vector on another dataset targeting the same concept would strengthen the evidence for the generality of its behavioral effects.

W3. The relationship to representation-finetuning methods such as ReFT [1] deserves more discussion. Both approaches use lightweight trainable interventions on a frozen model to influence representations and subsequent behavior. I understand that the training modules may differ, but given their similar effects, I think further comparisons with results from this line of work should be included.

[1] ReFT: Representation Finetuning for Language Models, NeurIPS 2024.

Questions for authors

Q1. Longer prefixes can improve geometric alignment while reducing ASR (Figure 6). If a single steering vector (e.g., 1×40961\times4096) can achieve the desired effect, could the additional capacity from learning multiple prefix embeddings increase the overfitting risk? I think a more rigorous study of the expressiveness of the learned prefix could help clarify the scope of the paper.

6 comments
AuthorsOpenReview ↗

Rebuttal (1/2) Weaknesses 1 and 2

W1. The learned soft prefixes lack a clear semantic interpretation. Geometric reorientation alone does not explain what semantic change the prefix induces. More behavioral analyses or comparisons with interpretable natural-language contexts would help connect the geometric findings to meaningful contextual variation.

Reply: CoRA aims to understand how strongly extraction context can change contrastive representations. Indeed, geometric reorientation alone does not establish what a soft prefix means in natural language. Our experiments address complementary aspects and implications of geometric controllability:

  • Geometric control (Appendix D.2): alignment with random prescribed directions quantifies attainable reorientation, without assigning semantic meaning to those targets.
  • Behavioral control (Section 5.1): geometric optimization produces steering through the input prefix alone, without injecting a steering vector. This connects geometric objectives to downstream behavior, rather than establishing semantic preservation. We also provide generated output samples in Appendix D.6 of the revised manuscript to show that how the model answers the original prompt, successfully bypassing refusal.
  • Concept-specific utility (Section 5.2): the same prefix conditions both contrastive classes, and CoRA encourages orthogonality between two concept vectors. The resulting vectors retain substantial steering effects on their respective concepts while improving selectivity across all three models, providing behavioral evidence that their concept-specific meaning remains useful. A hinge-loss regularizer constrains changes in unsteered behavior (Appendix C.3, Eq. 44) promoting semantic proximity of the reoriented steering vector to the original one supported by the empirical steering effectiveness.

W2. While the paper evaluates held-out prompts, cross-dataset transfer of the prefix-induced steering vector remains unclear. Testing the fixed vector on another dataset targeting the same concept would strengthen the evidence for the generality of its behavioral effects.

Reply: We appreciate the reviewer’s suggestion. We note that in Section 5.1.1, the object trained by CoRA, and hence the intervention, is the prefix itself, so we evaluate cross-dataset transfer of the prefix. Specifically, we evaluate the trained prefix on the 159 standard behaviours from the HarmBench test set [1] on all models, without any retraining. We report the results in Table 8, alongside the ASR with no intervention and with a random 4-token prefix as baselines for comparison. The table shows that the prefix achieves a high ASR on all models, ranging from 85.53% to 94.34%, well above both baselines. This provides evidence that the preifx is transferable across datasets, supporting the generality of the behavioural effect it induces.

Table 8: Attack Success Rates when prepending the CoRA trained prefix in Section 5.1.1 on 159 standard behaviours from HarmBench, on all models.

ModelNo InterventionRandom PrefixCoRA
Llama3.2-3B-Instruct8.1813.8493.71
Mistral-7B-Instruct-v0.361.0154.0991.20
Gemma-2-9b-it1.261.2694.34
Qwen3.5-9B1.260.6385.53
Qwen3.6-27B0.000.0088.05

[1] Mazeika et al., HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ICML 2024.

AuthorsOpenReview ↗

Rebuttal (2/2): Weakness 3 and Question 1

W3. The relationship to representation-finetuning methods such as ReFT [1] deserves more discussion. Both approaches use lightweight trainable interventions on a frozen model to influence representations and subsequent behavior. I understand that the training modules may differ, but given their similar effects, I think further comparisons with results from this line of work should be included.

Reply: ReFT is relevant as both approaches train lightweight interventions while freezing the base model. However, they differ in where they intervene and what they optimize. ReFT, particularly LoReFT, learns transformations that directly edit hidden states at selected layers and token positions, trained through supervised output losses to improve downstream task performance. CoRA instead optimizes input-prefix embeddings to control the orientation of a contrastive mean-difference vector, with changes to internal representations mediated by the frozen model’s existing computations.

This distinction is central to our scientific question: how strongly can extraction context reshape a representation returned by a fixed extraction rule? Successful direct hidden-state editing does not establish which orientations are reachable through context alone. Section 5.1.1 then tests whether this geometric objective produces behavioral steering without target-completion supervision, while Section 5.2 uses shared-context optimization to improve the selectivity of subsequently extracted steering vectors. ReFT therefore provides a relevant comparison for behavioral adaptation, while CoRA investigates contextual controllability and its consequences for representation extraction.

Q1. Longer prefixes can improve geometric alignment while reducing ASR (Figure 6). If a single steering vector (e.g., 1×40961 \times 4096) can achieve the desired effect, could the additional capacity from learning multiple prefix embeddings increase the overfitting risk? I think a more rigorous study of the expressiveness of the learned prefix could help clarify the scope of the paper.

Reply: It's an interesting question. While overfitting risk is not impossible, the number of prefix tokens is not directly comparable to the "signleness" steering vector in any trivial way. A single injected steering vector and a single prefix embedding have different expressive capabilities despite having the same dimensionality. ActAdd directly specifies a hidden-state displacement, whereas a prefix must induce that displacement indirectly through the frozen model, across many inputs. Our theory also characterizes this distinction by uniformly bounding prefix-induced displacement at the original-token readout and relating local reorientation to the rank and conditioning of the prefix-to-contrast Jacobian. By comparison, ActAdd's injected displacement can grow arbitrarily large if its steering coefficient is unrestricted.

Moreover, the statement about sufficiency of a single steering vector requires a further care. A single vector is not established to suffice for reliable steering in general. For example, Wollschläger et al. (2025) identify multiple independent refusal directions and multidimensional concept cones. These findings challenge general single-direction uniqueness, without implying that multiple directions are necessary in every setting.

Thank you for the response. My concerns regarding W3 and Q1 remain.

Regarding W3, CoRA’s “context” consists of trainable continuous embeddings, so it is prefix tuning with an objective defined at the representation level, rather than a study of how natural language semantics shape representations. I agree that its geometric question is distinct, but CoRA and ReFT both modify representations by optimizing a small number of parameters while keeping the base model frozen. Their behavioral performance therefore warrants comparison under comparable conditions to establish CoRA’s additional value in Section 5. Differences in intervention location and training objective alone do not resolve this concern.

Regarding Q1, I would prefer studies or experimental results demonstrating why CoRA is better. For example, is there a method to determine how many prefix tokens are needed for a given task (i.e., whether the prefixes are expressive enough to encode the required representation shifts)? Does CoRA's expressivity allow it to achieve comparable performance with fewer trainable parameters or less training data than steering methods? Merely discussing the limitations of steering methods without providing supporting examples or experimental results does not adequately address this concern.

Reviewer_fDWcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the extent to which input context can reorient linear concept representations in LLMs. The authors propose Contextual Representation Alignment (CoRA), which optimizes a shared continuous soft prefix to align a difference-in-means concept vector with a prescribed target direction while keeping the model and contrastive examples fixed. The paper provides theoretical bounds and conditions for such reorientation, and empirically demonstrates substantial alignment with opposite and randomly selected directions. The authors further evaluate CoRA on jailbreaking across five LLMs, achieving ASRs comparable to or higher than conventional ActAdd, incorporate its alignment objective into GCG to obtain modest ASR improvements, and apply it to multi-concept steering to reduce interference between concept vectors.

Critical strengths
  1. An interesting demonstration of geometric controllability: The paper shows that optimizing a shared soft prefix can substantially reorient difference-in-means representation vectors toward prescribed directions, including randomly sampled targets, while keeping the model, contrastive datasets, and extraction procedure fixed. This provides an interesting empirical observation about the sensitivity of extracted representation directions to continuous input perturbations.
  2. Reasonably thorough theoretical and empirical treatment: The paper provides theoretical conditions on reorientation, evaluates multiple model architectures and downstream applications, and includes additional analyses and implementation details in the supplementary materials. Overall, the study appears carefully conducted, even though some conclusions require further validation.
Critical weaknesses
  1. Limited novelty and unclear incremental contribution: Context-dependent changes in linear representations have already been documented [1, 2], while optimizing continuous inputs to manipulate internal activations is also well established [3]. CoRA introduces a particular geometric objective for reorienting contrastive mean-difference vectors, but the paper does not sufficiently explain why this constitutes a significant advance beyond existing representation manipulation methods. Moreover, much of the behavioral evaluation uses the concept vector as an optimization target for a soft prefix rather than investigating the properties or utility of the resulting context-conditioned concept vectors. If CoRA is intended as a steering method, its advantages over conventional or directly optimized steering vectors [4] and standard prompt optimization remain unclear. The GCG and multi-concept applications also overlap with existing representation-guided optimization [5] and orthogonality-based steering [6], without sufficiently establishing the value of CoRA's specific formulation.
  2. Insufficient justification of the central interpretation and behavioral contributions:
    • Ambiguous interpretation of concept reorientation: In the main experiments, the optimized prefix is applied only to the source class, while the reference class remains unchanged. Consequently, the observed reorientation can be explained by moving one class's mean activation relative to a fixed reference. While this establishes geometric controllability, it does not necessarily demonstrate that the representation of the same underlying concept can be arbitrarily reoriented under a shared context. More importantly, it remains unclear whether the optimized soft prefix preserves the original semantic interpretation of the contrastive samples. Without such validation, the reoriented difference-in-means vector may no longer represent the original concept, limiting the interpretability implications drawn from these results.
    • Limited attribution of downstream improvements: The jailbreaking experiments optimize a prefix toward essentially the same direction used by ActAdd, yet Figure 3 does not include the representations obtained after ActAdd, making it difficult to understand why CoRA achieves slightly higher ASRs or whether it provides better geometric alignment. The GCG extension primarily adds an activation-based loss to an existing optimization procedure, and its distinct contribution remains unclear. In the multi-concept experiments, the supplementary implementation appears to use the optimized prefix during both vector extraction and downstream evaluation, making it difficult to separate the effect of the extracted vectors from that of the prefix itself. Simple baselines, such as random-prefix extraction and direct vector orthogonalization, are also missing.

[1] Lampinen et al., Linear Representations in Language Models Can Change Dramatically over a Conversation. arXiv, 2026. https://arxiv.org/abs/2601.20834

[2] Adarsh et al., How Context Shapes Truth: Geometric Transformations of Statement-level Truth Representations in LLMs. ACL 2026. https://aclanthology.org/2026.acl-long.1695/

[3] Bailey et al., Obfuscated Activations Bypass LLM Latent-Space Defenses. ICLR 2026. https://openreview.net/forum?id=ktGmDGoWnB

[4] Cao et al., Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization. NeurIPS 2024. https://arxiv.org/abs/2406.00045

[5] Wollschläger et al., The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence. ICML 2025. https://proceedings.mlr.press/v267/wollschlager25a.html

[6] Nguyen et al., Multi-Attribute Steering of Language Models via Targeted Intervention. ACL 2025. https://aclanthology.org/2025.acl-long.1007/

Questions for authors
  1. Scientific contribution and novelty: Beyond demonstrating that optimized continuous inputs can manipulate internal activations, what is the key new insight provided by targeted reorientation of contrastive vectors? Is CoRA primarily intended as a diagnostic tool for representation stability or as a practical steering method? How should its contribution be distinguished from existing work on context-dependent representations, soft-prompt optimization, and representation-guided steering?
  2. Validity of the concept representation interpretation: The main experiments apply the optimized prefix only to the source class. How much reorientation remains achievable when the same prefix is applied to both contrastive classes? More fundamentally, how can we establish that the resulting vectors continue to represent the original concepts rather than differences induced by changed input semantics or distributions? Evidence using shared-context interventions and independent semantic or behavioral validation would materially strengthen the central claim.
  3. Necessity of the CoRA objective and interpretation of jailbreak results: Would directly optimizing an equally sized soft prefix for the desired model behavior, or using an optimized natural-language prefix, produce comparable effects? Since CoRA and ActAdd target essentially the same representation direction in the jailbreaking experiment, could the authors compare their resulting activations directly, including ActAdd in Figure 3 and reporting full-dimensional alignment metrics? Such comparisons would help clarify whether CoRA's behavioral effects are attributable to improved alignment or to other properties of the prefix intervention.
  4. Attribution of multi-concept improvements: Could the authors clarify whether the optimized prefix is retained during downstream steering evaluation, as suggested by the supplementary implementation? If so, can they disentangle its direct effect from that of the newly extracted steering vectors, for example by evaluating prefixed and unprefixed extraction/evaluation combinations? Comparisons with random-prefix extraction and direct orthogonalization of the original vectors would also help establish the necessity of CoRA.
9 comments
AuthorsOpenReview ↗

Rebuttal (1/7): Weakness 1

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.

Weakness 1. Limited novelty and unclear incremental contribution: Context-dependent changes in linear representations have already been documented [1, 2], while optimizing continuous inputs to manipulate internal activations is also well established [3]. CoRA introduces a particular geometric objective for reorienting contrastive mean-difference vectors, but the paper does not sufficiently explain why this constitutes a significant advance beyond existing representation manipulation methods. Moreover, much of the behavioral evaluation uses the concept vector as an optimization target for a soft prefix rather than investigating the properties or utility of the resulting context-conditioned concept vectors. If CoRA is intended as a steering method, its advantages over conventional or directly optimized steering vectors [4] and standard prompt optimization remain unclear. The GCG and multi-concept applications also overlap with existing representation-guided optimization [5] and orthogonality-based steering [6], without sufficiently establishing the value of CoRA's specific formulation.

Reply: The optimization precedents are relevant and discussed in Appendix A (Extended Related Work). However, CoRA asks a more specific exploratory question and it's not intended to be another superior steering method: how broadly can a shared input prefix reorient an extracted contrastive representation toward an arbitrarily prescribed direction, while keeping the model, original examples, and extraction rule fixed? Prior observations of contextual variation do not establish this degree of directional control.

Our key objective is to study feasibility of representation-guided optimization. Our empirical results and theoretical analysis provide systematic support for controlling contrastive geometry through optimized context, complementing prior demonstrations focused on particular behavioral objectives. In Appendix D.2, separately optimized prefixes achieve cosine similarity of at least 0.7 with all 50 random target directions.

Sections 5.1-5.2 demonstrate two practical uses of this control. Section 5.1 deliberately conditions harmful inputs against a fixed harmless reference to produce behavioral changes through the prefix alone, establishing utility without necessarily requiring preservation of the original steering vector's semantics. Section 5.2 instead applies the same prefix to both classes of each concept, so common additive translations cancel and reorientation requires different class-mean responses to inputs. Here, CoRA encourages orthogonality between the extracted concept vectors, which are subsequently used with ActAdd to reduce steering interference. Together, these experiments demonstrate the usefulness of geometric optimization both for modifying behavior through inputs and for constructing context-conditioned steering vectors.

Regarding the utility of the resulting vectors, in Section 5.2, we showed that optimizing a shared extraction context improves measured isolation between two concepts across all three evaluated models. Thus, extraction context becomes a design variable for more selective steering.

[1] Lampinen et al., Linear Representations in Language Models Can Change Dramatically over a Conversation. arXiv, 2026.

[2] Adarsh et al., How Context Shapes Truth: Geometric Transformations of Statement-level Truth Representations in LLMs. ACL 2026.

[3] Bailey et al., Obfuscated Activations Bypass LLM Latent-Space Defenses. ICLR 2026.

[4] Cao et al., Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization. NeurIPS 2024.

[5] Wollschläger et al., The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence. ICML 2025.

[6] Nguyen et al., Multi-Attribute Steering of Language Models via Targeted Intervention. ACL 2025.

AuthorsOpenReview ↗

Rebuttal (2/7): Weakness 2 (Ambiguous interpretation)

Weakness 2. Insufficient justification of the central interpretation and behavioral contributions:

  • Ambiguous interpretation of concept reorientation: In the main experiments, the optimized prefix is applied only to the source class, while the reference class remains unchanged. Consequently, the observed reorientation can be explained by moving one class's mean activation relative to a fixed reference. While this establishes geometric controllability, it does not necessarily demonstrate that the representation of the same underlying concept can be arbitrarily reoriented under a shared context. More importantly, it remains unclear whether the optimized soft prefix preserves the original semantic interpretation of the contrastive samples. Without such validation, the reoriented difference-in-means vector may no longer represent the original concept, limiting the interpretability implications drawn from these results.

Reply: Thanks for your comments. CoRA's central purpose is to understand how strongly extraction context can change the orientation of a contrastive representation as discussed above. We agree that source-only conditioning does not by itself establish semantic preservation or arbitrary reorientation under a context shared by both classes. Our experiments actually examine complementary aspects of this question. We clarify the role of each experiment section below:

Section 5.1 examines geometric control and its behavioral usefulness. The random-target experiment in Appendix D.2 demonstrates substantial alignment with arbitrarily prescribed directions. Section 5.1 further tests whether geometric optimization can produce useful steering through the input prefix alone, without injecting a steering vector into the model. Source-only conditioning is deliberate here: the harmful prompts are the inputs whose responses we seek to change, while the harmless reference remains fixed. These experiments establish directional controllability and behavioral utility, rather than preservation of the original contrast's semantics.

In contrast, Section 5.2 directly addresses conditioning both classes and the usefulness of the resulting concept vectors. The same prefix is applied to both the positive and negative prompt sets for each concept. A common additive translation therefore cancels from the difference in means so reorientation requires different mean responses from the two classes. We instantiate CoRA to encourage orthogonality between the two extracted concept vectors (not for steering), then use these vectors for ActAdd.

Crucially, the resulting vectors retain substantial steering effects on their respective concepts A and B while improving measured selectivity across all three models. This provides behavioral evidence that their concept-specific meaning remains useful after CoRA, supported by a hinge-loss regularizer that constrains changes in the unsteered model's behavior (Appendix C.3, Eq. 44), even though it does not establish complete semantic preservation. In Table 1 below, we report the results for S0AS_0^A and S0BS_0^B, where S0CS_0^C is the model's original performance on concept C∈A,BC \in \\{A,B\\}. In the ActAdd case it is measured when steering strength α=0\alpha=0 and in the CoRA + ActAdd case it is measured with prefix applied and steering strength set to 0. The results show that unsteered scores between ActAdd and CoRA + ActAdd differ by at most 9.8%.

Table 1: Unsteered Scores with and without prepending the CoRA trained prefix in Section 5.2 on concepts A: coordinate-other-ais and B: survival-instinct.

ModelMethodS0AS_0^AS0BS_0^B
Llama3.2-3B-InstructActAdd18.4855.27
CoRA + ActAdd12.3265.07
Mistral-7B-Instruct-v0.3ActAdd11.9658.10
CoRA + ActAdd7.1064.51
Qwen3.6-27BActAdd7.3071.67
CoRA + ActAdd5.5964.98

Together, these results show that contextual sensitivity can both complicate the interpretation of static steering vectors and provide a means of improving their selectivity through a shared extraction context. We believe that our framework and findings encourage future research to consider extraction context as a design variable for more flexible steering.

AuthorsOpenReview ↗

Rebuttal (3/7): Weakness 2 (Limited attribution), Questions 1 and 2

Weakness 2. Insufficient justification of the central interpretation and behavioral contributions:

  • Limited attribution of downstream improvements: The jailbreaking experiments optimize a prefix toward essentially the same direction used by ActAdd, yet Figure 3 does not include the representations obtained after ActAdd, making it difficult to understand why CoRA achieves slightly higher ASRs or whether it provides better geometric alignment. The GCG extension primarily adds an activation-based loss to an existing optimization procedure, and its distinct contribution remains unclear. In the multi-concept experiments, the supplementary implementation appears to use the optimized prefix during both vector extraction and downstream evaluation, making it difficult to separate the effect of the extracted vectors from that of the prefix itself. Simple baselines, such as random-prefix extraction and direct vector orthogonalization, are also missing.

Reply: We thank the reviewer for the suggestion. Regarding the jailbreaking experiments, we replot the PCA visualizations of Figure 3 with ActAdd included in Figure 11 of the new Appendix D.7, where ActAdd is performed at its best-performing earlier layer and the resulting downstream activations are collected at the layer on which the prefix is trained, since performing ActAdd at that layer would trivially translate the activations by the steering vector. We also report full-dimensional metrics comparing the source-mean displacements induced by CoRA and ActAdd, which show that, at the last token, CoRA displaces the harmful activations along a common direction with ActAdd. We provide the detailed response to this in Q3 below. On the GCG extension, we intend it as a case study showing that CoRA's alignment loss can complement discrete suffix optimization, yielding modest ASR gains over standard GCG.

On the multi-concept experiments, we clarify that the optimized prefix is indeed used for both vector extraction and downstream evaluation. To separate their effects, we evaluate all prefixed and unprefixed extraction and evaluation combinations, and find that while the source of the gain depends on the model, using the prefix for both remains consistently best or second best. Regarding the missing baselines, we find that neither a random prefix of the same shape nor a direct orthogonalization of the original ActAdd vectors consistently reproduces CoRA. We provide the detailed response to this in Q4 below.

Question 1. Scientific contribution and novelty: Beyond demonstrating that optimized continuous inputs can manipulate internal activations, what is the key new insight provided by targeted reorientation of contrastive vectors? Is CoRA primarily intended as a diagnostic tool for representation stability or as a practical steering method? How should its contribution be distinguished from existing work on context-dependent representations, soft-prompt optimization, and representation-guided steering?

Reply: Please refer to our detailed response to Weaknesses 1 and 2 above. In summary, beyond characterizing the strength of contextual control, our framework and findings show that contextual sensitivity can both complicate the interpretation of static steering vectors (complementing prior observations [1, 2, 5] with direct, arbitrary-direction alignment experiments and relevant theory) and provide a means of improving their selectivity through a shared extraction context. We believe that our framework and findings encourage future research to consider extraction context as a design variable for more flexible steering.

Question 2. Validity of the concept representation interpretation: The main experiments apply the optimized prefix only to the source class. How much reorientation remains achievable when the same prefix is applied to both contrastive classes? More fundamentally, how can we establish that the resulting vectors continue to represent the original concepts rather than differences induced by changed input semantics or distributions? Evidence using shared-context interventions and independent semantic or behavioral validation would materially strengthen the central claim.

Reply: Please refer to our detailed response to Weakness 2 above. In summary, Section 5.2 applies the same prefix to both contrastive classes and shows improved selectivity while retaining concept-specific steering effects, providing behavioral support for the resulting interpretation of vectors.

Reviewer_7N8oOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper is introducing CoRA, a method that steers concept vectors by soft prompting. The authors show that they are able to tune a prefix to manipulate the direction of a concept vector. They achieve similar performance (potentially slightly higher ASR and sometimes lower side-effects) to ActAdd, i.e., direct steering of the vector. The authors then show that the CoRA loss can be beneficial for GCG and the prefixes can help to reduce multi-concept interference.

Critical strengths

The usefulness to decrease interference in the multi-concept steering is interesting.

The change of the interference of the prefix only at the final position is an interesting finding.

Critical weaknesses

To me the novelty / insights are quite limited here.

We already know that: Soft-prompting i.e. prefix tuning can optimize various things and can be used for jailbreaking [1, 2] Input attacks can manipulate representations of the concept [3,4, 5]

So I am not surprised that with that many degrees of freedom you can move the representation to an arbitrary position showing the reorientation of a steering vector.

Hence, I am missing the broader implications of this finding: what downstream insights does this actually bear? The work should provide stronger evidence for these. Sec- 5.2 is a good start but I am not yet convinced that this is enough.

[1] Xhonneux et al. Efficient Adversarial Training in LLMs with Continuous Attacks [2] Schwinn et al. Soft Prompt Threats [3] Wollschläger et al. Geometry of Refusal [4] Huang et al. Stronger Universal and Transfer Attacks by Suppressing Refusals [5] Cakar et al. Optimizing Against Safety Representations: Activation-Guided Adversarial Suffixes and the Geometry of Refusal

Questions for authors
  1. Do you have a clear vision of how this helps alignment research or our understanding of jailbreaks?
  2. The change of the interference of the prefix only at the final position is an interesting finding. Did you investigate this further? It would be interesting to see whether it just overwrites the actual answer or actually only steers it?
  3. When measuring ASR, do you evaluate whether the output is actually harmful or only classified as such? I.e., do the answers of the model actually correspond to the individual task? It has been shown that you can steer the output of a model via a soft prompt to emit a specific sequence. So it could be that the trained prefix just makes the model emit a similar sequence across inputs that gets classified as harmful. Could you share a few qualitative examples of the generations?

I am happy to engage in a discussion and change my mind. So please view the weaknesses as preliminary.

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2): Weakness 1 and Question 1

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.

W1. To me the novelty / insights are quite limited here. We already know that: Soft-prompting i.e. prefix tuning can optimize various things and can be used for jailbreaking [1, 2] Input attacks can manipulate representations of the concept [3,4, 5] So I am not surprised that with that many degrees of freedom you can move the representation to an arbitrary position showing the reorientation of a steering vector. Hence, I am missing the broader implications of this finding: what downstream insights does this actually bear? The work should provide stronger evidence for these. Sec- 5.2 is a good start but I am not yet convinced that this is enough.

Reply: We would like to clarify the contributions of our work and highlight the key novel aspects that distinguish it from prior work in response to the reviewer's novelty concern.

First, the optimization precedents are relevant. As discussed in Appendix A (Extended Related Work), these studies establish that optimized inputs can manipulate internal representations and that representation-level objectives can guide behavioral interventions. CoRA instead investigates the extent to which a shared input prefix can control the orientation of a contrastive representation toward an arbitrary prescribed direction, containing optimizing a particular behavioral feature or refusal direction as a special case.

Second, our key objective is to study feasibility of representation-guided optimization. our empirical results and theoretical analysis provide systematic support for controlling contrastive geometry through optimized context, complementing prior demonstrations focused on particular behavioral objectives. In Appendix D.2, separately optimized prefixes achieve cosine similarity of at least 0.7 with all 50 random target directions.

Sections 5.1-5.2 demonstrate two practical uses of this control. In particular, Section 5.1 deliberately conditions harmful inputs against a fixed harmless reference to produce behavioral changes through the prefix alone, establishing utility without necessarily requiring preservation of the original steering vector's semantics. Section 5.2 instead applies the same prefix to both classes of each concept, so common additive translations cancel and reorientation requires different class-mean responses to inputs. Here, CoRA encourages orthogonality between the extracted concept vectors, which are subsequently used with ActAdd to reduce steering interference. Together, these experiments demonstrate the usefulness of geometric optimization both for modifying behavior through inputs and for constructing context-conditioned steering vectors.

Q1. Do you have a clear vision of how this helps alignment research or our understanding of jailbreaks?

Reply: The key takeaway from our work and experimental design is that steering direction is very sensitive to the extraction context and the variation induced by context can be significant. On one hand, it challenges the faithfulness of static steering vectors by default. On the other hand, with carefully designed context, one could positively improve steering flexibility. We ellaborate on these below.

The immediate implication is that the context used to extract a steering direction is part of its experimental specification. Fixing the model, layer, original examples, labels, and extraction rule does not make the returned direction stable or general when source context changes. CoRA gives a way to deliberately test that sensitivity against arbitrarily chosen targets. This matters when interpreting a contrastive vector as a model property, a safety threat model or reusing it for steering under different contextual conditions.

Furthermore, the CoRA framework also suggests a way to reduce interference when several behavioral attributes are steered simultaneously. By optimizing the extraction context to accommodate selectivity, one can seek directions that produce more selective interventions, with smaller unintended effects on other attributes. Section 5.2 provides an initial demonstration: applying the same prefix to both contrastive classes and optimizing the relative geometry of two concept vectors leads to improved measured isolation for the CoRA + ActAdd procedure. This makes extraction context a potential design variable for multi-concept steering, alongside intervention strength and layer choice.

AuthorsOpenReview ↗

Rebuttal (2/2): Questions 2 and 3

Q2. The change of the interference of the prefix only at the final position is an interesting finding. Did you investigate this further? It would be interesting to see whether it just overwrites the actual answer or actually only steers it?

Reply: We thank the reviewer for the encouraging question, and would first like to clarify that the prefix changes the activations at every position, not only the final one. As shown in Figure 6 (right), the trained prefix induces a significant displacement throughout the sequence, with the ratio of the displacement magnitude to the unprefixed activation magnitude remaining substantial. It is only the alignment of this displacement with the steering vector computed at the layer on which the prefix is trained that is high near the end, as reflected by the large projection values at the final token positions in Figure 6 (left).

We study this further in Appendix D.5, where we examine how the source-mean displacement varies with token position through the pairwise cosine similarities and PCA projection in Figure 10. From the pairwise cosine-similarity matrix (Figure 10 (left)), the source-mean displacements at the content tokens remain strongly aligned with one another, while those at the chat-template tokens form a separate block. Consistently, in the PCA projection (Figure 10 (right)), the content-token displacements form a compact, ordered trajectory, whereas the chat-template displacements move abruptly into separated regions. Thus, unlike ActAdd performed at the same layer, which applies the same translation at every token position, CoRA's prefix induces a non-uniform intervention.

As for whether the prefix overwrites or steers the answer, the generations in Appendix D.6 (refer to Q3) differ across prompts and they do address each prompt’s specific request, instead of the refusal the model would normally produce. Thus, this suggests that the prefix steers the model, as the model still responds to the content of each prompt, rather than overwriting its response with a fixed output. We hope this addresses the reviewer’s question on overwriting versus steering, and are happy to clarify further if we have misunderstood it.

Q3. When measuring ASR, do you evaluate whether the output is actually harmful or only classified as such? I.e., do the answers of the model actually correspond to the individual task? It has been shown that you can steer the output of a model via a soft prompt to emit a specific sequence. So it could be that the trained prefix just makes the model emit a similar sequence across inputs that gets classified as harmful. Could you share a few qualitative examples of the generations?

Reply: We thank the reviewer for raising this point, and we have added Appendix D.6 with sample generations for selected prompts from the held-out set used in Section 5.1.1. We provide generations from both Llama3.2-3B-Instruct and Gemma-2-9b-it, and for both models the responses are fluent and appear to address each prompt’s specific request. Additionally, we note that the HarmBench classifier we use is given both the original prompt and the generation, and only counts a generation as harmful if it is a clear instance of the requested behaviour [1].

[1] Mazeika et al., HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ICML2024.

AuthorsOpenReview ↗

Any Questions from Reviewer 7N8o on Our Rebuttal?

Dear Reviewer 7N8o,

We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Paper Discussion Phase will end in less than 12 hours from this comment, i.e., 11:59 pm AoE on November 18. We are happy to answer any remaining questions you may have or provide further clarification before then, but we will no longer be able to respond once the discussion phase has concluded.

If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments!

Sincerely,

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response and Summary

Dear AC and Reviewers,

We thank the reviewers for their thoughtful feedback, which has helped us improve our paper. We are encouraged that the reviewers found that our paper addresses a clear and relevant question on how context affects concept vectors, and provides an interesting demonstration of geometric controllability, as noted by Reviewers UnuV and fDWc. They also found our theoretical and empirical treatment to be thorough and well-controlled across several model families. In addition, Reviewer 7N8o found the reduced interference in multi-concept steering interesting. Moreover, all reviewers found the paper clear and easy to follow. We have revised our manuscript accordingly, with all changes highlighted in blue.

One of the main concerns was the novelty and contribution of CoRA. We would like to clarify that CoRA is not intended to be another superior steering method. Instead, CoRA studies how strongly a shared input prefix can reorient a contrastive representation, while keeping the model, layer, examples, and extraction rule fixed. We show that the extracted direction can be substantially reoriented toward arbitrarily chosen directions through the prefix alone, which suggests that a steering vector should be interpreted together with the context it is extracted in. Building on this, in Section 5.2, we optimize a shared prefix applied to both classes, and find that the resulting steering vectors improve isolation between two concepts across all three evaluated models.

Another concern was whether the reoriented vectors preserve the original concepts. Our two settings differ in which classes receive the prefix, and intended to study complementary aspects of context sensitivity. In Section 5.1, the prefix is only applied to the harmful prompts, and the re-extracted vector is not used for steering (prefix is directly used for steering instead). In Section 5.2, the prefix is applied to both classes, and the extracted vectors still steer their original concepts. This provides behavioral evidence that they continue to represent these concepts.

Finally, the reviewers suggested additional comparisons and baselines. We compare CoRA with a behavior-optimized soft prefix and ReFT, which achieve comparable ASR but require partial target completions. For multi-concept steering, we also separated the effect of the prefix from that of the extracted vectors by evaluating all prefixed and unprefixed extraction and evaluation combinations. The source of the gain depends on the model, and using the prefix for both is consistently the best or second-best setting. We also add a random prefix and direct orthogonalization as baselines and show that they do not consistently reproduce these results.

AuthorsOpenReview ↗

Summary of Revisions

Based on the reviewers' comments and suggestions, we summarize below the main changes in the revised manuscript, with all changes highlighted in blue:

  1. We evaluate the cross-dataset transfer of the CoRA-trained prefix on the 159 standard behaviors of the HarmBench test set without any retraining, and report the results in Table 2 of Section 5.1.1.
  2. We provide qualitative examples of CoRA-induced generations from Llama3.2-3B-Instruct and Gemma-2-9b-it in Table 7 of Appendix D.6.
  3. We compare the activations induced by CoRA and ActAdd in Appendix D.7, where we include ActAdd in the PCA visualization of Figure 3 in Figure 11, and report the cosine similarity and projection of their source-mean displacements in Tables 8 and 9.
  4. We separate the effect of the prefix on steering vector extraction and downstream evaluation in the multi-concept experiment by evaluating all prefixed and unprefixed combinations in Table 5 of Section 5.2.1.
  5. We add a random prefix of the same shape and a direct orthogonalization of the ActAdd steering vectors as baselines for the multi-concept experiment in Tables 10 and 11 of Appendix D.8.

In addition, in our responses to the reviewers, we provide comparisons with a behavior-optimized soft prefix and ReFT on the jailbreaking task, the unsteered scores with and without the prefix in the multi-concept experiment, and results with a single-token prefix trained on 50 contrastive pairs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.