- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies how input context can shape contrastive concept vectors in frozen language models. It proposes CoRA, a prefix-tuning framework that optimizes a shared soft prefix applied to source examples to align their steering vector with a prescribed direction while keeping reference examples unchanged. The paper provides necessary and sufficient local conditions for alignment and evaluates behavioral applications across diverse benchmarks, demonstrating the feasibility of the proposed method.
Critical strengths
S1. The paper addresses a clear and relevant question about how context affects concept vectors. Targeted reorientation provides a useful perspective beyond simply observing variation across contexts.
S2. The method is simple. By fixing the model, layer, original examples, and extraction rule while optimizing a shared prefix, the framework makes the intervention and its geometric effects easy to follow. The experimental setup is clean and well controlled, and the results align well with the observations.
S3. The empirical evaluation covers several model families and examines layer choice, prefix length, and alignment with random target directions, providing useful evidence about the method’s capabilities.
Critical weaknesses
W1. The learned soft prefixes lack a clear semantic interpretation. Geometric reorientation alone does not explain what semantic change the prefix induces. More behavioral analyses or comparisons with interpretable natural-language contexts would help connect the geometric findings to meaningful contextual variation.
W2. While the paper evaluates held-out prompts, cross-dataset transfer of the prefix-induced steering vector remains unclear. Testing the fixed vector on another dataset targeting the same concept would strengthen the evidence for the generality of its behavioral effects.
W3. The relationship to representation-finetuning methods such as ReFT [1] deserves more discussion. Both approaches use lightweight trainable interventions on a frozen model to influence representations and subsequent behavior. I understand that the training modules may differ, but given their similar effects, I think further comparisons with results from this line of work should be included.
[1] ReFT: Representation Finetuning for Language Models, NeurIPS 2024.
Questions for authors
Q1. Longer prefixes can improve geometric alignment while reducing ASR (Figure 6). If a single steering vector (e.g., ) can achieve the desired effect, could the additional capacity from learning multiple prefix embeddings increase the overfitting risk? I think a more rigorous study of the expressiveness of the learned prefix could help clarify the scope of the paper.
6 comments
Rebuttal (2/2): Weakness 3 and Question 1
W3. The relationship to representation-finetuning methods such as ReFT [1] deserves more discussion. Both approaches use lightweight trainable interventions on a frozen model to influence representations and subsequent behavior. I understand that the training modules may differ, but given their similar effects, I think further comparisons with results from this line of work should be included.
Reply: ReFT is relevant as both approaches train lightweight interventions while freezing the base model. However, they differ in where they intervene and what they optimize. ReFT, particularly LoReFT, learns transformations that directly edit hidden states at selected layers and token positions, trained through supervised output losses to improve downstream task performance. CoRA instead optimizes input-prefix embeddings to control the orientation of a contrastive mean-difference vector, with changes to internal representations mediated by the frozen model’s existing computations.
This distinction is central to our scientific question: how strongly can extraction context reshape a representation returned by a fixed extraction rule? Successful direct hidden-state editing does not establish which orientations are reachable through context alone. Section 5.1.1 then tests whether this geometric objective produces behavioral steering without target-completion supervision, while Section 5.2 uses shared-context optimization to improve the selectivity of subsequently extracted steering vectors. ReFT therefore provides a relevant comparison for behavioral adaptation, while CoRA investigates contextual controllability and its consequences for representation extraction.
Q1. Longer prefixes can improve geometric alignment while reducing ASR (Figure 6). If a single steering vector (e.g., ) can achieve the desired effect, could the additional capacity from learning multiple prefix embeddings increase the overfitting risk? I think a more rigorous study of the expressiveness of the learned prefix could help clarify the scope of the paper.
Reply: It's an interesting question. While overfitting risk is not impossible, the number of prefix tokens is not directly comparable to the "signleness" steering vector in any trivial way. A single injected steering vector and a single prefix embedding have different expressive capabilities despite having the same dimensionality. ActAdd directly specifies a hidden-state displacement, whereas a prefix must induce that displacement indirectly through the frozen model, across many inputs. Our theory also characterizes this distinction by uniformly bounding prefix-induced displacement at the original-token readout and relating local reorientation to the rank and conditioning of the prefix-to-contrast Jacobian. By comparison, ActAdd's injected displacement can grow arbitrarily large if its steering coefficient is unrestricted.
Moreover, the statement about sufficiency of a single steering vector requires a further care. A single vector is not established to suffice for reliable steering in general. For example, Wollschläger et al. (2025) identify multiple independent refusal directions and multidimensional concept cones. These findings challenge general single-direction uniqueness, without implying that multiple directions are necessary in every setting.
Thank you for the response. My concerns regarding W3 and Q1 remain.
Regarding W3, CoRA’s “context” consists of trainable continuous embeddings, so it is prefix tuning with an objective defined at the representation level, rather than a study of how natural language semantics shape representations. I agree that its geometric question is distinct, but CoRA and ReFT both modify representations by optimizing a small number of parameters while keeping the base model frozen. Their behavioral performance therefore warrants comparison under comparable conditions to establish CoRA’s additional value in Section 5. Differences in intervention location and training objective alone do not resolve this concern.
Regarding Q1, I would prefer studies or experimental results demonstrating why CoRA is better. For example, is there a method to determine how many prefix tokens are needed for a given task (i.e., whether the prefixes are expressive enough to encode the required representation shifts)? Does CoRA's expressivity allow it to achieve comparable performance with fewer trainable parameters or less training data than steering methods? Merely discussing the limitations of steering methods without providing supporting examples or experimental results does not adequately address this concern.
Rebuttal (1/2) Weaknesses 1 and 2
Reply: CoRA aims to understand how strongly extraction context can change contrastive representations. Indeed, geometric reorientation alone does not establish what a soft prefix means in natural language. Our experiments address complementary aspects and implications of geometric controllability:
Reply: We appreciate the reviewer’s suggestion. We note that in Section 5.1.1, the object trained by CoRA, and hence the intervention, is the prefix itself, so we evaluate cross-dataset transfer of the prefix. Specifically, we evaluate the trained prefix on the 159 standard behaviours from the HarmBench test set [1] on all models, without any retraining. We report the results in Table 8, alongside the ASR with no intervention and with a random 4-token prefix as baselines for comparison. The table shows that the prefix achieves a high ASR on all models, ranging from 85.53% to 94.34%, well above both baselines. This provides evidence that the preifx is transferable across datasets, supporting the generality of the behavioural effect it induces.
Table 8: Attack Success Rates when prepending the CoRA trained prefix in Section 5.1.1 on 159 standard behaviours from HarmBench, on all models.
[1] Mazeika et al., HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ICML 2024.