acceptodds
Under review as a conference paper at ICLR 2027

Half-Debiased: Two Template Axes in Steering-Vector Geometry, and A Method They License

Abstract

Steering vectors push a large fraction of inputs in the unintended direction. A growing literature explains this using the geometry of per-sample activation differences, employing clustering, mixture-fitting, and routing techniques. Recent works identified answer-order and affirmative/negative biases in this benchmark as behavioral confounds. By measuring them instead as directions on 151 datasets across ten models, we establish three structural consequences that purely behavioral metrics cannot capture. Both axes are recoverable from individual differences at AUC 1.000 on all 2,114 measurements against a controlled null. Since they are near-orthogonal, the rank-one correction applied by the field removes one axis but leaves the other intact; removing both leads to an additional 0.037 of anti-steerability on Llama-2-7B. The extent to which these axes contaminate the deployed vector is governed by representational resolution, falling from a cosine of 0.887 at 0.5B to 0.118 at 14B. Fed into the cluster-then-route primitive these methods share, this geometry results in letter-pure prototypes on all 48,320 clusterings. While geometry predicts behavioral reliability across models, identical optimizers transfer geometric gains to behavior at different rates (17% versus 58%). Selective steering accounts for these facts: abstaining where the geometry predicts a misaligned push reduces anti-steerability by 0.071, supported by a distribution-free risk certificate and a probe requiring only the base prompt.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Steering Vectors, Contrastive Activation Addition, Template Axes, Selective Steering, Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_aCUYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

According to this paper, their contribution seems to be about improving the steering literature and the identification of a geometric score. However, the contributions are very obscure and hard to understand and summarize.

Critical strengths

N/A

Critical weaknesses

I have significant concerns about the overall presentation, clarity, significance and completeness of this manuscript. Several sections appear insufficiently proofread, which substantially affects the readability and makes it difficult to properly assess the scientific contributions.

First, the abstract is poorly structured and lacks clarity. In particular, I find the emphasis on numerical results somewhat unusual, especially given that the broader context and motivation are not clearly established. An abstract should provide a concise and self-contained overview of the work, including the context, the problem being addressed, the main challenges, and the proposed approach. Unfortunately, the current abstract does not adequately fulfill these objectives. Similar issues arise in the outline of contributions, which remain difficult to identify and understand. A reader should be able to clearly identify the research question, its motivation, and the authors' proposed contributions, but this is not achieved in the current manuscript.

Second, the submission appears to be incomplete, as the appendix is missing despite being explicitly referenced throughout the paper. For instance, the authors write:

"This tokenization split affects three of the ten models (both Llama-2 variants and Mistral, Appendix B.1; Table 6) and survives major tokenizer-library updates, confirming it as a durable property rather than a temporary artifact (Appendix A.1)." The absence of the referenced appendix makes it impossible to verify these claims and assess the supporting evidence. This is a major concerning issue for a submission at ICLR 2027.

Third, the related works, as written, lack clarity. Previous approaches are not sufficiently introduced, contextualized, or referenced, making it difficult to understand how the proposed work relates to the existing literature and what distinguishes its contributions from prior research.

Furthermore, the mathematical definitions are unclear and seem incomplete.

More generally, the manuscript lacks a clear and coherent narrative. The different sections do not seem to follow a well-defined line of reasoning, and it is often difficult to understand how the arguments connect or where the authors intend to lead the reader. This significantly limits the accessibility of the paper and the reader's ability to evaluate its scientific relevance.

Finally, the manuscript gives the impression of having been written very fast or with substantial assistance from AI-generated content. I want to emphasize that I am not suggesting that this was necessarily the case, but rather that this is the impression perceived as a reader, particularly given the lack of clarity, coherence, and careful proofreading throughout the paper. A clear example is the conclusion which is very hard to understand:

"The control the steering literature uses against answer-order bias protects the mean but gives no persample guarantee. Measured as directions rather than as behavioral variance, the standard construction imposes two near-orthogonal surface axes; therefore, the field’s rank-one correction removes only half of this structure: the other half remains decodable at ceiling on every model and costs 0.037 of anti-steerability on Llama-2-7B, while without any correction a standard cluster-then-route primitive recovers the letter on all 48,320 clusterings. How far these axes reach the deployed vector is governed by representational resolution. The resulting geometry is a valuable signal but not a reliable objective, and selective steering is the method that distinction licenses."

Overall, I strongly recommend rejection. In its current form, I believe the manuscript falls substantially below the standards expected for publication at a major machine learning conference such as ICLR. The issues outlined above are not only editorial: the lack of clarity, incomplete supplementary material, and insufficient positioning with respect to prior work make it particularly difficult to properly assess the validity, significance, and novelty of the proposed contributions.

Questions for authors

N/A

Reviewer_gVVpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper examines how superficial features of answer choices (such as letter assignment and Yes/No wording) can influence the geometry of activation steering vectors. Experiments across 10 LLMs and 151 datasets show that standard debiasing leaves residual answer-format biases, and that an ACT-style clustering procedure can recover these biases instead of the intended concepts. The authors then propose selective steering, which skips interventions that are predicted to have unintended effects.

Critical strengths
  1. The experimental is thorough, with cross validation, permutation tests and numerical precision checks to differentiate true geometric patterns from measurement biases.
  2. The findings reveal useful insights about the reliability of existing approaches: how superficial features of data can influence steering vector geometry and mislead results
Critical weaknesses
  1. The auditing focuses on multiple-choice benchmark, while steering methods examined (ACT, PDS, G-ACT) use different types of data. Thus, the broader claims about existing steering methods are not strongly backed by the empirical results. The stated problem that the paper sets out to fix is also centered only on the case of steering using multiple-choice training data.
  2. The evidence offered to show that selective steering relies on meaningful signals rather than residual answer-format is mostly correlational. Additionally, the deployed version is evaluated primarily through prediction accuracy rather than steering effects, raising concern about the true effectiveness of the proposed techniques.
  3. [Extra]: The paper is extremely dense and hard to read, with confusing organization and unclear interpretation. It also references a nonexistent appendix.
Questions for authors
  1. How would the findings extend to ACT, PDS, and G-ACT, when their original evaluation datasets do not contain the answer-letter and polarity biases studied here?
  2. Can the authors test whether selective steering relies on residual template signal, such as through controlled answer swapping or ablating the nuisance direction?
  3. How do deployable base-prompt probe perform in terms of actual behavioral metrics, rather than AUC?
  4. Can the authors supply evidence to support that clustering recovers superficial templates rather than structures related to actual concepts.
Reviewer_fM4kOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the effect of answer letter order and Yes/No polarity on Contrastive Activation Addition (CAA) differences per sample across a few small language models. The goal is to identify a universal steering vector representing each across models and datasets and mitigating the effect without affecting the target steering vector's effect (low anti-steerability). Shallow findings demonstrate that this can be done albeit with limited efficacy and only for one axis via rank-one probes across different settings.

Critical strengths

Thorough and very detailed analysis but not much high level substance - which is good during research but not in a write-up.

Critical weaknesses

The writing is very poor, abstruse and difficult to follow. A lot of detail and info density per sentence implies this wasn't written or read by a human.

The findings are quite shallow and over technically described. Polarity is not an issue in CAA means as noted by this paper already and only an issue per-sample which can be said for many other concepts that could be represented and are removed by definition by the CAA application.

Appendices are cited throughout but not presented.

MERA is mentioned only in spirit and a proper related works section with general techniques in this paper is absent

Questions for authors

See above weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.