- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies activation steering in the setting where neither contrastive prompts nor paired preferred and nonpreferred completions are available. The proposed method, Unsupervised Latent Region Matching (ULRM), first runs the model zero-shot on an A/B behavior dataset and splits the examples by whether the model already gives the preferred answer. Prompt-side decision activations at one layer then form a preferred region and a nonpreferred region. A residual MLP is trained to move nonpreferred activations toward the preferred region while staying close to the identity on preferred activations. The pull toward the preferred region is measured either by a weighted nearest-neighbor distance (kNN-ULRM) or by a multi-scale kernel maximum mean discrepancy (MMD-ULRM). At inference, the learned map replaces the hidden state at the decision position. Experiments on two selections of a model-written sycophancy dataset with Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct compare both variants with the unsteered model, a mean-difference vector built from the same split, and Contrastive Activation Addition. The paper reports that MMD-ULRM achieves the highest average accuracy and correct-token probability, and further studies the identity term, the nonlinearity of the map, the training-set size, unbalanced training pools and computational cost.
Critical strengths
S1. Nonlinear generalization of mean-difference steering
The paper replaces a single global displacement between success and failure activations with a learned map between the two activation distributions. The architecture ablation supports this design. Replacing the MLP with a residual affine map lowers both metrics in every model, pool, and method combination (Table 2, lines 313-316). On Llama with the PAS pool, MMD-ULRM also exceeds the mean-difference baseline built from the same split by a clear margin (Table 1). These results indicate that the difference between preferred and nonpreferred decision states is not fully captured by one translation, which is the main empirical contribution of the submission.
S2. Controlled and transparent protocol
Layers, strengths, and checkpoints are selected on a held-out validation set. The data-scaling study fixes the number of optimizer updates across training-set sizes, which separates data quantity from optimization time (lines 345-351). The paper reports the additional compute of ULRM relative to the vector baselines (Table 4) and lists every selected configuration (App. C-D). This level of detail makes the empirical claims checkable and the method straightforward to reproduce.
Critical weaknesses
W1. Limited methodological novelty
The motivation of steering from observed successes and failures is reasonable, but the methodological novelty is limited. Partitioning examples by baseline success was introduced by Painless Activation Steering, which the paper cites. Learning a map that moves one unpaired activation distribution toward another has been studied in activation-transport steering[1] and in affine distribution-matching steering[2]. Relative to these works, the new element is mainly the use of a residual MLP with a kNN or MMD discrepancy and an identity term on the preferred region. The paper does not discuss these transport-based methods, clarify how region matching differs from them conceptually, or compare against them, even though they address the same unpaired setting. Beyond the choice of discrepancy and mapper, the paper offers limited insight into what the learned map changes in the representation and why this leads to the preferred behavior.
W2. Unclear supervision gap with CAA
The paper presents ULRM as using weaker supervision than CAA, and the title describes the method as unsupervised. However, constructing the split requires knowing, for every example, whether the model's choice is the preferred answer. In a two-choice task, the model's choice together with its correctness identifies the preferred answer, which is the same label CAA uses to build its pairs. The difference between the two methods therefore appears to lie in how the label is used, as an unpaired split versus paired completions, rather than in the amount of supervision. The main result is framed as outperforming CAA despite a supervision gap, so its significance depends on whether such a gap exists in the evaluated setting. Without this gap, the relevant comparison set also extends to methods that learn interventions from the same labeled activations, such as representation finetuning[3] and inference-time intervention[4], none of which is evaluated.
[1]: Controlling Language and Diffusion Models by Transporting Activations
[2]: Representation Surgery: Theory and Practice of Affine Steering
[3]: ReFT: Representation Finetuning for Language Models
[4]: Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Questions for authors
Clarifications that could affect the assessment:
In a two-choice task, the behavior split appears to identify the preferred answer for every example. In what sense is the supervision used by ULRM weaker than that used by CAA?
How does ULRM differ from activation-transport steering[1] and affine distribution-matching steering[2], and how does it compare with them empirically on the same data?
The sycophancy data pair a limited set of questions with many different user profiles. Are the training and test sets separated at the question level, and how do the results change when test questions are unseen during training?
Suggestions for future work:
The objective preserves per-example information only on the preferred region. An analysis of what the map changes in the representation may help clarify whether it moves activations toward the correct answer or toward regions associated with easier questions.
[1]: Controlling Language and Diffusion Models by Transporting Activations
[2]: Representation Surgery: Theory and Practice of Affine Steering
I would be happy to raise the score if the author could provide more evidence to answer my question.