- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a simple intervention for LLM-as-classifier. The work builds on CAA (Rimsky et al.), which constructs a task-specific steering vector by computing the mean difference between hidden-state representations associated with two classes. The main extension proposed in this paper is the introduction of an instance-specific steering strength, which is combined with the task-specific steering vector during inference. Experiments are conducted on six binary classification datasets, and the authors report that the proposed AdaSteer method outperforms baseline (no steering). However, I did not find the description of the baseline and had difficulty understanding exactly how they (both baseline and proposed method) are implemented.
Critical strengths
- The high-level idea of the proposed method makes sense and is easy to understand.
- The authors conduct statistical significance tests to determine whether the observed improvements over the baseline are statistically significant.
Critical weaknesses
- The writing quality is not great, and I found many of the technical details difficult to follow. In particular, Section 3.2, which describes the core proposed method, is not well written. I still could not fully understand it after several readings, and I believe the description lacks sufficient detail.
- Line 202: Do you mean eight random numbers sampled separately for each example, or are they fixed for all examples?
- Lines 205-206: What does mean?
- Line 220: Why are the values +3 and -3 used here? Are they related to the range defined in Line 202?
- Line 232:
predict the document-specific steering strength using a lightweight Multilayer Perceptronis confusing. My understanding is that the method evaluates several candidate steering strengths, obtains classifier logits, and uses them to construct a document representation. However, it is not clear how the steering strength itself is predicted. - Line 256: The text states that the predictor produces a magnitude, whereas the earlier description suggests that is a scalar?
- I also fail to understand the claim that, with a frozen LLM, the proposed method modifies the model's
internal computationWhat do theseinternal computationrefer to, and how are they being altered?
- There is very little discussion of the experimental results. Section 4.4 mainly describes what was done and references tables and figures, rather than describing the findings. The authors should provide a clearer summary of the key results. In addition, the claims made in Section 5 should be more explicitly linked to the empirical evidence. Rather than repeatedly saying
as shown in Tables 1, 2, 3, 4, and 5the authors should explain which specific results support each claim and why.
Questions for authors
- All evaluated datasets are binary classification tasks. It would be interesting to investigate whether the proposed approach can be extended to multi-class and even multi-label classification settings, if technically feasible.
- The evaluation covers four model families (Llama 3.2, Gemma 2, GPT-2, and Qwen 2.5), but these models are already somewhat outdated. It would be interesting to see whether the findings generalize to more recent LLMs.
- Line 269: The equation is difficult to follow, and I am not sure what it is intended to represent.
- Figure 3: I did not find this figure informative. The authors may want to consider a different visualization.