- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose to extract directions corresponding to the transition from interpretation to non-interpretation of increasingly corrupted text inputs in LLMs. They construct a contrastive dataset by progressively corrupting inputs and identifying transition pairs between interpretation and non-interpretation. They then extract directions using the difference-in-means method and validate the effects of adding or removing these directions. They thoroughly investigate generalization across datasets and corruption mechanisms, and compare the effects with random and refusal directions. Experiments are conducted on several models from different families and sizes.
Critical strengths
-
Interesting and thorough experiments, including generalization across datasets and corruption mechanisms.
-
Informative comparisons with random and refusal directions, as well as evaluations showing that general benchmark performance is largely preserved.
-
Clever construction of the contrastive dataset.
Critical weaknesses
The main weakness is the limited methodological novelty. Similar difference-in-means approaches have already been used to extract steering directions for behaviors such as refusal and sycophancy, as the authors acknowledge. The contribution is therefore primarily empirical.
Questions for authors
-
Since the interventions are applied throughout generation, could there be a compounding effect that causes outputs to drift over longer generations? Have the authors investigated the effect of generation length on output quality?
-
I found the effects of random directions in Table 2 somewhat surprising. Adding and removing random directions generally shift behavior in the intended direction (toward non-interpretation and interpretation, respectively). Do the authors have an intuition of why ?
-
In Section 3.2, the authors compute activation differences at matched token positions. Could they clarify whether the last five positions always correspond to fixed chat-template tokens? Otherwise, corruption-induced changes in tokenization could misalign the positions, potentially resulting in differences between representations of semantically different tokens.