- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper aims to identify sparse autoencoder features that have a targeted downstream effect when used for steering, in particular features that change the model’s generated text. The key idea is to select features that, when steered, increase the probability of target tokens on a set of prompts, while normalizing against the probability of a set of unrelated distractor tokens on those same prompts.
Critical strengths
I appreciate the scope across several SAE families, and I agree there is value in studying features by their downstream effects rather than static metrics.
Critical weaknesses
Fundamentally, the overall contribution of this paper feels relatively minor and unsurprising. If you explicitly select features by how much they increase the probability of some tokens and decrease the probability of others, it is unsurprising that the same metric looks better on held-out prompts than selecting on positive token mass alone or on logit-lens-style methods. Moreover, the held-out and selection prompts are relatively similar, and both only consider single-token behavior.
Methodologically, there are many other ways to select features that are not compared against — for example, gradient-based attribution or activating text examples.
When evaluating steering, a larger existing benchmark such as AxBench would make more sense than a small homemade dataset, especially because it lets you consider the effects of steering as a frontier over steering strengths: effect size against overall degradation and model coherence.
The writing and clarity are somewhat poor. Much of the prose reads as though it was primarily language-model-generated, and is hard to follow due to heavy use of jargon; simpler language when describing methods and metrics would help. I acknowledge that some of this is a matter of taste, but it is worth raising as a separate critique from the weakness of the results themselves.
Questions for authors
None