- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies "targeted feature learning" that aims at identifying features for a concept chosen in advance, rather than finding it post-hoc as typically done with SAE dictionaries. The authors analyze 3 model signals (activation values, activation gradients and parameter gradients) with 2 estimators. This gives six methods: CAA and GRADIEND, which already exist, plus four new ones. The methods are compared with pretrained SAEs on 15 task. The main finding is that activation-value methods are the best for identifying whether a concept is present in the model's representations, while gradient-based methods are the best for causally steering the model's behaviour toward or away from that concept. Overall, the authors conclude that feature quality depends on both the signal and the estimator, and that detection and intervention measure different properties.
Critical strengths
- The authors conduct a systematic comparison that changes one thing at a time: what information the feature is built from, and how it is built. This analysis makes clear which choice drives the results, identifying the methods best at spotting a concept vs those that are best at changing the model's behaviour. I never saw this observation in previous literature and I think it should be taken into account in interpretability vs causal intervention work.
- The paper also introduces a new middle-ground method called AGIEND that uses gradient information but steers the model the same way activation methods do, by nudging its hidden states. It still steers almost as well as the weight-editing method showing that the steering advantage comes from the gradient information itself, not from editing the weights. I find this an interesting finding of the analysis.
Critical weaknesses
- Two of the three models are GPT-2 and Pythia-70M, which are small and old by today's standards, so Llama-3.1-8B is the only current-scale model. It is not clear to me why the authors decided to focus on these models rather than alternatives. Is the quality of SAEs good for these models?
- The practical takeaway remains a bit vague to me. The paper shows that spotting and steering favour different methods, but it never turns this into a concrete recommendation. The authors suggest a hybrid method that would combine the strengths of both families, but they don't test it. Even a small experiment would show whether the trade-off can be overcome or is more fundamental, which would make the paper's message more actionable.
Questions for authors
- Do you have an intuition or evidence for why gradient-based features change the model's behaviour more?
2 comments
I thank the authors for their response and clarifications. I will maintain my score.
Response to Reviewer vNcY
Thank you for your review. Please also see our general response summarizing the manuscript changes.
Weakness 1 (small, old models; SAE quality): The revised paper uses ten models instead of three, adding Gemma, Qwen, Mistral and DeepSeek models with up to 9B params. This broadens the model coverage across families and scale. The main findings hold across all ten. The additional models also include more recent SAE architectures and releases (e.g., Gemma Scope). The main findings remain consistent across the expanded evaluation. We further examine SAE performance on EMOTION in a new case study (Appendix C.7), showing that feature recovery varies across models and SAE suites.
(Note: The results were already computed during the review period, which made this fast and large extension of the results feasible.)
Weakness 2 (practical takeaway; hybrid untested): We have added explicit recommendations to the discussion (Section 6): CAA is particularly effective for detection, CAGA for activation space intervention with layer selection, and CGA for persistent model editing. We also evaluate the suggested hybrid approach (HYBRIEND) in a new appendix ablation (Appendix C.8). HYBRIEND combined activation value encoding with parameter gradient decoding and generally improves on the other IEND variants, but does not consistently outperform contrastive mean methods. Thus, combining the strengths of different signals is promising, although the trade-off is not fully resolved.
Question (why do gradient-based features change behaviour more?): Gradients literally point in the direction that changes the output the most by definition. Activation differences capture only how representations differ without necessarily indicating how to change the model's output. This makes gradients more directly aligned with the intervention objective. However, whether gradients aggregated across examples yield effective intervention directions is an empirical question. Our results support this intuition: CAGA (activation gradients) achieves roughly twice the intervention performance of CAA (activation values) with layer selection, despite both intervening in activation space. Thus, the advantage cannot be explained only by parameter space editing.