- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares attention heads localized through attribution patching on multiple-choice and free-form tasks, then evaluates their usefulness for activation steering. It reports that free-form localization improves free-form steering at small head budgets, while MCQA localization performs better under MCQA evaluation. Additional analyses examine head overlap, representation rank, steering-vector readouts, and residual-stream interventions.
Critical strengths
-
The paper investigates an important limitation of MCQA-based circuit localization and motivates the need to study model behaviors under more realistic free-form generation settings.
-
The evaluation covers four behaviors across five language models, with controlled comparisons between MCQA and free-form localization under different steering budgets and evaluation formats.
Critical weaknesses
-
Limited support for the main claim. The empirical results largely reflect a format-matching effect: free-form localization performs better on free-form evaluation, while MCQA localization performs better on MCQA evaluation. The advantage also vanishes at larger head budgets. It is therefore unclear whether the proposed approach provides a fundamentally better localization method.
-
Insufficient mechanistic evidence. The lower-dimensional representations and logit-lens results do not necessarily imply more precise or causally meaningful circuits. In particular, the relatively weak transfer observed in Figure 4 raises concerns about whether the proposed causal abstraction is adequately supported by the experiments.
-
Questionable evaluation of steering effectiveness. The qualitative examples in Appendix H show that steering often introduces hallucinations or substantially changes the original response content. This raises concerns about whether the reported improvements reflect precise behavioral control rather than unintended changes in generation. More rigorous evaluations of content preservation and off-target effects are needed.
Questions for authors
Can the authors provide length-matched or supervision-matched attribution experiments to distinguish the effect of response format from the effect of the likelihood objective?