- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether attention heads with low task-specific importance can still exhibit strong downstream output sensitivity. The authors propose a head-resolved Jacobian framework to distinguish broad output responsiveness, task-directed sensitivity, and behavioral relevance measured through activation patching and direct logit attribution. Experiments on GPT-2 Small, Pythia-410M, and GPT-Neo-125M reveal that some heads exhibit high Jacobian responsiveness despite low importance on the indirect object identification task. Finite interventions validate the local Jacobian predictions, while cross-task comparisons suggest that output alignment depends on the selected task.
Critical strengths
- The paper has clear motivation that highlights the distinction between task-specific importance and broader downstream sensitivity.
- The estimand seems correct which provides a clear and appropriate measure of local downstream sensitivity, distinct from task-specific importance.
Critical weaknesses
- The novelty seems limited. The conceptual motivation and Jacobian-based methodology overlap substantially with prior work, with limited new mechanistic insights.
- The generalizability is questionable. Experiments focus on three small models and two token-level tasks, without evaluating modern LLMs or multi-token generation.
- The Insights are weak as downstream sensitivity does not establish computational roles or causal relevance during inference.
Questions for authors
Major:
- Equation (4) represents task relevance through a predefined logit contrast, which is natural for IOI and greater-than but less clear for other tasks and realistic language generation. How should task importance and alignment be defined for multi-token outputs, where predictions depend on autoregressive trajectories and intermediate computations? Can the framework be extended to sequence-level likelihoods or task-level performance metrics?
- The selected Pythia heads exhibit an increase in mean alignment from 0.0018 on IOI to 0.0153 on greater-than (Section 5.2). However, both the task contrast and input distribution change between these experiments. How can the authors distinguish changes in task-direction alignment from changes in the prompt-conditioned Jacobian itself? Furthermore, given that matched controls reach a similar greater-than alignment of 0.0144, what evidence supports additional task-specific functionality?
- Equation (6) constructs interventions along Jacobian-derived gradient directions, and the resulting finite-difference errors are approximately 0.7%–1.1% at the primary perturbation scale. While this validates local sensitivity, how does it establish that these directions correspond to computations naturally used by the model? Could the authors investigate whether the identified heads participate in meaningful computational circuits or exhibit measurable causal effects on task performance beyond artificially selected perturbations?
- Equation (5) defines behavioral visibility as the maximum absolute effect from activation patching and DLA, although these methods capture different aspects of component contribution. What justifies combining them into a single ranking? More importantly, behavioral measurements rely on only 25 patching pairs and 40 DLA prompts, while the 2,000 bootstrap resamples assess broad-response rankings rather than behavioral-ranking stability. How stable are the five selected Pythia heads under resampling, alternative prompt distributions, and independent behavioral evaluations?
- The main high-response/weak-behavior evidence comes from Pythia-410M, while GPT-2 shows stronger coupling (Spearman ρ = 0.436) and no heads satisfying the same selection criterion. Can the authors evaluate modern architectures such as Qwen, Llama, or Gemma under a consistent protocol? How sensitive are the findings to model scale, architecture, and instruction tuning?
Minor:
- The figures are not clear enough.
Novelty and Experimental Claims Have Been Misinterpreted
We are happy that the reviewer acknowledges that our Jacobian estimand is suitable for measuring local downstream sensitivity appropriately. But we do not agree with the way the contributions of our paper are stated as well as with some of the criticisms made regarding our experimental findings.
Originality: We do not argue that Jacobians or gradient-based attribution are novel ideas. We have already discussed previous works in Section 2. Our contribution is that we are conducting a systematic comparison of head-level output sensitivity, task-related sensitivity and behavioral effectiveness, and performing our own validation experiments. The fact that the two most responsive Pythia heads have only 263rd and 374th positions in IOI behavioral effectiveness proves our point. This is more than just stating that gradients are different from behavioral relevance.
Greater than: The reviewer raises the question whether more alignment implies more causation. Section 5.2 already addresses this concern: the selected heads and matched controls show comparable Greater-than alignment (0.0153 and 0.0144, respectively). We explicitly avoid claiming special Greater-than causal functionality.
Finite interventions: It has already been shown in the previous experiments that local responses predicted by Jacobian estimation actually occur when with approximately 0.7%–1.1% relative error. They are not presented as evidence of naturally occurring computational circuits.
Overall, we believe some novelty and experimental claims were misinterpreted. Our conclusions concerned with the distinction between local downstream capacity and task-conditioned behavioral importance. We respectfully request that the novelty and evidence be criticized against these actual claims rather than stronger conclusions outside the paper's intended scope.