acceptodds
Under review as a conference paper at ICLR 2027

Beyond Task-Conditioned Importance: Head-Resolved Output Jacobians Separate Broad Response from Behavioral Relevance

Abstract

Interpretability in explainable AI is often quantified by the importance of an internal component by how much it 'contributes' to a given prediction/task/behavior. Here we ask the orthogonal question: can an unimportant component for one behavior be highly gainful in other directions of output space? We formulate a head-resolved framework that disentangles gross downstream responsivity, task-guided sensitivity/alignment, and behavioral visibility at the prediction head. This is one of the first systematic comparison of head-output-to-vocabulary Jacobian response with both task-directed sensitivity and behavioral readouts across multiple language models. We study GPT-2 Small, Pythia-410M, and GPT-Neo-125M using output Jacobians, activation patching, and direct logit attribution. The clearest case occurs in Pythia-410M: its two most broadly responsive heads rank only 263rd and 374th in behavioral visibility on indirect object identification. Finite interventions closely match the Jacobian-predicted slopes (approximately 1% relative error at the tested scales), while normalized output gain remains comparable to high-response controls matched with sensitivity. We find that their finite-head gain also varies when we change the task contrasts. This demonstrates that poor importance of task-conditioned task need not correlate with weak local responses downstream. Gross gain, task-guided alignment, and behavioral visibility are therefore orthogonal views of a given model component, rather than synonyms for importance.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, transformer interpretability, attention heads, Jacobian analysis, task-conditioned importance, activation patching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3DLUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether attention heads with low task-specific importance can still exhibit strong downstream output sensitivity. The authors propose a head-resolved Jacobian framework to distinguish broad output responsiveness, task-directed sensitivity, and behavioral relevance measured through activation patching and direct logit attribution. Experiments on GPT-2 Small, Pythia-410M, and GPT-Neo-125M reveal that some heads exhibit high Jacobian responsiveness despite low importance on the indirect object identification task. Finite interventions validate the local Jacobian predictions, while cross-task comparisons suggest that output alignment depends on the selected task.

Critical strengths
  • The paper has clear motivation that highlights the distinction between task-specific importance and broader downstream sensitivity.
  • The estimand seems correct which provides a clear and appropriate measure of local downstream sensitivity, distinct from task-specific importance.
Critical weaknesses
  • The novelty seems limited. The conceptual motivation and Jacobian-based methodology overlap substantially with prior work, with limited new mechanistic insights.
  • The generalizability is questionable. Experiments focus on three small models and two token-level tasks, without evaluating modern LLMs or multi-token generation.
  • The Insights are weak as downstream sensitivity does not establish computational roles or causal relevance during inference.
Questions for authors

Major:

  • Equation (4) represents task relevance through a predefined logit contrast, which is natural for IOI and greater-than but less clear for other tasks and realistic language generation. How should task importance and alignment be defined for multi-token outputs, where predictions depend on autoregressive trajectories and intermediate computations? Can the framework be extended to sequence-level likelihoods or task-level performance metrics?
  • The selected Pythia heads exhibit an increase in mean alignment from 0.0018 on IOI to 0.0153 on greater-than (Section 5.2). However, both the task contrast and input distribution change between these experiments. How can the authors distinguish changes in task-direction alignment from changes in the prompt-conditioned Jacobian itself? Furthermore, given that matched controls reach a similar greater-than alignment of 0.0144, what evidence supports additional task-specific functionality?
  • Equation (6) constructs interventions along Jacobian-derived gradient directions, and the resulting finite-difference errors are approximately 0.7%–1.1% at the primary perturbation scale. While this validates local sensitivity, how does it establish that these directions correspond to computations naturally used by the model? Could the authors investigate whether the identified heads participate in meaningful computational circuits or exhibit measurable causal effects on task performance beyond artificially selected perturbations?
  • Equation (5) defines behavioral visibility as the maximum absolute effect from activation patching and DLA, although these methods capture different aspects of component contribution. What justifies combining them into a single ranking? More importantly, behavioral measurements rely on only 25 patching pairs and 40 DLA prompts, while the 2,000 bootstrap resamples assess broad-response rankings rather than behavioral-ranking stability. How stable are the five selected Pythia heads under resampling, alternative prompt distributions, and independent behavioral evaluations?
  • The main high-response/weak-behavior evidence comes from Pythia-410M, while GPT-2 shows stronger coupling (Spearman ρ = 0.436) and no heads satisfying the same selection criterion. Can the authors evaluate modern architectures such as Qwen, Llama, or Gemma under a consistent protocol? How sensitive are the findings to model scale, architecture, and instruction tuning?

Minor:

  • The figures are not clear enough.
1 comment
AuthorsOpenReview ↗

Novelty and Experimental Claims Have Been Misinterpreted

We are happy that the reviewer acknowledges that our Jacobian estimand is suitable for measuring local downstream sensitivity appropriately. But we do not agree with the way the contributions of our paper are stated as well as with some of the criticisms made regarding our experimental findings.

Originality: We do not argue that Jacobians or gradient-based attribution are novel ideas. We have already discussed previous works in Section 2. Our contribution is that we are conducting a systematic comparison of head-level output sensitivity, task-related sensitivity and behavioral effectiveness, and performing our own validation experiments. The fact that the two most responsive Pythia heads have only 263rd and 374th positions in IOI behavioral effectiveness proves our point. This is more than just stating that gradients are different from behavioral relevance.

Greater than: The reviewer raises the question whether more alignment implies more causation. Section 5.2 already addresses this concern: the selected heads and matched controls show comparable Greater-than alignment (0.0153 and 0.0144, respectively). We explicitly avoid claiming special Greater-than causal functionality.

Finite interventions: It has already been shown in the previous experiments that local responses predicted by Jacobian estimation actually occur when with approximately 0.7%–1.1% relative error. They are not presented as evidence of naturally occurring computational circuits.

Overall, we believe some novelty and experimental claims were misinterpreted. Our conclusions concerned with the distinction between local downstream capacity and task-conditioned behavioral importance. We respectfully request that the novelty and evidence be criticized against these actual claims rather than stronger conclusions outside the paper's intended scope.

Reviewer_kfRxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this study, the authors argue that having a low importance score on a task does not necessarily mean that the attention head would not change the model output significant so they measure the output, Jacoby and response for each of the heads, and they compare that with task alignment and behavior of visibility and three models. The authors claim that the results show: response, alignment and behavior visibility could be distinct views of a component.

Critical strengths
  1. This study presents a clean way to distinguish a head's capacity to change the output from how important it is for a task

  2. The Jacobian predictions are checked against actual perturbations. Careful and transparent controls are strong.

Critical weaknesses
  1. This study shows that the heads can show low task behavior but high local response, but they do not check whether the activations change significantly during the task. Considering that the selected heads have small norms and depend a lot on the first token, without checking how much their natural activation changes between different prompts, the weak behavior while having high response may just reflect inactive heads. The authors acknowledge this but do not test for it.

  2. The response scores are aggregated across many vocabulary tokens, and the control heads are matched on this same aggregated score (including tokens the model never predicts). The fact that they have similar gain might be circular. The authors need to show the results under probability-weighted or task-relevant token subsets rather than random directions.

Questions for authors
  1. Could you report how much the selected heads' output change for clean and corrupted IOI prompts, comparing them to the controls, and whether this would explain the patching effect? This would address my first concern in the weakness section.

  2. Do the results hold if you had replaced the aggregated score over all tokens with only the tokens that the model might actually predict?

2 comments
AuthorsOpenReview ↗

Natural Activation and Control Matching Are Being Conflated with Our Main Claim

We acknowledge the reviewer’s comment on the importance of our framework and rigor in finite intervention controls. However, we must disagree with the understanding of two methodological points.

Natural activation changes: In Section 3, we differentiate between Jacobian responsiveness and natural activation changes introduced by Δℓ ≈ J_hΔz_h. In this case, an activation may have highly responsive downstream activity, but the head may underperform a task because its natural activation changes are small or not properly matching. In Section 5.3, we also show that our selected heads have low activation norms. The mentioned point is essential and not a minor misunderstanding. We do not need to prove that our heads have natural activation in the other tasks.

Control-group circularity: The decision to match controls based on broad Jacobian response stems from an intention to compare behavioral significance between heads with equivalent screened sensitivity. The following limited interventions entail real nonzero output movement instead of retracing the previous Jacobian computation. Matching alone does not guarantee that there will be any concordance between finite perturbation levels, although there are expectations regarding local linearity.

Probability-weighted sensitivity and natural activation variation are very useful analyses; however, they answer different questions compared to the full vocabulary logit-response that we precisely define. Neither concern contradicted the observed separation between broad downstream responsiveness and IOI behavioral importance.

Thank you for the response.

1. Are the selected heads simply inactive? Activation norm is not the same as the task-induced change Δz, as Section 5.3 itself notes, so it doesn't answer my question. My worry is also more specific. R_h is measured before W_O, which means you can scale W_V up and W_O down without changing the model at all, yet R_h changes. A head with a small activation and a large W_O will look highly responsive. Your own Table 6 hints at this: G_h · ‖z‖ is roughly 9 for the selected heads, versus about 39 and 46 for the two control groups. Could you report ‖z_clean − z_corr‖ on IOI for each group, and recompute R_h on the projected head output? I'd like to know whether the selected heads still stand out once their small scale is accounted for.

2. Does the raw-logit gain reflect anything meaningful? Since the finite changes match the Jacobian to within about 1%, and G_h also uses random directions, it is expected that groups matched on R_h end up with similar G_h. More importantly, random directions over the full vocabulary are dominated by tokens the model essentially never predicts, plus the uniform logit shift that softmax ignores. Could you recompute R_h with centered logits, or with a probability-weighted (KL) version, and check whether the same heads are selected?

If the intended claim is limited to raw-logit capacity under artificial perturbation then the title and abstract should say so explicitly. Until these points are addressed, I'll keep my current score.

Reviewer_MAoqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the distinction between an attention head's broad downstream influence on vocabulary logits and its importance for a specific task. The authors introduce a head-resolved output-Jacobian framework that measures broad vocabulary-wide responsiveness, task-directed sensitivity and alignment, and behavioral visibility through activation patching and direct logit attribution. The framework is evaluated on GPT-2 Small, Pythia-410M, and GPT-Neo-125M, primarily using indirect object identification (IOI), with Greater-than as an additional task contrast. The main finding is that some attention heads, particularly in Pythia-410M, exhibit high vocabulary-wide Jacobian responses despite relatively low IOI behavioral importance. Finite interventions along Jacobian-derived directions closely match the predicted local responses and demonstrate substantial normalized output gain for these heads. The authors also observe that alignment with task-specific output directions varies across tasks. The paper argues that task-conditioned importance should not be interpreted as a complete measure of an attention head's downstream responsiveness.

Critical strengths
  1. The paper makes a useful conceptual distinction between an attention head's overall influence on vocabulary logits and its contribution to a particular task or behavioral contrast. Comparing head-resolved output Jacobians with task-conditioned attribution and intervention measurements provides a complementary perspective on attention-head importance and illustrates why low task-specific importance does not necessarily imply weak downstream sensitivity.
Critical weaknesses

1. Insufficient methodological clarity and justification of experimental choices, metrics, and control groups

Several important aspects of the experimental methodology are insufficiently explained or justified, making the results difficult to interpret and the comparisons less convincing.

First, some central measurements are introduced without sufficiently precise definitions. In Section 3.2, activation patching and direct logit attribution (DLA) are described verbally, but the paper does not explicitly formulate how the quantities PhP_h and DhD_h are computed and aggregated. Since these measurements are combined into the behavioral visibility score BhB_h, the exact computation and normalization conventions are important for interpreting the selected-head rankings.

Second, the experimental configurations differ substantially across models without adequate justification. Table 1 specifies 32, 16, and 1 random vocabulary probes per prompt for GPT-2, Pythia, and GPT-Neo, respectively, while the numbers of screening and finite-validation prompts also vary. The paper does not sufficiently explain these choices or examine their effects on the reliability of the measurements. GPT-2's finite validation is marked as "Not needed." Although the authors explain that no GPT-2 heads satisfy the selected-head criterion, this does not adequately justify omitting finite-response validation for GPT-2 entirely.

Third, the selection and role of the control groups remain insufficiently motivated. Section 4.2 introduces high-response/weak-behavior heads, behaviorally strong controls matched by broad response, and an additional high-behavior group. Although these groups are identified in Figure 1 and subsequently compared in Figure 2, the paper does not clearly explain what hypotheses the different control groups are intended to test, or what conclusions can be drawn from the comparisons. In particular, matching broad response alone does not control for other potentially important properties such as layer position, activation magnitude, or attention patterns. The choice of the top-10% response and bottom-50% behavioral thresholds is also not justified, and the bootstrap analysis only tests response-rank stability rather than the stability of the complete joint selection criterion.

Fourth, the choice of the Jacobian's intervention point requires further justification. The paper differentiates vocabulary logits with respect to the value-weighted head output before the head-specific output projection, rather than the projected per-head contribution to the residual stream. The latter is a natural object in established attention-head circuit analyses, such as A Mathematical Framework for Transformer Circuits (Elhage et al., 2021). Although these two representations are related by the output projection, their associated sensitivity norms and allowable intervention directions are not generally equivalent. The authors should clarify why the pre-projection representation is preferred and whether this choice affects the resulting head rankings.

Finally, the empirical roles of several proposed metrics are not clearly connected to the reported results. For example, task-directed sensitivity Th(x)T_h(x) is formally introduced in Section 3.2, but is not systematically analyzed across models alongside broad response, finite gain, and task alignment. A more coherent presentation of how each metric is used to support the main conclusions would improve the paper substantially.

2. Limited empirical scope and a lack of mechanistic analysis of the identified high-response heads

The experimental evaluation is limited to three relatively small language models and primarily one task, IOI, with Greater-than serving as a limited secondary comparison. The strongest results are concentrated in Pythia-410M, and the main finite-intervention analysis involves only five selected heads. GPT-2 does not exhibit the same selected-head regime, while GPT-Neo uses a substantially different and less comprehensive evaluation protocol. Consequently, the empirical evidence is insufficient to establish how broadly the proposed distinction manifests across model scales, architectures, or tasks.

More importantly, the paper provides little mechanistic characterization of the heads it identifies. Despite presenting high-response/weak-behavior heads as the main phenomenon of interest, the authors do not systematically investigate their functional roles, attention patterns, output representations, or downstream computations. Appendix C reports some descriptive observations, such as the selected heads' relatively high attention to the BOS token, but these observations are not connected to an explanation of why these heads exhibit large Jacobian responses or low IOI behavioral visibility. There is also no systematic analysis of their projected output vectors, the vocabulary directions most strongly affected by their activations, or the relationship between their high-gain directions and interpretable representations.

A particularly important missing experiment is the evaluation of these heads through ablation or causal interventions on broader natural language modeling behavior. If a head has high vocabulary-wide output sensitivity but low IOI importance, it would be valuable to determine whether ablating it substantially affects language modeling tasks where far more tokens are involved as compared to the restricted IOI setting, or other tasks with larger output spaces. Such experiments could establish whether the identified heads have meaningful general-purpose functions or merely exhibit large responses to specially constructed perturbations. The finite-intervention experiments do not resolve this issue. The perturbation directions are chosen using the Jacobian gradients themselves, so the agreement between predicted and observed output changes primarily validates local linear sensitivity (and along those specific directions only, instead of randomly chosen directions). It does not show that these directions arise during natural computation or correspond to meaningful head functions, and these experiments are only intended for validataing the accuracy of the first-order Jacobian approximation anyway. Furthermore, the selected heads have substantially smaller activation norms than their controls, making the relationship between normalized finite gain and naturally realized output influence particularly important to investigate.

The Greater-than experiment also offers limited additional behavioral evidence. Although the selected heads exhibit higher alignment under the Greater-than contrast than under IOI, their alignment is comparable to that of the matched controls. No corresponding behavioral interventions demonstrate that these heads meaningfully contribute to Greater-than performance. The results therefore establish task-dependent geometric alignment, rather than identifying alternative computational roles for these heads.

Finally, the main sensitivity measures operate on raw vocabulary logits. The paper does not establish whether large logit-space changes translate into meaningful changes in output probabilities, predictions, or task performance. As a result, the interpretation of broad response as a meaningful form of general head influence remains insufficiently supported.

Questions for authors

Clarification Requests

1. Definitions and computation of behavioral visibility.

Could the authors provide explicit mathematical definitions and implementation details for the activation-patching effect PhP_h and the DLA contribution DhD_h, including how they are aggregated and normalized before being combined into the behavioral visibility score BhB_h? In particular, how is the comparability of these two quantities ensured?

2. Head-selection thresholds, control-group design, and omitted GPT-2 validation.

Could the authors clarify the motivation for selecting heads using the top 10% of broad response and bottom 50% of behavioral visibility, as well as the specific criteria and hypotheses underlying the two control groups? The current explanation does not make sufficiently clear what the comparisons with sensitivity-matched and high-behavior controls are intended to establish. More importantly, the selection thresholds are sufficiently restrictive that no GPT-2 heads qualify, resulting in the complete omission of the corresponding finite-response validation for GPT-2. Why are these particular thresholds necessary, and why are they maintained at the expense of evaluating the proposed phenomenon in one of the three models? Have the authors conducted any sensitivity or threshold-ablation analyses to determine whether the conclusions remain stable under alternative selection criteria?

3. Inconsistent experimental configurations across models.

What motivates the substantially different numbers of random vocabulary probes, screening prompts, and finite-validation prompts across models? In particular, why does GPT-Neo use only one random probe per prompt, compared with 16 and 32 for Pythia and GPT-2? How do these differences affect the reliability and comparability of the resulting Jacobian-response rankings?

4. Choice of the Jacobian intervention point.

Why is the Jacobian computed with respect to the pre-output-projection head activation, rather than the head's projected contribution to the residual stream? The latter is commonly used in attention-head circuit analyses, as in Elhage et al. (2021), A Mathematical Framework for Transformer Circuits. Since the two intervention spaces are related by the output projection but do not generally yield equivalent sensitivity norms, could the authors clarify the motivation for their choice and its implications for interpreting head-level responsiveness?

5. Role of task-directed sensitivity in the empirical results.

What role does the task-directed sensitivity metric Th(x)T_h(x) play in supporting the paper's main conclusions? Although it is introduced as a central component of the framework, its distributions and relationships with broad response, finite gain, task alignment, and behavioral visibility are not systematically reported across models. Could the authors clarify how this metric contributes to the empirical analysis?

Suggestions for Further Experiments

1. Mechanistic characterization of high-response/weak-behavior heads.

A more detailed mechanistic analysis of the selected heads would substantially strengthen the paper. In particular, it would be useful to examine their attention patterns, projected output representations, dominant vocabulary-response directions, and interactions with downstream components. The unusually high BOS attention reported for selected Pythia heads in Appendix C is potentially interesting, but currently remains descriptive. Investigating whether this pattern explains their Jacobian responsiveness or behavioral properties could provide a more concrete interpretation of the findings.

2. Ablation and causal evaluation on broader language modeling behavior.

The authors could examine whether the selected high-response/weak-behavior heads have meaningful functions beyond IOI by evaluating their effects on language modeling loss, next-token prediction distributions, or other tasks under head ablation or causal interventions. Such experiments would help determine whether high vocabulary-wide Jacobian response corresponds to substantial influence on natural model behavior. In particular, it would be informative to test whether these heads contribute strongly to general language modeling despite their weak IOI attribution scores.

3. Natural activation changes and behavioral relevance of high-gain directions.

The finite interventions are constructed along Jacobian-gradient directions, rather than directions necessarily realized during natural computation. The authors could compare these high-gain directions with naturally occurring head-activation differences across prompts or tasks. This would help establish whether the model actually uses the directions that produce large downstream responses, rather than merely being sensitive to artificial perturbations.

4. Stronger cross-task and cross-model validation.

The Greater-than results show increased alignment for selected heads, but the matched controls reach similar levels, and no corresponding behavioral intervention demonstrates that these heads are functionally important for Greater-than. Extending the analysis to additional tasks, larger language models, and task-specific activation patching or ablation would help establish whether the observed phenomenon generalizes and whether the selected heads serve identifiable alternative functions.

5. Output sensitivity beyond raw-logit displacement.

The authors could evaluate downstream responsiveness using changes in output probabilities, KL divergence, or prediction performance in addition to raw-logit norms. Since large changes in vocabulary logits do not necessarily imply meaningful changes in the predictive distribution, such experiments would clarify the functional significance of the proposed broad-response measurements.

2 comments
AuthorsOpenReview ↗

Mischaracterization of Existing Evidence and Soundness Concerns

We appreciate the reviewer's evaluation, but we have some differing viewpoints about some of the criticisms raised about our manuscript.

Natural activation: Section 3 states that local Jacobian sensitivity is different from natural activation because Δℓ ≈ J_hΔz_h. Section 5.3 confirms that heads with small activation norms (0.581, as opposed to 2.636 and 3.776). We never said that high Jacobian local sensitivity measurements results prove that this micro-model is important in a broader sense. To criticize the article for not proving this broader micro-model implication is an inaccurate reflection of its purpose.

Greater-than: In section 5.2, we demonstrated that the somewhat aligned activation between states, which could be analogous to the controls, is not an indication of the special Greater-than feature. This distinction was not overlooked.

GPT-Neo and statistical proofing: The DLA-only protocol is mentioned as a weak evidence. Moreover, Section 4.3 states that minimum two-sided Wilcoxon p value is 0.0625 (5 matched pairs).

Experimental controls: Response-matched controls are chosen independently of finite-validation outcomes, while the subsequent interventions test the presence of actual changes in output against the predictions offered by Jacobian.

This has already been explained as methodological omissions vs conclusions about the findings. The absence of additional experiments does not establish that the reported Jacobian results are entirely incorrect. Given the severe Soundness score of 1, we respectfully ask the reviewer to identify which specific mathematical derivation or empirical measurement is considered invalid.

Thank you for the authors' response. I would like to clarify the reasons for my Soundness score and address several apparent misunderstandings of my original review.

1. Reasons for the low Soundness score

My Soundness assessment does not question the mathematical correctness of the Jacobian or the accuracy of the finite-difference approximation. Rather, it reflects concerns about the clarity, justification, consistency, and completeness of the experimental methodology, as detailed in Weakness 1 and Questions 1–5.

The experimental protocols differ substantially across models, including the numbers of random vocabulary probes (32, 16, and 1), screening and validation prompts, and behavioral measurements, without adequate justification or analysis of their effects. Acknowledging that GPT-Neo's DLA-only protocol provides weaker evidence does not justify these inconsistencies or establish the reliability and comparability of the measurements. Similarly, explaining the minimum attainable Wilcoxon p-value does not address these concerns; my review did not question its correctness.

Other unresolved issues include the justification and robustness of head-selection thresholds, the omission of GPT-2 finite validation, and the choice of pre-output-projection activations as the Jacobian intervention space. Furthermore, task-directed sensitivity ThT_h is introduced as a central metric but is not systematically analyzed alongside the other metrics in the main experiments.

2. Natural activation and behavioral relevance

My original review did not claim that the authors equated local Jacobian sensitivity with naturally realized behavioral importance, or that they claimed to establish the broader functional importance of these heads. I explicitly acknowledged this distinction.

My concern is that the experiments characterize potential sensitivity without examining whether it is realized during natural computation. Precisely because natural activation changes may differ from the constructed Jacobian-gradient directions, I suggested investigating their relationship. Comparing high-gain directions with naturally occurring activation differences would help determine whether these directions are utilized during normal computation. Reporting activation norms does not answer this question, since activation magnitude does not characterize the directions or downstream effects of natural activation changes.

3. Greater-than experiment and functional significance

My review did not claim that the authors interpreted Greater-than alignment as evidence of special Greater-than functionality. I explicitly acknowledged that selected heads exhibit alignment comparable to matched controls and that the experiment establishes task-dependent geometric alignment rather than alternative computational roles.

My criticism is that the experiment provides limited additional behavioral evidence. The paper identifies heads with high vocabulary-wide Jacobian response but weak IOI behavioral relevance, yet does not investigate their actual functional roles. Greater-than does not resolve this either, as it provides alignment measurements without causal evidence of these heads' contributions.

I am not assuming that high-response heads must be important for general language modeling or any particular task. Rather, given their weak behavioral relevance in the examined tasks, I suggest investigating whether and where their high sensitivity has meaningful functional consequences. This motivated my proposal to evaluate these heads through ablation or causal interventions on broader language modeling behavior or other tasks. Acknowledging that Greater-than alignment does not establish functionality does not address this concern.

4. Experimental controls and finite validation

I agree that finite interventions validate Jacobian predictions along the constructed directions. However, this was not my main concern. Since these directions are derived from Jacobian gradients, the experiments primarily validate local sensitivity under deliberately constructed perturbations rather than naturally realized functional importance. Likewise, independent control selection does not resolve questions about the justification of metrics, thresholds, matching criteria, or cross-model configurations.

Overall, I recognize the distinction between broad local sensitivity and task-conditioned behavioral importance and do not attribute unsupported claims to the paper. My concerns are that the experimental methodology remains insufficiently justified and that the results characterize sensitivity differences without investigating their functional or mechanistic significance. I hope this clarifies my original review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.