- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors study to what extent localizers, methods that identify where, e.g. in which layers, in a model a desired task performance is located. This is motivated by the need to decide where parameter-efficient fine-tuning should be placed for optimization. The authors propose a benchmark that tests to what extent localizers are able to uncover the best placement of a parameter update. This benchmark is then used for a study across seven localizers, three model families and four domains (code, Json, Q&A, regex).
Critical strengths
- The reframing of capability-drift adaptation as a benchmark question is relevant and interesting.
- The definition of appropriate baselines (random, uniform) is appreciated and grounds the overall analysis.
- The empirical study covers a reasonable range of models, tasks and localizers.
Critical weaknesses
-
The study assumes that localization under the investigated tasks is linked to an identifiable location that is optimal for parameter-efficient finetuning, but localization as identified by localization methods do not necessarily correspond to where the parameter updates should appear. Localizers may instead correspond to layers where relevant computations happen, but not where they originate.
-
The paper’s results and methods are at times difficult to parse. The authors use condensed language and jargon to describe simple results, e.g. “Headroom is ample (H = 33.3pp, z= 8.8, R= 11.7: whole-network reaches 0.580 over the 0.247 no-T2 floor, n= 300), so the flatness is informative.“ (l.246 - 247). Some language like “flat grid” vs “peaked grid”, (e.g “The code grid is flat” (l. 244)) are not sufficiently well introduced, making it difficult to understand what precisely is meant. See also questions below.
-
Presentation of results overall is not clear, e.g. Figure 1, rightmost panel: What does the hatching pattern on the rightmost panel indicate? It may refer to “QA panel grayed, ungraded” in the caption, but it is easily confused with “Gray shading = hook-reachable zone”.
-
Gradient localization (AtP) has been shown to be a noisy approximation [Jaf25], which can be robustified through application of selected propagation rules (see [Jaf25]). These have been shown to support downstream applications like circuit localization [Aro26a] and automated circuit description [Aro26b], and thus would be a relevant data point to stress-test the benchmark on a state-of-the-art localization method.
References - [Jaf25] FR Jafari et al. RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patching, Mechanistic Interpretability Workshop, NeurIPS 2025 - [Aro26b] Arora, A., et al. Language model circuits are sparse in the neuron basis. arXiv preprint arXiv:2601.22594 (2026). - [Aro26b] Arora, A., et al. . ADAG: Automatically describing attribution graphs. COLM 2026
Questions for authors
- What is “an offset ladder across depth”? (l. 154)
- What does “hook-reachable zone” mean, why is the hook not always “reachable”? (see caption Fig. 1)
4 comments
Response to Reviewer Jdea (2/2)
Response to Reviewer Jdea: Part 2 of 2
W4. RelP as a stronger localization method. We added RelP (Rezaei Jafari et al., 2025) to the complete 34-window study in Section 4.8, Figure 2 and Appendix B. AtP, RelP and exact activation patching use the same ten development pairs, continuation target and attention-output locations across all 42 layers. We adapted the authors' TransformerLens fork for the merged bf16 Gemma-2 9B, using the LN, Identity and Half propagation rules without AH. This is an adapted implementation, not an unmodified reproduction of their pipeline; Appendix B defines these choices and distinguishes attribution from adaptation.
RelP selects layers 33-41 (16/40), whereas AtP and exact patching select 20-28 (22/40). RelP minus AtP is -15.0 percentage points, with task-level 95% interval [-25.0, -5.0] and exact McNemar p = 0.0703. Accounting for the eight construction groups gives interval [-35.0, +2.5] and sign-flip p = 0.375. The direction favors AtP, but the exact test does not establish superiority. The secondary reference negative log-likelihood, which measures how much probability the model assigns to the reference answer, favors RelP instead: difference -0.00556 (lower is better), task interval [-0.00877, -0.00229], construction interval [-0.02191, 0.01042], sign-flip p = 0.539. Thus the behavioral and likelihood endpoints need not agree.
We also report how closely the methods match exact patching. On the ten development pairs, mean per-pair Pearson correlations of attention-output layer profiles are 0.838 for AtP and 0.651 for RelP's forward-exact variant. Here exact patching inserts clean activations into the counterfactual run. The variant preserves the ordinary activation's bf16 forward value while retaining the Identity rule's backward coefficient; Appendix B defines it. These are attribution-agreement measures, not correlations with adaptation accuracy.
Leaving out individual development pairs can move AtP to RelP's window and the primary RelP selector to layers 23-31; the forward-exact variant retains 33-41 in these checks. We report these sensitivities rather than claiming RelP is generally worse. The study directly tests whether its circuit-discovery improvements translate to selecting useful update locations, without assuming that they must.
Q1. "Offset ladder across depth". We removed the term. Section 3.1 describes a separate five-layer window sweep with centers {14, 18, 20, 22, 26, 30, 34} and a smaller parameter budget, not extra points on the nine-layer curve.
Q2. "Hook-reachable zone". We replaced the phrase with an explanation of where activations are captured. A hook is that capture point. In the original SAE and AtP+GradDrop configurations it is at residual layer 20, so later layers cannot contribute to that captured activation. Averaging five distinct indices from layers 0-20 cannot produce a center above 18; the band-end rule cannot name a center above 22. These limits apply to the readout, not the trainable layers. Sections 3.2-3.3, Table 1 and Figure 1 now explain this, and the all-layer hook sweep tests capture points 0-41.
Reply
I appreciate the provided rebuttal, additional evaluations on RelP and clarifications. As also noted by the other reviewers, the paper suffers from limited readability and lacks a sufficiently strong motivation for its main claim. Unfortunately, these issues cannot be adequately addressed within the scope of the rebuttal. I will maintain my score and continue to follow the discussion.
Thank you for considering the additional evaluations and clarifications. We would like to clarify that the revised manuscript itself, beyond the rebuttal comments, rewrites the motivation in the Introduction, replaces the quoted headroom passage with numeric accuracy comparisons, and presents the new evaluations in Sections 4.8–4.9. Could you clarify what remains insufficient in the revised motivation: the importance of testing whether localization guides adaptation, or the contribution established by our results? This would help us understand the remaining objection without repeating the technical rebuttal.
Response to Reviewer Jdea (1/2)
Thank you for the concrete suggestions, including the RelP stress test. We have made the computation-versus-update distinction the organizing question and added the requested comparison. The common response summarizes the two new studies.
W1. Computation location need not identify update location. We agree; the revision tests this link rather than assuming it. The new adaptation-oriented selection experiment provides a positive case: target-calibrated selection improves PIQA over the random mean, and target-domain calibration beats cross-domain transfer on both PIQA and ARC-Challenge. The latter need not imply a random-baseline advantage, as ARC-Challenge illustrates.
The complete-window study separately tests attribution agreement and update performance. Exact activation patching selects the same window as AtP (layers 20-28, 22/40), showing that this choice does not depend on the gradient approximation. But changing the measured activations from attention outputs to MLP outputs or residual inputs moves all three methods to layers 33-41 (16/40). The choice of what to measure is therefore part of the placement recommendation. Three no-signal controls at the same parameter count score 17-18/40 and are not statistically separated from either attribution choice. Together, these results support evaluating the signal, its layer-selection rule and the task, rather than assuming computation location determines useful update location.
W2. Condensed language: H, z, R and "flat" / "peaked". We rewrote the sentence you quoted in terms of the observed accuracy range, all-layer reference, accuracy before adaptation to the shifted task and sample size. Section 3.4 and the glossary define H, z and R as summaries of adaptation gain and uncertainty; neither ratio is presented as a paired test or a bound on other placements. Main result paragraphs give numeric ranges instead of relying on "flat" or "peaked". Where these short labels remain in the landscape-summary table (Table 3) and appendix matrix (Table 5), the main caption explains that they describe the tested curves, not statistical equivalence or a global optimum.
W3. Figure 1 hatching and shading. The rewritten caption distinguishes them. Full-panel hatching means QA is not graded. The two grey bands show which windows the layer-20 measurements can recommend: the darker band ends at center 18 for the SAE top-five readout; the lighter band ends at center 22 for the AtP+GradDrop rule based on the end of a high-score band. They restrict what the measurement can recommend, not which layers can be trained. Error bars are marginal binomial standard errors, not uncertainty in paired differences, and do not account for repeated records or training/evaluation overlap.