acceptodds
Under review as a conference paper at ICLR 2027

Where Do Head-Level Adaptation Gains Come From? Design-Space Evidence

Abstract

Head-level adaptation has an attribution gap: complete pipelines are evaluated, but individual stages receive credit. Existing methods typically fine-tune random or all heads, or localize important heads and update them directly, bundling design choices whose contributions remain unresolved. Direct updating further assumes that diagnostic importance implies intervention utility, leaving a consequential choice implicit: *Bridging*, the mapping from localized heads to intervention targets. On Qwen2.5 with GSM8K, adding bridging shifts a localization-guided configuration from 5.9 percentage points below a random-head baseline to 4.8 points above it. To disentangle these contributions, we formulate an Evaluatology-based self-contained adaptation design system \(SCDS\). SCDS represents localization, bridging, and fine-tuning as attribute designed objects \(DOs\), their composition into adaptation paths as a structural DO, and model, dataset, and run-level conditions as confounding objects \(COs\). A blocked factorial study with matched interventions examines 14 localization rules, three bridging rules, and four fine-tuning rules across four model-dataset blocks, alongside random-head and all-head baselines. Across all four blocks, the structural path, bridging, and fine-tuning exhibit significant effects, whereas localization has a small, non-significant main effect. Of the two bridging rules, representation bridging significantly outperforms both Localize-FT and Random-FT in three of four blocks, while information-flow bridging shows no significant difference from the best baseline in the fourth. Bridging can therefore help, but only when the rule matches the block. Generalization tests of the same adapted models on two new datasets broadly corroborate the structural and bridging-rule findings. SCDS thus turns pipeline comparison into evidence of which design choices contribute to adaptation gains and how their effects depend on composition and evaluation conditions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, parameter-efficient fine-tuning, attention heads, localization, head-level adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_byk3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Self-Contained Adaptation Design System (SCDS) to isolate the impact of localization, bridging, and fine-tuning in LLM attention-head adaptation. Experiments on Llama-3.1-8B and Qwen2.5-1.5B across multiple benchmarks reveal that bridging and fine-tuning choices affect performance more than head localization alone, and that optimal configurations are context-dependent.

Critical strengths

The paper addresses an important methodological issue in attention-head adaptation improvements are frequently attributed to head localization even though the final outcome depends jointly on the selected heads, the intervention rule, the update operator, and the optimization configuration. The paper also provides evidence that directly updating diagnostically important heads is not necessarily the best intervention strategy.

Critical weaknesses

The central contribution of the paper is the explicit separation of localization from bridging. However, several aspects of the experimental design make it difficult to determine whether the observed gains are attributable to the proposed bridging principles rather than differences in intervention capacity, additional supervised information, or evaluation-time selection. The methodological contribution in this paper primarily consists of organizing established head-localization and adaptation approaches into a factorial evaluation framework. Hase et al. (NeurIPS 2023) previously demonstrated that causal localization and effective knowledge editing can differ substantially, albeit in a different model-editing setting. Similarly, JoLA (ICML 2025) already jointly optimizes head selection and activation editing, while HeadMap (ICLR 2025) considers layer-conditioned head localization and output remapping. The distinctive contribution of SCDS seems to be the controlled comparative analysis and explicit bridging design space. Also the absence of a consistently effective bridging rule substantially creates limitations. Representation bridging performs significantly worse than direct updating on Llama ARC, and the paper does not provide a validated mechanism for selecting the appropriate rule before observing evaluation accuracy.

Questions for authors
  1. Can the authors evaluate bridging-rule selection without using the final evaluation outcomes?
  2. How are statistical dependencies and evaluation uncertainty handled?
  3. How robust are the central findings under an independent benchmark evaluation protocol and broader adaptation settings?
5 comments
AuthorsOpenReview ↗

Response to Reviewer byk3

Dear Reviewer byk3, thank you for your thoughtful and constructive review. We present our clarifications and additional results below, addressing the attribution of Bridging gains, the distinction from prior work, and evaluation robustness, with the corresponding updates incorporated into the accompanying revised manuscript.

 

W1: Contributions and distinction from prior work

Thank you for highlighting these related works. Our contribution has two complementary parts: introducing Bridging as an explicit adaptation design factor and systematically attributing adaptation gains to the major design choices.

First, Bridging explicitly maps diagnosed heads to intervention targets. Given a head set II identified by Localization, Bridging constructs the set HH through which adaptation is performed, without assuming H=IH=I. We introduce representation and information-flow Bridging rules and examine their effects while holding Localization and Fine-tuning fixed. The distinction from the cited works is as follows:

  • Hase et al. (NeurIPS 2023): MLP-layer localization and factual editing. They show that the MLP layers identified by causal tracing need not be the MLP layers best suited for editing a stored fact. They document this mismatch but do not introduce a rule that maps the localization results to editing targets. Our setting differs in both granularity and objective: we study attention-head sets for task adaptation, rather than MLP layers for factual editing. Their finding provides a conceptual connection, but whether such a mismatch exists in head-level adaptation, and how to bridge it, remain open questions in that work. We address both: we examine the gap between diagnosed heads and effective update targets, introduce representation and information-flow rules to connect them, and isolate the effects of these mappings through matched interventions.

  • JoLA (ICML 2025): identifying operation mismatch within the same selected head set. JoLA states that “head selection criteria do not necessarily align with interventions.” In its LoFIT example, heads are selected using multiplicative parameters, and those same selected heads are subsequently adapted using additive offsets. The cited mismatch concerns the operations used for selection and adaptation, not a difference between the diagnosed and updated head sets. JoLA addresses selection–intervention compatibility through joint optimization. Our Bridging explicitly separates the two sets: with the diagnosed heads II fixed, the intervention targets can be a different set HH. Thus, our contribution concerns which heads should receive updates given a diagnosis, beyond the operations used to score or modify the selected heads.

  • HeadMap (ICLR 2025): modifying the outputs of the same localized heads. HeadMap applies a low-rank transformation to the activations of the localized heads and adds the transformed outputs to the residual stream or attention-block output. The heads used for adaptation are still the same heads identified by localization; the mapping changes their output representations, not the head set. Our Bridging explicitly allows the diagnosed heads and the intervention targets to differ: it uses the diagnosis II to construct an update set HH, rather than only modifying the activations of the same heads.

Our Bridging contribution is therefore substantively distinct from Hase et al., JoLA, and HeadMap: we introduce and evaluate an explicit, independently controllable mapping between diagnosed heads and intervention targets, establishing this mapping as a separate adaptation design factor.

Second, we systematically attribute adaptation gains to the major design factors. Our formulation brings together Localization, Bridging, Fine-tuning, adaptation paths, and relevant training and evaluation conditions. Factorial comparisons and matched interventions quantify their contributions and interactions within this unified system. Our findings show that Bridging and Fine-tuning generally have larger main effects than the choice of Localization rule, and that their interactions also shape adaptation outcomes. We have revised the contribution statement and related-work discussion to make these distinctions explicit (revised Sections 1 and 2, Lines 37–141).

AuthorsOpenReview ↗

Response to Reviewer byk3 (2)

W2 & Q1: Whether the Bridging gains are identified and can the authors evaluate Bridging-rule selection without using the final evaluation outcomes?

Thank you for raising these concerns. We have added capacity controls and validation-based selection experiments, and clarified the supervision used for head selection.

 

(1) Intervention capacity.

Representation Bridging preserves the head budget and keeps ∣H∣=∣I∣|H|=|I|. Table 3 shows significant improvements over both Localize–FT and Random-FT on Llama–GSM8K, Qwen–GSM8K, and Qwen–ARC, with gains over Random-FT of 1.9, 1.6, and 0.3 percentage points, respectively. The additional All-FT comparisons show significant improvements on both GSM8K blocks (+10.5 and +14.6 points), while neither ARC comparison is statistically significant. Furthermore, the matched interventions in Table 4 hold Localization, Fine-tuning, and run conditions fixed: representation Bridging improves over directly updating the localized heads by 1.9, 2.5, and 0.4 points in the same three blocks, with all three 95% confidence intervals excluding zero. These gains therefore cannot be explained by updating more heads.

Information-flow Bridging expands the head set but performs worse than representation Bridging in these three blocks despite updating more heads. We added controls on the remaining block, Llama–ARC, where the original flow Bridging result showed no significant difference from Random-FT. We compare (i) localized KK heads followed by flow Bridging and Fine-tuning, (ii) KK randomly selected seed heads followed by the same flow rule and Fine-tuning, and (iii) Random-FT with 3K3K heads. The two flow pipelines start with the same seed-head budget and expand according to their respective information-flow relationships, so their final head counts can differ. Random-FT with 3K3K heads provides a common reference at an upper bound on the number of heads updated by either flow pipeline.

Contrast on Llama–ARCAccuracy difference (pp)95% CIpp
LBfFLB_fF − RandomBfFRandomB_fF−0.58[−2.79, +1.87]0.602
LBfFLB_fF − Random-FT−1.80[−3.77, +0.25]0.095
RandomBfFRandomB_fF − Random-FT−1.21[−3.37, +0.93]0.275

No statistically significant differences are detected among these three pipelines. Flow Bridging shows no statistically significant difference from either random-seeded flow Bridging or threefold-budget Random-FT, consistent with the original non-significant flow–Random-FT comparison on Llama–ARC.

 

(2) Supervision used for head selection.

Representation Bridging uses nBn_B labeled training examples to select intervention targets HH. Subsequent Fine-tuning uses the same objective, training-sample grid nFn_F, and update operators as the baselines.

To test whether its gains can be explained by supervised head re-selection alone, we added a gradient-magnitude selection control on Qwen–ARC using sampling plan 0. We hold the model, Localization output II, Bridging examples, final head budget K=∣I∣K=\lvert I\rvert, and subsequent Fine-tuning configuration fixed. Using the same task loss and Bridging examples DBD_B, the control computes Gj=∇WO,jL(DB)G_j=\nabla_{W_{O,j}}\mathcal{L}(D_B) for each candidate head and selects Hgrad=TopKj(∥Gj∥F2,K)H_{\mathrm{grad}}=\mathrm{TopK}_j(\lVert G_j\rVert_F^2,K). It uses the same supervised Bridging data without constructing or projecting onto the diagnostic subspace UIU_I.

Representation Bridging exceeds this control by 0.4 percentage points (95% CI [0.1, 0.8], p=0.020p=0.020). With the labeled examples and final head budget matched, gradient-magnitude selection does not recover the performance of representation Bridging. This provides controlled evidence that the proposed diagnosis-guided selection criterion adds value beyond this supervised selection baseline in the evaluated setting. We have clarified the allocation of supervision and included this control in revised Appendix I (Lines 1142–1153), with a summary in revised Section 4.7 (Lines 491–495).

BlockRepresentation − gradient-magnitude selection (pp)95% CIpp
Qwen–ARC+0.4[+0.1, +0.8]0.020
AuthorsOpenReview ↗

Response to Reviewer byk3 (3)

W3 & Q2: How are statistical dependencies and evaluation uncertainty handled?

Thank you for raising this concern. We added two complementary analyses: inference at the sampling-plan level to address dependence across runs, and evaluation on previously unused test examples to assess sensitivity to the evaluation sample.

 

(1) Dependence across experimental runs.

We repeated the analysis using sampling plans, rather than individual runs, as the units of replication. Within each of the four plans, we compute the mean accuracy of each method and take their difference. We then test the resulting four differences against zero using a two-sided Student’s tt test with three degrees of freedom. This analysis accounts for variation across sampling plans without treating runs within the same plan as independent replicates.

On the original test set, the representation-minus-flow contrast retains its direction and remains significant in all four blocks. The original LBmFLB_mF summary also remains significantly above Localize–FT in three blocks, with a positive but non-significant difference on Llama–ARC.

BlockRepresentation − flow (pp)Plan-level ppLBmFLB_mF − LFLF (pp)Plan-level pp
Llama–GSM8K+2.70.008+2.10.003
Llama–ARC−7.5<0.001+0.50.290
Qwen–GSM8K+4.60.010+3.40.014
Qwen–ARC+0.60.014+0.70.006

 

(2) Uncertainty associated with the evaluation sample.

We additionally evaluated the existing representation- and information-flow-Bridging checkpoints on 200 previously unused test examples per task, disjoint from the original 200 examples. The checkpoints and adaptation configurations were held fixed, and the new examples were used only for testing.

The representation-minus-flow difference retains its direction in every block: representation Bridging remains better on both GSM8K blocks and Qwen–ARC, while flow Bridging remains better on Llama–ARC.

BlockOriginal test set: representation − flow (pp)New test set: representation − flow (pp)
Llama–GSM8K+2.72+2.78
Llama–ARC−7.48−5.94
Qwen–GSM8K+4.60+4.17
Qwen–ARC+0.63+1.00

This consistency also holds at the sampling-plan level. For each block and plan, we compute the difference between mean representation-Bridging and mean flow-Bridging accuracy on each test set. All 16 block–plan contrasts retain their signs across the two sets. Their magnitudes and rankings show strong descriptive agreement (Pearson r=0.997r=0.997; Spearman ρ=0.988\rho=0.988).

Thus, the comparative Bridging pattern is reproduced on previously unused test examples, while the plan-level analysis supports its stability across sampling plans. We report both analyses in revised Appendix J (Tables 20–21, Lines 1155–1196) and explicitly distinguish uncertainty across runs or plans, conditional on the evaluated items, from robustness to a fresh test sample.

Reviewer_njghOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Many different model intervention strategies, ranging from intervention location, head selection, and more, have been developed. However, which of these strategies are actually helpful is difficult to identify, as there are often many confounding factors. For example, by changing the learning rate of some attribution method can change which heads are attributed to some change in behavior of a model. This work investigates combinations of methods, locations, and bridges (4, 14, and 3, respectively). The importance of the location and bridge is isolated by performing these settings using all of the heads instead of what the location component suggests, and similarly for the bridge. Across 2 models and 2 datasets, certain design choices are shown to have a large impact on performance, such as learning rate, bridging, and fine-tuning method; others are shown to have a much lower impact, such as localization.

Critical strengths

The results are directly applicable in practice. This method provides coverage of different methods, locations, and bridges across two different model architectures and two different datasets. These experiments identify components that improve performance and, notably, those that do not. This includes head localization, meaning that, at least for the strategies tested, this paper clearly shows that practitioners can focus on the bridging and fine-tuning method instead of localization.

Critical weaknesses

The presentation of the paper could be refined, both in the writing and the notation.

The writing in the abstract and paper indicates that this paper introduces a self-contained design system and provides it with the acronym SCDS, similar to that of a title for a novel method. However, this system is, at its core, just a well-designed set of experiments. It is unclear that this should receive an acronym. Future work should be able to reference the results in this paper without needing a name.

There is also a lot of introduced notation to help keep the footprint of the results small, but the definitions of each symbol are embedded in the text and can be hard to find. For instance, Table 2 uses the notation nBn_B, defined two pages earlier, which is omitted from Table 1 despite that table listing the instantiated values of nBn_B. Some minor inconsistencies in notation are also present, such as BrefB^{ref} and BrefB^{\text{ref}}.

Questions for authors
  1. Can a table of notation be added to the appendix? For instance, each subscript in Table 2 can be identified from the provided caption, but it would be far easier if it were additionally able to be found in a consolidated table.
  2. To my understanding, bridging is a filtering of the selected heads by the localization step. Is this correct? If so, what makes the bridging step different from the localization step? Both filter the set of heads to a smaller subset to then be fine-tuned on.
  3. Can the table and figure captions be on the same side of the table/figure? Some consistency would help with readability.
5 comments
AuthorsOpenReview ↗

Response to Reviewer njgh

Dear Reviewer njgh, thank you for your constructive review and for recognizing the practical value of our findings. Below, we address your concerns about presentation and notation, clarify the distinction between Localization and Bridging, and identify the corresponding changes in the accompanying revised manuscript.

 

W1: Name of the design system

Thank you for this suggestion. Our intention is to formulate head-level adaptation as a self-contained system for design attribution, bringing together the major design factors and their relationships: Localization, Bridging, and Fine-tuning; their composition into adaptation paths; and the training and evaluation conditions under which their effects are assessed. Experiments operate on this system through factorial comparisons, randomized sampling of run conditions, and matched interventions to assess individual contributions and interactions. “SCDS” was intended as shorthand for this integrated system of design factors, rather than the name of a new adaptation algorithm. We agree that a dedicated acronym is unnecessary and have removed “SCDS” from the revised manuscript, including the abstract (Lines 11–34) and Section 3.1 (Lines 149–227).

 

W2: Use of notation

We apologize for the difficulty caused by dispersed definitions and inconsistent notation. To improve readability, we have removed “SCDS,” added a consolidated notation table (revised Appendix A, Table 7, Lines 710–797), included the corresponding symbols alongside their instantiated values in revised Table 1 (Lines 270–278), and standardized notation throughout the text, tables, and figures.

Notation used in the revised manuscript.

Symbol or labelDefinition
DODesigned object: an adaptation design choice whose contribution is assessed.
COConfounding object: a non-target training or evaluation condition considered in design attribution.
M0M_0, M1M_1The base model and the adapted model, respectively.
DSDSDataset defining the task setting.
IIDiagnosed head set returned by Localization.
HHIntervention head set through which adaptation is performed.
YYEvaluation outcome, measured as accuracy.
LLLocalization: identifies the diagnosed head set II. The study considers 14 Localization rules.
BBBridging: maps the diagnosed set II to intervention targets HH.
BidB^{\mathrm{id}}Identity Bridging: H=IH=I, corresponding to directly updating the localized heads.
BrepB^{\mathrm{rep}}Representation Bridging: selects intervention targets by residual-update alignment with the diagnosed representations, with ∣H∣=∣I∣\lvert H\rvert=\lvert I\rvert.
BflowB^{\mathrm{flow}}Information-flow Bridging: constructs HH from the diagnosed heads and selected upstream/downstream neighbors, with I⊆HI\subseteq H.
FFUpdate rule applied through HH: selective head fine-tuning, selective head LoRA, head output bias, or head output remapping.
Head (Head-FT), LoRA, Bias, RemapShort labels for the four update rules, respectively.
SSAdaptation path: specifies which operations are active and how the intervention set HH is constructed.
AFAFAll-FT: adaptation through all heads.
RFRFRandom-FT: adaptation through a randomly selected set of KK heads.
LFLFLocalize–FT: direct adaptation through the diagnosed set, H=IH=I.
LBFLBFLocalize–Bridge–FT: adaptation through H=B(I)H=B(I) using representation or information-flow Bridging.
hhRun-level configuration: h=(nL,nB,nF,k,lr,e,r)h=(n_L,n_B,n_F,k,{lr},e,r), restricted to the coordinates applicable to the selected operations.
nLn_LLocalization sample budget: {64,128,192}\{64,128,192\}, or {300,600,900}\{300,600,900\} for linear probing and LoFiT scale selection.
nBn_BBridging sample budget: {64,128,192}\{64,128,192\}; applicable to representation and information-flow Bridging.
nFn_FSample budget for the update stage: {256,512,768}\{256,512,768\}.
kkLocalization-budget multiplier, k∈{1,2,3}k\in\{1,2,3\}, defining K=k⋅nlayersK=k\cdot n_{\mathrm{layers}}. It does not impose an equal per-layer quota except for HeadMap’s layer-conditioned localization.
KKTotal Localization budget, K=∣I∣K=\lvert I\rvert: 32/64/96 heads for Llama and 28/56/84 for Qwen.
nlayersn_{\mathrm{layers}}Number of layers: 32 for Llama-3.1-8B-Instruct and 28 for Qwen2.5-1.5B-Instruct.
HBHead-budget label in Table 1b. The factor-grid entry 3HB3_{\mathrm{HB}} denotes three budget levels, not a threefold increase in the number of heads.
lr\mathrm{lr}Learning rate: {10−5,10−4}\{10^{-5},10^{-4}\} for Head (Head-FT), LoRA, and Remap.
eeNumber of training epochs: {1,2}\{1,2\} for Head (Head-FT), LoRA, and Remap.
rrLow-rank dimension: {8,16}\{8,16\} for LoRA and Remap. Bias uses none of lr{lr}, ee, or rr.
(ℓ,h)(\ell,h)Layer and head indices. Here, hh is a head index, distinct from the run-configuration vector.
jjCandidate-head index in representation Bridging.
AuthorsOpenReview ↗

Response to Reviewer njgh (2)

Notation used in the revised manuscript (continued).

Symbol or labelDefinition
zℓ,hz_{\ell,h}, zjz_jPre-output-projection activations of head (ℓ,h)(\ell,h) or candidate head jj.
WO(ℓ,h)W_O^{(\ell,h)}, WO,jW_{O,j}Output-projection matrix associated with the indicated head.
oℓ,ho_{\ell,h}Residual write of head (ℓ,h)(\ell,h): oℓ,h=zℓ,hWO(ℓ,h)o_{\ell,h}=z_{\ell,h}W_O^{(\ell,h)}.
oAo_AResidual write of head AA, including seed heads and their neighboring heads.
loss\mathrm{loss}Continuation loss on the Bridging examples.
UIU_IPrincipal subspace estimated from the centered residual writes of the diagnosed heads in II.
Δoj\Delta o_jResidual-update direction for candidate head jj: Δoj=zj(−∇WO,jloss)\Delta o_j=z_j(-\nabla_{W_{O,j}}\mathrm{loss}).
ΠUI\Pi_{U_I}Orthogonal projector onto UIU_I.
AℓA_\ellA diagnosed seed head in layer ℓ\ell under information-flow Bridging.
Aℓ−1A_{\ell-1}, Aℓ+1A_{\ell+1}Selected upstream and downstream neighbors of AℓA_\ell, where such layers exist.
PAP_AOperator that weights a residual write using head AA’s attention pattern.
ηX2\eta_X^2ANOVA effect size for factor XX. The subscript identifies the factor: SS, LL, BB, FF, nLn_L, kk, nBn_B, nFn_F, lr\mathrm{lr}, ee, or rr. The model and sum-of-squares definition are specified for each analysis.
ηmodel2\eta_{\mathrm{model}}^2, ηdataset2\eta_{\mathrm{dataset}}^2Effect sizes for the model and dataset factors in the cross-setting analyses.
ηL×B2\eta_{L\times B}^2, ηL×F2\eta_{L\times F}^2, ηB×F2\eta_{B\times F}^2Interaction effect sizes. Other interaction subscripts similarly identify the factors involved.
pppp-value for the test specified in the accompanying caption or analysis description, such as ANOVA or a bootstrap contrast.
ppPercentage points, the unit used for accuracy differences.
ρ\rhoSpearman rank correlation, used to assess the agreement of rankings.
τ\tauKendall rank correlation, used to compare Localization rankings across blocks; distinct from the matched-intervention estimator below.
djd_jTarget designed object in a matched intervention.
aa, a′a'Two levels of the target object djd_j.
xix_iFixed non-target components of matched configuration ii.
mmNumber of matched configurations.
τ^j(a,a′)\hat{\tau}_j(a,a')Mean paired outcome difference between levels aa and a′a' of djd_j, averaged over the mm matched configurations.
LBrFLB_rFLBFLBF with representation Bridging.
LBfFLB_fFLBFLBF with information-flow Bridging.
LBaFLB_aFMean performance across the representation Bridging and information-flow Bridging variants.
LBmFLB_mFLBFLBF with test-selected Bridging in the original evaluation; LBFLBF with the source-selected rule held fixed in transfer evaluation.

 

Q1: Can a notation table be added to the appendix?

Yes. The table has already been provided in the response to W2 above, and it is included in revised Appendix A (Table 7, Lines 710–797).

 

Q2: Is Bridging a second filter on the heads Localization selected?

Thank you for the question. Bridging connects diagnosis to intervention; it is not a second filtering step over the localized heads. Localization identifies heads II relevant to the behavior being diagnosed, whereas Bridging uses that diagnosis to determine the update targets HH. Directly fine-tuning localized heads sets H=IH=I, but diagnostic importance need not imply intervention utility. Bridging makes this mapping an explicit design choice.

Representation Bridging identifies update targets through representational alignment with the diagnosed heads, while information-flow Bridging uses their information-flow relationships. Both are guided by Localization, but neither is restricted to selecting a subset of II.

Empirically, representation Bridging significantly outperforms both direct updating and random-head adaptation in three of the four blocks. In the remaining block, Llama–ARC, information-flow Bridging shows no significant difference from the best baseline, Random-FT (p=0.148p=0.148; Table 3). These results support distinguishing diagnostic heads from intervention targets, with the benefit of Bridging depending on the rule and setting. We have made this distinction explicit in revised Section 3.1 (Lines 160–192).

 

Q3: Can table and figure captions sit on the same side?

We apologize that the caption layout made the presentation harder to follow. We carefully checked the official ICLR formatting instructions, which state that “The figure number and caption always appear after the figure” and “The table number and title always appear before the table.” While adhering to these requirements, we have improved the layout by rearranging adjacent figures and tables to distinguish them more clearly and keep each caption visually associated with its corresponding item (e.g., revised Tables 2 and 3, Lines 281–302 and 390–405, and Table 11, Lines 1193–1220).

I thank the authors for their detailed response. First, I want to acknowledge the responses to Q1 and Q3. For Q1, the table will significantly improve the parsing of the paper. As for Q3, I was unaware that this is the style choice for ICLR. One remaining point from that point stands, however: placing both captions next to each other is difficult to read. Might it make sense to place all tables above all figures? Some formatting could go a long way towards readability.

My greater concern stems from W1. The paper provides a set of experiments that test different design choices in model adaptation. It identifies certain components in this design space that have a significant impact on downstream performance. However, this paper and abstract read as though this experimental design is novel and that this system uncovers the different components that affect downstream performance. The end of the abstract is as it was before the revision: "This system turns pipeline comparison into evidence of which design choices contribute to adaptation gains and how their effects depend on composition and evaluation conditions." It's the choice of "system" that reads as misleading. Additionally, the abstract reenforces that this a "design system" earlier as well. It is the framing of creating a new system that reads poorly, when, again, this is mostly a comprehensive group of experiments. This work has interesting insights, but should be reframed accordingly.

As a result, I maintain my weak recommendation of rejection.

Reviewer_6RDTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study which design choice matters most in head-level LLM fine-tuning by testing combinations of 14 head-localization rules, 3 target-mapping ("bridging") rules, and 4 update rules across four pipeline types on Llama-3.1-8B-Instruct and Qwen2.5-1.5B-Instruct for GSM8K and ARC-Challenge.

Critical strengths
  • The factorial experiment is clean and useful because testing combinations across four model-dataset pairs isolates each stage rather than tuning one preferred pipeline.
  • The null result in Tab. 3 and Tab. 10 is interesting: localized heads perform no better than random heads (LF vs. RF), questioning a common assumption in head-selection work.
Critical weaknesses
  • Poor writing, missing formulas, and sloppy editing. Calling basic experimental conditions "confounding objects" (and then adding in the very next sentence that they are not confounders) plus introducing eight pipeline acronyms (AF to LBfF) makes the ablation study unclear. Meanwhile, the authors give zero formulas for all 14 localization rules and all 4 update rules, never specify how head output bias is trained despite being by far the slowest method on GSM8K (Tab. 13), and never explain how shared GQA key-value heads are updated. They also include an uncited duplicate of Tab. 2 as Tab. 7 and end with a two-sentence conclusion and no limitations section.
  • Missing base-model and all-head baselines in key tables. Every pipeline in Eq. 2 starts from the pretrained model M_0, yet they never report M_0 accuracy anywhere. Llama-3.1-8B is normally strong on ARC-Challenge, whereas here its adapted average is lower than Qwen2.5-1.5B (and one Llama-ARC setup in Tab. 15 averages below 50%), so we cannot tell whether adaptation improves M_0 or just hurts it less. They also omit All-FT from Tab. 3 and Tab. 10, despite All-FT outperforming every head-subset method on Qwen-ARC to OpenBookQA in Fig. 11.
  • Unfair head budgets and test-set rule selection in bridging. In Eq. 1, B^rep picks heads whose one-step task gradient aligns with the seed subspace, acting as a gradient-based selector with no random-seed control, while B^flow adds neighbors and updates up to three times as many heads as LF or RF at the same k. Worse, LBmF picks whichever bridging rule scores higher on the evaluation set itself (Tab. 3, Tab. 4), turning flat or negative single-rule results on Llama-ARC into a positive gain.
  • Per-layer head budget may cause the localization null. Fixing the same head count k in every layer (Tab. 1) differs from LoFiT [1] (one of their 14 rules), which selects top heads globally across all layers. Forcing equal heads per layer makes localized sets much closer to random sampling and could by itself explain why LF matches RF.
  • Separate one-way ANOVAs weaken the main test. Running separate one-way ANOVAs per factor in Tab. 2 instead of the joint L × B × F model from Section 3.2.1 leaves large update-rule and learning-rate variance in the residual error when testing localization (in Fig. 9, full head tuning drops sharply at the higher learning rate on GSM8K but improves on ARC). [1]. LoFiT: Localized Fine-Tuning on LLM Representations. Fangcong Yin, et. al. NeurIPS 2024.
Questions for authors
  1. What is the unadapted M_0 accuracy on each task, and why is All-FT omitted from Tab. 3 and Tab. 10 despite ranking first on Qwen-ARC to OpenBookQA in Fig. 11?

  2. How does bridging perform when seeded with random heads at a matched total head count, and how does LBmF perform when the rule is chosen on a validation split?

  3. How do update rules handle shared GQA key-value heads, how is head output bias trained without lr or epochs, and do localization p-values change under a joint multi-way ANOVA or global head selection?

10 comments
AuthorsOpenReview ↗

Response to Reviewer 6RDT

Dear Reviewer 6RDT, thank you for your careful review and constructive feedback. We appreciate your recognition of the value of our factorial analysis and its implications for head selection. We present our clarifications and additional results below, together with the corresponding changes in the accompanying revised manuscript.

W1 & Q3: Writing, missing formulas, conclusion, and limitations

 

Confounding objects.

We apologize for the misleading wording. In our Evaluatology-based formulation, “confounding objects” are non-target conditions that influence the outcome YY alongside the design choice XX, without requiring that they also influence XX. The subsequent qualification refers to the common-cause notion of confounding in structural causal models (SCMs), where CC affects both XX and YY (X←C→YX \leftarrow C \rightarrow Y). The two statements distinguish different meanings of “confounding,” rather than make contradictory claims within the same framework. We have revised the passage to state this distinction explicitly (revised Section 3.1, Lines 149–154).

 

Eight pipeline acronyms make the ablation study unclear.

Thank you for highlighting this presentation issue. The eight abbreviations describe three levels of a single ablation design: four adaptation paths—All-FT (AFAF), Random-FT (RFRF), Localize–FT (LFLF), and Localize–Bridge–FT (LBFLBF); two LBFLBF variants—representation Bridging (LBrFLB_rF) and information-flow Bridging (LBfFLB_fF); and two summaries—the average across these variants (LBaFLB_aF) and the result of selecting the variant with higher mean test accuracy within each Localization–Fine-tuning pair (LBmFLB_mF).

Each comparison answers a specific ablation question. LFLF versus RFRF tests whether diagnosis-guided head selection improves upon budget-matched random selection; LBFLBF versus LFLF tests whether mapping diagnosed heads to intervention targets improves upon updating them directly. LBFLBF versus RFRF evaluates whether the combined Localization–Bridging pipeline improves upon random-head adaptation. AFAF provides the all-head adaptation reference. Within each model–dataset block, the Localization-guided design enumerates all 14×3×414\times3\times4 combinations of Localization, Bridging, and Fine-tuning across four sampling plans, with run conditions sampled within each cell. Identity Bridging directly updates the diagnosed heads and therefore recovers LFLF. This factorial coverage allows us to assess each operation across alternative choices of the other operations and examine their interactions. Additional matched comparisons vary the Bridging rule while holding Localization, Fine-tuning, and the shared run conditions fixed. To make this ablation logic explicit, we have added a consolidated notation table and clarified the ablation question addressed by each principal comparison in revised Appendix A (Table 7, Lines 702–797), with a pointer in revised Section 3.1 (Lines 211–213).

 

How shared GQA key-value heads are updated

The shared key and value projections, WKW_K and WVW_V, remain frozen in all reported experiments. In GQA, multiple query heads share the same key–value head. Updating its K/V projections would therefore change the key–value representations used by every query head in that group, including unselected heads. We freeze these shared projections to preserve the intended scope of head selection. Specifically:

  • Selective head fine-tuning updates only the WQW_Q and WOW_O slices corresponding to the selected query heads.
  • Selective head LoRA freezes the base weights and restricts the effective low-rank updates to those same Q/O slices.
  • Head output bias (ITI) adds a shift to the selected query heads’ activations before WOW_O, with the base weights frozen.
  • Head output remapping (HeadMap) trains a low-rank mapping from the selected heads’ pre-WOW_O activations and adds its output as a residual contribution, with the base weights frozen.

Thus, none of the four update rules modifies the shared GQA key–value projections.

AuthorsOpenReview ↗

Response to Reviewer 6RDT (2)

How is head output bias trained

Head output bias. Localization and Fine-tuning each have an extensive literature, which we organize into 14 and 4 categories, respectively, corresponding to the levels of LL and FF in the factorial. Crossing every proposed method with the remaining design factors would lead to a combinatorial explosion. We therefore shortlist methods within each category by jointly considering publication year, citation count, and venue, then randomly select one method from each shortlist. For the head output bias category, the selected method is Inference-Time Intervention (ITI; Li et al., NeurIPS 2023) [1]. ITI estimates an additive bias from labeled head activations. For each selected head hh, it applies zh←zh+ασhθhz_h \leftarrow z_h + \alpha \sigma_h \theta_h, where θh\theta_h is the normalized difference between positive- and negative-class mean activations, and σh\sigma_h is the standard deviation of activations projected onto that direction. We select the intervention strength α\alpha on held-out calibration examples, disjoint from both the data used to estimate θh\theta_h and σh\sigma_h and the final test set. This procedure requires no gradient-based optimization or low-rank parameterization, so learning rate lrlr, epochs ee, and rank rr are inapplicable, as indicated in Table 1. Although gradient-free, it still requires activation collection and repeated model evaluations during calibration. Full implementation details are available in our anonymous code repository: https://anonymous.4open.science/r/LBF-F361/.

Runtime of head output bias on GSM8K. ITI selects its intervention strength through full greedy decoding on held-out calibration examples, repeated across candidate strengths. Head-FT, LoRA, and Remap instead use teacher-forced supervised fine-tuning, which processes target tokens in parallel within each sequence. We use CoT generation for GSM8K and direct answer-option generation for ARC. This task-dependent use of CoT has precedent in Hase et al. (2024) [2], who use CoT on GSM8K and non-CoT evaluation on ARC. Sprague et al. (2025) [3] provide broader evidence that CoT benefits are concentrated in mathematical and symbolic reasoning. The longer GSM8K responses make ITI’s repeated autoregressive calibration substantially more expensive, despite requiring no gradient-based training. Table 14 includes this calibration cost in the adaptation time; final test evaluation is excluded.

Reference [1] Kenneth Li, et al. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. 37th Conference on Neural Information Processing Systems (NeurIPS 2023). [2] Peter Hase, et al. The Unreasonable Effectiveness of Easy Training Data for Hard Tasks. ACL, 2024. [3] Zayne Sprague, et al. To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.

 

Formulas for Localization rules

We have added pseudocode for all 14 Localization rules and all 4 Fine-tuning rules in revised Appendices C.1 (Lines 1367–1541) and C.2 (Lines 1566–1635), respectively, and provide the full implementations at the anonymous link: https://anonymous.4open.science/r/LBF-F361/. Below we give an illustrative example.

Localization. Each procedure is the sweep default of the named module. DD is the Localization sample. (ℓ,h)(\ell,h) indexes the attention head at layer ℓ\ell, head index hh, and s(ℓ,h)s(\ell,h) is that head's score. zz is the pre-WOW_O head state. TopK(s)\mathrm{TopK}(s) keeps the K=k⋅nlayersK=k\cdot n_{\mathrm{layers}} largest scores and BotK(s)\mathrm{BotK}(s) the smallest; ties break by (layer,head)(\mathrm{layer},\mathrm{head}). Llama uses K∈{32,64,96}K\in\{32,64,96\} and Qwen K∈{28,56,84}K\in\{28,56,84\}.


Algorithm L1. Path patching


  1. function PathPatching(D,K)\mathrm{PathPatching}(D, K)
  2.  for each reference/counterfactual pair do
  3.   cache clean states zrz^{r} and counterfactual states zcz^{c}
  4.   ℓo←\ell_o \leftarrow sum of gold-answer logits on the reference
  5.   for each head (ℓ,h)(\ell, h) do
  6.    freeze every head state to zrz^{r}
  7.    at the last prompt token, write zcz^{c} into head (ℓ,h)(\ell, h)
  8.    recompute the residual and the later MLPs
  9.    ℓp←\ell_p \leftarrow gold-answer logit sum on this patched run
  10.    s(ℓ,h)←s(ℓ,h)+(ℓp−ℓo)/ℓos(\ell, h) \leftarrow s(\ell, h) + (\ell_p - \ell_o)/\ell_o
  11.   end for
  12.  end for
  13.  return BotK(mean s)\mathrm{BotK}(\mathrm{mean}\, s)
  14. end function

zrz^{r} and zcz^{c} are the cached states on the reference and the counterfactual. ℓo\ell_o and ℓp\ell_p are gold-answer logit sums on the reference and on the patched run.

AuthorsOpenReview ↗

Response to Reviewer 6RDT (3)

Formulas for Fine-tuning rules

Fine-tuning. All four rules edit the same HH. DFD_F is the Fine-tuning sample. The three gradient rules use AdamW, task cross-entropy, a linear schedule, and a 3% warmup. Their learning rate is in {10−5,10−4}\{10^{-5},10^{-4}\} and their epoch count is in {1,2}\{1,2\}.


Algorithm F1. Selective head fine-tuning


  1. function SelectiveHeadFT(H,DF)\mathrm{SelectiveHeadFT}(H, D_F)
  2.  freeze every parameter
  3.  for each head (ℓ,h)∈H(\ell, h)\in H do
  4.   unfreeze the head-hh blocks of the default attention matrices at layer ℓ\ell
  5.  end for
  6.  for each minibatch from DFD_F do
  7.   compute task cross-entropy
  8.   AdamW-update only the unfrozen blocks
  9.  end for
  10. end function

Default matrices: multi-head attention uses WQ,WK,WV,WOW_Q,W_K,W_V,W_O; grouped-query / multi-query attention uses WQ,WOW_Q,W_O only, leaving the shared WK,WVW_K,W_V frozen.

 

Tables 2 and 7.

We apologize for the redundancy. Table 2 provides a consolidated summary of the ANOVA results, including the structural-path comparison, whereas Table 7 was intended to focus on object-level effects within the factorial design runs. However, its results duplicate the corresponding entries in Table 2. We have therefore removed the original duplicate Table 7 and retained Table 2 as the consolidated presentation referenced in revised Section 4.2 (Lines 324–348).

 

Conclusion and limitations.

We apologize for the overly brief conclusion and the lack of a dedicated limitations section. Under the page limit, we compressed the conclusion and left the relevant qualifications scattered across the experimental sections. We have expanded Section 5, retitled it “Conclusion and Discussion,” and consolidated the relevant qualifications in its clearly labeled “Limitation” paragraph (revised manuscript, Lines 522–539), as follows.

Conclusion. We develop a self-contained system for design attribution in head-level adaptation, integrating Localization, Bridging, Fine-tuning, adaptation paths, and relevant training and evaluation conditions. Within this system, we introduce Bridging as an explicit design factor connecting diagnosed heads to intervention targets. Factorial analysis and matched interventions characterize the contributions of these choices and their interactions. Bridging and Fine-tuning generally exhibit larger main effects than Localization, while Localization’s statistical significance depends on the setting and statistical model specification. Representation Bridging significantly outperforms direct updating and random-head adaptation in three of the four original settings. These findings highlight the importance of jointly evaluating diagnostic head selection, intervention targets, and update rules. Additional evaluations support the robustness of the comparative findings, and validation-based experiments demonstrate that Bridging rules can be selected before test evaluation.

Limitations. Our study evaluates a finite selection of models, tasks, method instantiations, and adaptation budgets, with hyperparameter configurations sampled within these settings. The Localization rules span several methodological families but do not exhaust the available approaches. The estimated effects, including the Bridging effects assessed through matched interventions, are conditional on the evaluated configurations. Broader coverage of models, tasks, methods, and configurations is needed to assess how these effects generalize beyond the settings examined.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.