- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the Self-Contained Adaptation Design System (SCDS) to isolate the impact of localization, bridging, and fine-tuning in LLM attention-head adaptation. Experiments on Llama-3.1-8B and Qwen2.5-1.5B across multiple benchmarks reveal that bridging and fine-tuning choices affect performance more than head localization alone, and that optimal configurations are context-dependent.
Critical strengths
The paper addresses an important methodological issue in attention-head adaptation improvements are frequently attributed to head localization even though the final outcome depends jointly on the selected heads, the intervention rule, the update operator, and the optimization configuration. The paper also provides evidence that directly updating diagnostically important heads is not necessarily the best intervention strategy.
Critical weaknesses
The central contribution of the paper is the explicit separation of localization from bridging. However, several aspects of the experimental design make it difficult to determine whether the observed gains are attributable to the proposed bridging principles rather than differences in intervention capacity, additional supervised information, or evaluation-time selection. The methodological contribution in this paper primarily consists of organizing established head-localization and adaptation approaches into a factorial evaluation framework. Hase et al. (NeurIPS 2023) previously demonstrated that causal localization and effective knowledge editing can differ substantially, albeit in a different model-editing setting. Similarly, JoLA (ICML 2025) already jointly optimizes head selection and activation editing, while HeadMap (ICLR 2025) considers layer-conditioned head localization and output remapping. The distinctive contribution of SCDS seems to be the controlled comparative analysis and explicit bridging design space. Also the absence of a consistently effective bridging rule substantially creates limitations. Representation bridging performs significantly worse than direct updating on Llama ARC, and the paper does not provide a validated mechanism for selecting the appropriate rule before observing evaluation accuracy.
Questions for authors
- Can the authors evaluate bridging-rule selection without using the final evaluation outcomes?
- How are statistical dependencies and evaluation uncertainty handled?
- How robust are the central findings under an independent benchmark evaluation protocol and broader adaptation settings?
5 comments
Response to Reviewer byk3 (2)
W2 & Q1: Whether the Bridging gains are identified and can the authors evaluate Bridging-rule selection without using the final evaluation outcomes?
Thank you for raising these concerns. We have added capacity controls and validation-based selection experiments, and clarified the supervision used for head selection.
(1) Intervention capacity.
Representation Bridging preserves the head budget and keeps . Table 3 shows significant improvements over both Localize–FT and Random-FT on Llama–GSM8K, Qwen–GSM8K, and Qwen–ARC, with gains over Random-FT of 1.9, 1.6, and 0.3 percentage points, respectively. The additional All-FT comparisons show significant improvements on both GSM8K blocks (+10.5 and +14.6 points), while neither ARC comparison is statistically significant. Furthermore, the matched interventions in Table 4 hold Localization, Fine-tuning, and run conditions fixed: representation Bridging improves over directly updating the localized heads by 1.9, 2.5, and 0.4 points in the same three blocks, with all three 95% confidence intervals excluding zero. These gains therefore cannot be explained by updating more heads.
Information-flow Bridging expands the head set but performs worse than representation Bridging in these three blocks despite updating more heads. We added controls on the remaining block, Llama–ARC, where the original flow Bridging result showed no significant difference from Random-FT. We compare (i) localized heads followed by flow Bridging and Fine-tuning, (ii) randomly selected seed heads followed by the same flow rule and Fine-tuning, and (iii) Random-FT with heads. The two flow pipelines start with the same seed-head budget and expand according to their respective information-flow relationships, so their final head counts can differ. Random-FT with heads provides a common reference at an upper bound on the number of heads updated by either flow pipeline.
| Contrast on Llama–ARC | Accuracy difference (pp) | 95% CI | |
|---|---|---|---|
| − | −0.58 | [−2.79, +1.87] | 0.602 |
| − Random-FT | −1.80 | [−3.77, +0.25] | 0.095 |
| − Random-FT | −1.21 | [−3.37, +0.93] | 0.275 |
No statistically significant differences are detected among these three pipelines. Flow Bridging shows no statistically significant difference from either random-seeded flow Bridging or threefold-budget Random-FT, consistent with the original non-significant flow–Random-FT comparison on Llama–ARC.
(2) Supervision used for head selection.
Representation Bridging uses labeled training examples to select intervention targets . Subsequent Fine-tuning uses the same objective, training-sample grid , and update operators as the baselines.
To test whether its gains can be explained by supervised head re-selection alone, we added a gradient-magnitude selection control on Qwen–ARC using sampling plan 0. We hold the model, Localization output , Bridging examples, final head budget , and subsequent Fine-tuning configuration fixed. Using the same task loss and Bridging examples , the control computes for each candidate head and selects . It uses the same supervised Bridging data without constructing or projecting onto the diagnostic subspace .
Representation Bridging exceeds this control by 0.4 percentage points (95% CI [0.1, 0.8], ). With the labeled examples and final head budget matched, gradient-magnitude selection does not recover the performance of representation Bridging. This provides controlled evidence that the proposed diagnosis-guided selection criterion adds value beyond this supervised selection baseline in the evaluated setting. We have clarified the allocation of supervision and included this control in revised Appendix I (Lines 1142–1153), with a summary in revised Section 4.7 (Lines 491–495).
| Block | Representation − gradient-magnitude selection (pp) | 95% CI | |
|---|---|---|---|
| Qwen–ARC | +0.4 | [+0.1, +0.8] | 0.020 |
Response to Reviewer byk3 (3)
W3 & Q2: How are statistical dependencies and evaluation uncertainty handled?
Thank you for raising this concern. We added two complementary analyses: inference at the sampling-plan level to address dependence across runs, and evaluation on previously unused test examples to assess sensitivity to the evaluation sample.
(1) Dependence across experimental runs.
We repeated the analysis using sampling plans, rather than individual runs, as the units of replication. Within each of the four plans, we compute the mean accuracy of each method and take their difference. We then test the resulting four differences against zero using a two-sided Student’s test with three degrees of freedom. This analysis accounts for variation across sampling plans without treating runs within the same plan as independent replicates.
On the original test set, the representation-minus-flow contrast retains its direction and remains significant in all four blocks. The original summary also remains significantly above Localize–FT in three blocks, with a positive but non-significant difference on Llama–ARC.
| Block | Representation − flow (pp) | Plan-level | − (pp) | Plan-level |
|---|---|---|---|---|
| Llama–GSM8K | +2.7 | 0.008 | +2.1 | 0.003 |
| Llama–ARC | −7.5 | <0.001 | +0.5 | 0.290 |
| Qwen–GSM8K | +4.6 | 0.010 | +3.4 | 0.014 |
| Qwen–ARC | +0.6 | 0.014 | +0.7 | 0.006 |
(2) Uncertainty associated with the evaluation sample.
We additionally evaluated the existing representation- and information-flow-Bridging checkpoints on 200 previously unused test examples per task, disjoint from the original 200 examples. The checkpoints and adaptation configurations were held fixed, and the new examples were used only for testing.
The representation-minus-flow difference retains its direction in every block: representation Bridging remains better on both GSM8K blocks and Qwen–ARC, while flow Bridging remains better on Llama–ARC.
| Block | Original test set: representation − flow (pp) | New test set: representation − flow (pp) |
|---|---|---|
| Llama–GSM8K | +2.72 | +2.78 |
| Llama–ARC | −7.48 | −5.94 |
| Qwen–GSM8K | +4.60 | +4.17 |
| Qwen–ARC | +0.63 | +1.00 |
This consistency also holds at the sampling-plan level. For each block and plan, we compute the difference between mean representation-Bridging and mean flow-Bridging accuracy on each test set. All 16 block–plan contrasts retain their signs across the two sets. Their magnitudes and rankings show strong descriptive agreement (Pearson ; Spearman ).
Thus, the comparative Bridging pattern is reproduced on previously unused test examples, while the plan-level analysis supports its stability across sampling plans. We report both analyses in revised Appendix J (Tables 20–21, Lines 1155–1196) and explicitly distinguish uncertainty across runs or plans, conditional on the evaluated items, from robustness to a fresh test sample.
Response to Reviewer byk3
Dear Reviewer byk3, thank you for your thoughtful and constructive review. We present our clarifications and additional results below, addressing the attribution of Bridging gains, the distinction from prior work, and evaluation robustness, with the corresponding updates incorporated into the accompanying revised manuscript.
W1: Contributions and distinction from prior work
Thank you for highlighting these related works. Our contribution has two complementary parts: introducing Bridging as an explicit adaptation design factor and systematically attributing adaptation gains to the major design choices.
First, Bridging explicitly maps diagnosed heads to intervention targets. Given a head set I identified by Localization, Bridging constructs the set H through which adaptation is performed, without assuming H=I. We introduce representation and information-flow Bridging rules and examine their effects while holding Localization and Fine-tuning fixed. The distinction from the cited works is as follows:
Hase et al. (NeurIPS 2023): MLP-layer localization and factual editing. They show that the MLP layers identified by causal tracing need not be the MLP layers best suited for editing a stored fact. They document this mismatch but do not introduce a rule that maps the localization results to editing targets. Our setting differs in both granularity and objective: we study attention-head sets for task adaptation, rather than MLP layers for factual editing. Their finding provides a conceptual connection, but whether such a mismatch exists in head-level adaptation, and how to bridge it, remain open questions in that work. We address both: we examine the gap between diagnosed heads and effective update targets, introduce representation and information-flow rules to connect them, and isolate the effects of these mappings through matched interventions.
JoLA (ICML 2025): identifying operation mismatch within the same selected head set. JoLA states that “head selection criteria do not necessarily align with interventions.” In its LoFIT example, heads are selected using multiplicative parameters, and those same selected heads are subsequently adapted using additive offsets. The cited mismatch concerns the operations used for selection and adaptation, not a difference between the diagnosed and updated head sets. JoLA addresses selection–intervention compatibility through joint optimization. Our Bridging explicitly separates the two sets: with the diagnosed heads I fixed, the intervention targets can be a different set H. Thus, our contribution concerns which heads should receive updates given a diagnosis, beyond the operations used to score or modify the selected heads.
HeadMap (ICLR 2025): modifying the outputs of the same localized heads. HeadMap applies a low-rank transformation to the activations of the localized heads and adds the transformed outputs to the residual stream or attention-block output. The heads used for adaptation are still the same heads identified by localization; the mapping changes their output representations, not the head set. Our Bridging explicitly allows the diagnosed heads and the intervention targets to differ: it uses the diagnosis I to construct an update set H, rather than only modifying the activations of the same heads.
Our Bridging contribution is therefore substantively distinct from Hase et al., JoLA, and HeadMap: we introduce and evaluate an explicit, independently controllable mapping between diagnosed heads and intervention targets, establishing this mapping as a separate adaptation design factor.
Second, we systematically attribute adaptation gains to the major design factors. Our formulation brings together Localization, Bridging, Fine-tuning, adaptation paths, and relevant training and evaluation conditions. Factorial comparisons and matched interventions quantify their contributions and interactions within this unified system. Our findings show that Bridging and Fine-tuning generally have larger main effects than the choice of Localization rule, and that their interactions also shape adaptation outcomes. We have revised the contribution statement and related-work discussion to make these distinctions explicit (revised Sections 1 and 2, Lines 37–141).