acceptodds
Under review as a conference paper at ICLR 2027

Grafts Add Up: Scoring Every Layer Swap in One Forward–Backward Pass

Abstract

Layer swapping combines two models fine-tuned from the same base by replacing some layers of one model with layers of the other. Choosing which layers to swap usually requires building and testing many merged models. We study whether the effect of each donor layer can instead be scored on the host model's own trajectory. The scores add across swapped layers; the error is second order when both experts stay near their shared base. SUTURE uses these scores to rank layer windows from one forward and backward pass per probe, then uses a fixed number of built models to calibrate a language constraint and check its choice. Controlled stacks support the approximation for small expert changes and show where its ranking weakens. On public multilingual models, selected windows improve a teacher-forced answer measure over the host but do not surpass published windows. Their effect on the language of free generation remains unmeasured. The method offers a way to narrow layer-swap search while keeping the final choice subject to direct validation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
layer grafting, model composition, transfer learning, first-order approximation, Transformer language models, multilingual transfer, efficient model selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7RKYOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies the problem of selecting a band of layers to swap in layer swapping, where a contiguous block of layers from a donor expert replaces the corresponding block in a host expert. Existing methods typically require building and evaluating many merged models. The authors propose SUTURE, which scores all candidate windows using a single forward and backward pass on the host model. They prove a first-order additivity theorem: the readout change from a multi-layer graft approximately equals the sum of single-layer scores, with a second-order remainder. SUTURE then calibrates a language constraint on a small number of built windows, selects a window, and certifies the choice using a fixed budget of built competitors. Controlled-stack experiments confirm the additivity, second-order remainder, ranking performance, and certification behavior, while showing that ranking degrades as experts move away from the shared base. On public multilingual models, the selected windows improve a teacher-forced answer measure over the host but do not surpass published windows, and their effect on free-generation language remains unmeasured. The paper also reports negative results, including the lack of predictive power of the λ factor and the failure of a small from-scratch transformer to reproduce the phenomenon. Overall, the paper offers a way to score all layer windows before building any merged model, with theoretical and controlled empirical support, but the evidence on language models remains limited.

Critical strengths

The method is computationally efficient. SUTURE scores all candidate layer windows using only one forward and one backward pass on the host model, avoiding the need to build and evaluate a separate merged model for each candidate. This directly addresses the main cost of prior layer-swapping search methods, where selection typically requires sweeping or searching over many built candidates.

Critical weaknesses
  1. Why choose a model with strong language ability as the host and a model with strong reasoning ability as the donor? My intuition is that reasoning is a more difficult task than language and may require more parameters to preserve, whereas language ability seems easier to acquire by modifying only part of the parameters. Should the authors instead try using the reasoning-strong model as the host and the language-strong model as the donor?
  2. The source of the language models. According to the referenced work(Lasbordes et al. (2026)), the models used in this paper appear to have already been trained on reasoning data in the target language. If so, why is layer swapping still necessary? Alternatively, would training a language model and a reasoning model separately and then merging them be better than training a single model on both language and reasoning abilities? The authors need experimental evidence, or at least should cite existing evidence to justify this choice, and should state it clearly in the paper.
  3. Missing details on data construction. For example, translation templates, quality control, training data sources, and construction procedures are not described in either the main text or the appendix.
  4. The method does not outperform other existing methods.
  5. The writing is not well standardized. There is a half-page blank in the main text, and the equations, tables, and other elements in the appendix have formatting issues.
Questions for authors

I think weakness 1 and weakness 2 are crucial.

Reviewer_rLDZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper addresses how to choose which contiguous window (or subset SS) of layers from a donor expert θd\theta^d to swap into a host expert θh\theta^h (both fine-tuned from a shared base model θb\theta^b) without constructing and evaluating every candidate merged model. By evaluating each donor layer off-path at the host's cached residual stream states xℓhx_\ell^h to obtain the injection vℓ=Bℓ(xℓh;θℓd)−Bℓ(xℓh;θℓh)v_\ell = B_\ell(x_\ell^h; \theta_\ell^d) - B_\ell(x_\ell^h; \theta_\ell^h) and taking its inner product with the host's readout gradient sℓφ=∂φ(xLh)/∂xℓ+1s_\ell^\varphi = \partial \varphi(x_L^h) / \partial x_{\ell+1}, the authors define per-layer scores aℓφ=Ep⟨sℓφ(p),vℓ(p)⟩a_\ell^\varphi = \mathbb{E}_p\langle s_\ell^\varphi(p), v_\ell(p)\rangle from a single host forward-backward pass per probe. Theorem 1 proves that under local smoothness, the readout change Δφ(S)\Delta \varphi(S) decomposes into the subset sum ∑ℓ∈Saℓφ\sum_{\ell \in S} a_\ell^\varphi plus a second-order remainder ∣r(S)∣≤C1λεS+C2εS2|r(S)| \le C_1 \lambda \varepsilon_S + C_2 \varepsilon_S^2 .

The proposed algorithm, SUTURE (Algorithm 1), computes first-response-token log-probability readouts for task capability (φtask\varphi_{\text{task}}) and host language retention (φlang\varphi_{\text{lang}}), builds mc=10m_c=10 random windows to calibrate a linear slope bb on the language constraint, selects S^=arg⁡max⁡S∈FAtask(S)\hat{S} = \arg\max_{S \in \mathcal{F}} A_{\text{task}}(S) subject to bAlang(S)≥−τb A_{\text{lang}}(S) \ge -\tau, and optionally builds m=10m=10 feasible competitors to certify S^\hat{S} via a conformal exchangeability bound on the remainders (Proposition 3).

  • Controlled Stacks: On 12-layer synthetic residual MLP stacks (d=24,T=5d=24, T=5 ), the linear approximation achieves median Pearson 0.950.95, quadratic remainder decay (log-log slope 2.0452.045–2.0552.055), and valid competitor certificates.
  • Language Models (Section 6): On public Qwen3-8B multilingual specialists (French and Chinese hosts, English donor), SUTURE selects [13,15][13, 15] for French and [20,31][20, 31] for Chinese. On 250 held-out MGSM-Rev2 test questions, these windows improve teacher-forced answer log-probability over the host (+0.398 and +1.326 nats/token) but underperform the published windows ([13,22][13, 22] and [13,20][13, 20] ) by −0.355-0.355 and −0.131-0.131 nats/token. On Qwen2.5-Coder-1.5B-Instruct LoRA experts, the scores correlate with teacher-forced log-probability (Spearman 0.8770.877) but fail to rank free-generation exact match (Spearman −0.018-0.018 and 0.1290.129).
Critical strengths
  • Clean first-order superposition analysis and conformal certification: Expanding multi-layer grafts around the host trajectory (Lemma 2 and Theorem 1) neatly decouples donor layers to first order, allowing all 2L2^L subsets or L(L+1)/2L(L+1)/2 windows to be scored from one host forward-backward pass per probe. Combining this with a 10-build constraint calibration step and a finite-sample exchangeability bound on unbuilt competitor remainders (Proposition 3) is well-reasoned and mathematically sound.
  • Commendable scientific candor regarding negative results: The authors are exceptionally transparent about where their approximation and experiments fall short. They explicitly report that SUTURE loses to published bands at 8B, fails to rank free-generation accuracy at 1.5B, fails to produce cross-lingual transfer in a from-scratch transformer testbed, and disproves their own conjecture that the Jacobian gap λ\lambda predicts remainder magnitude at fixed weight distance (Appendix C).
Critical weaknesses
  • Incomplete and negative validation on actual language models: The motivating application of cross-lingual layer swapping is to import multi-step mathematical reasoning while preserving the host's output language during free autoregressive generation. However, on the 8B models, free-generation accuracy and generated-text language identification were never measured at all (attributed to a "decoding-cache mismatch"); only teacher-forced answer log-probability and a first-token language proxy were evaluated. Because the language readout (Eq. 5) only inspects the very first response token, it cannot detect mid-generation language switching: for Chinese, SUTURE selects layers [20,31][20, 31] , extending deep into the late layers past layer 20 where Lasbordes et al. (2026) already showed reasoning traces revert to English. Moreover, where free generation is evaluated (the 1.5B study), SUTURE's ranking of exact-match accuracy collapses to chance (Spearman −0.018-0.018 on 406 windows and 0.1290.129 on 64 held-out items).
  • No demonstrated empirical advantage over published windows or simpler attribution baselines: On the 8B teacher-forced evaluation, SUTURE's chosen windows lose significantly to the published bands (−0.355-0.355 nats/token in French, −0.131-0.131 in Chinese), and standard first-order weight attribution selects the exact same windows ([13,15][13, 15] and [20,31][20, 31] ). Likewise, in the controlled stacks, once experts are obtained by gradient training rather than random perturbation (Table 4), SUTURE (0.073±0.1400.073 \pm 0.140 regret) is statistically indistinguishable from first-order weight attribution (pHolm=0.632p_{\text{Holm}} = 0.632), dropping the adjoint ∥vℓ∥\Vert{}v_\ell\Vert{}{}{} (pHolm=0.632p_{\text{Holm}} = 0.632), and simple parameter movement from the base (pHolm=0.535p_{\text{Holm}} = 0.535). Thus, whenever experts are trained via gradient descent (both in synthetic stacks and 8B LLMs), SUTURE shows no empirical gain over existing first-order weight attribution.
Questions for authors
  • Clarification on the 8B decoding-cache mismatch and free generation: Why did a "decoding-cache mismatch" prevent running free generation on the Qwen3-8B models? Since grafting a window [i,j][i, j] produces a standard Qwen3-8B checkpoint with donor weights copied into layers i…ji \dots j, why couldn't standard HuggingFace generate() or vLLM be run on the three final checkpoints (host, selected, published) for the 250 MGSM-Rev2 test items to measure actual pass@1 accuracy and fastText language ID? In particular, does the Chinese [20,31][20, 31] graft revert to English during chain-of-thought generation?
  • Clarification on the Chinese calibration slope (b=0.012b = 0.012): In Section 6, fitting the language calibration slope on 10 windows for Chinese yields b=0.012b = 0.012 with RMSE 0.9160.916. Doesn't multiplying Alang(S)A_{\text{lang}}(S) by 0.0120.012 shrink the predicted language loss by ∼83×\sim 83\times, effectively disabling the constraint bAlang(S)≥−0.05b A_{\text{lang}}(S) \ge -0.05 and allowing the optimizer to pick the aggressive [20,31][20, 31] window?
  • Suggestion for future work: Replacing the single-token readouts (Eq. 5) with sequence-averaged log-probabilities over teacher-forced chain-of-thought traces in the target language would retain differentiability for Theorem 1 while capturing multi-step reasoning and mid-trace language drift.
Reviewer_JTTaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Layer Swapping is when you merge two models together by taking entire layers from one and the other. This paper proposes a method, SUTURE, to more efficiently select layers by designing using gradient-based scores to estimate the effect of each swap on target task and language performance. This method uses the fact that, because of the residual stream, contributions of each swapped layer is approximately additive. The experiments show that this can quickly rank candidates.

Critical strengths
  1. While previous work shoes layers for Layer Swapping through hand-wavy, intuition-based methods or small-scale sweeps. This paper proposes empirical way to evaluate potential layer choices with just a few forward & backward passes.
  2. good Reproducibility; many details provided
Critical weaknesses
  1. Presentation/Writing is terrible, paper is impossible to follow. Word choices make paper really hard to follow: “band”, “donor”, “window”, “grafts”, “swap”, “host” “source”, “sweep”, “candidate builds”, “readout”, “patching”, “suture”, “stacks”, “teacher”, “imported”, “practitioner, “partition”, “surrogate”, “injection”, “adjoint”, “first-order decomposition”. My strong suggestion is, for clarity, re-use the same terms (or at least narrow the set of terms) to allow the reader to follow. E.g. call the method Layer swapping, the relevant models “merged/source” models, etc. In addition, you cannot wait until Section 2 to define these terms; I had no idea what was going on in the Intro. Writing everywhere is near-illegible, over-jargonized and unclear.
  2. Proposed method is unnecessarily complicated: so much machinery doing activation patching, gradients, and stuff that was difficult to understand. Many individual decisions/design choices are not justified. And there seems to be no evidence that this provides practical benefits over already existing methods (“published”).
  3. Beyond no contribution of this paper, the motivation of studying this subject is never really presented. As someone very familiar with this niche space, I see why trying to come up with a fast way to elect model layers for swapping could be helpful. But I’d never understand it from reading this paper.
Questions for authors
  1. (suggestion) I understood it eventually, but the notation in Equation 1 is not well-explained/defined.
  2. L80-81 what is a practitioner ?
  3. L83-89 I didn’t answer what is going on here, what function is \phi ? I initially thought it would be something related to logits, but it takes in hidden states as inputs ? also what does readout mean ? Why only take last prompt position ? What is a probe in this context ?

Comments on the paper

1 comment
Reviewer_rLDZOpenReview ↗

Response to the authors

I appreciate the authors' response to my questions, and for running the 8B free-generation evaluation on MGSM-Rev2, clarifying that the earlier cache note came from a stopped German pilot, and updating Section 6 and Appendix D.1 with such candor. The new generation results resolve our factual questions and clarify both the strengths and the core boundary of SUTURE. On French, the selected shallow window [13,15][13, 15] improves Numeric EM by +6.0+6.0 percentage points over the host (95.2%95.2\% vs.\ 89.2%89.2\%, 95%95\% CI [2.0,10.0][2.0, 10.0]) and Integer EM by +13.6+13.6 points (88.8%88.8\% vs.\ 75.2%75.2\%), while staying in French across all 247 evaluable traces (98.8%98.8\% Target ID, 0.0%0.0\% English-classified text segments) and avoiding the 55 length-cap truncations suffered by the wider published [13,22][13, 22] swap (65.6%65.6\% Numeric EM). However, on Chinese, the selected window [20,31][20, 31] fails the language-preservation goal---switching to English on 225 of 248 complete traces (9.2%9.2\% Target ID, 95.2%95.2\% containing English-classified text segments) with no statistically significant Numeric EM gain over the host (+0.4+0.4 percentage points, 95%95\% CI [−3.2,4.0][-3.2, 4.0]). Crucially, because [20,31][20, 31] is already selected under the uncalibrated b=1b=1 threshold (not only under the fitted b=0.012119b=0.012119 slope), this confirms that a first-response-token log-probability proxy (φlang\varphi_{\text{lang}}) cannot detect mid-generation language collapse when late decoder layers are replaced. Furthermore, constrained first-order weight attribution selects the exact same windows ([13,15][13, 15] and [20,31][20, 31]), matching the synthetic-stack finding where \textsc{Suture} and weight attribution are statistically indistinguishable on gradient-trained experts (pHolm=0.632p_{\text{Holm}} = 0.632), and the 1.5B generation accuracy correlations (−0.018-0.018, 0.1290.129) remain inconclusive. As a final clarification question: if you restrict the 8B candidate search to exclude late language-commitment layers (e.g., ℓ≤20\ell \le 20), which Chinese window does SUTURE select, and does it still coincide with weight attribution? In recognition of the completed generation runs and transparent reporting, we have raised our soundness score from 2 to 3, and our overall rating from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.