- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies the problem of selecting a band of layers to swap in layer swapping, where a contiguous block of layers from a donor expert replaces the corresponding block in a host expert. Existing methods typically require building and evaluating many merged models. The authors propose SUTURE, which scores all candidate windows using a single forward and backward pass on the host model. They prove a first-order additivity theorem: the readout change from a multi-layer graft approximately equals the sum of single-layer scores, with a second-order remainder. SUTURE then calibrates a language constraint on a small number of built windows, selects a window, and certifies the choice using a fixed budget of built competitors. Controlled-stack experiments confirm the additivity, second-order remainder, ranking performance, and certification behavior, while showing that ranking degrades as experts move away from the shared base. On public multilingual models, the selected windows improve a teacher-forced answer measure over the host but do not surpass published windows, and their effect on free-generation language remains unmeasured. The paper also reports negative results, including the lack of predictive power of the λ factor and the failure of a small from-scratch transformer to reproduce the phenomenon. Overall, the paper offers a way to score all layer windows before building any merged model, with theoretical and controlled empirical support, but the evidence on language models remains limited.
Critical strengths
The method is computationally efficient. SUTURE scores all candidate layer windows using only one forward and one backward pass on the host model, avoiding the need to build and evaluate a separate merged model for each candidate. This directly addresses the main cost of prior layer-swapping search methods, where selection typically requires sweeping or searching over many built candidates.
Critical weaknesses
- Why choose a model with strong language ability as the host and a model with strong reasoning ability as the donor? My intuition is that reasoning is a more difficult task than language and may require more parameters to preserve, whereas language ability seems easier to acquire by modifying only part of the parameters. Should the authors instead try using the reasoning-strong model as the host and the language-strong model as the donor?
- The source of the language models. According to the referenced work(Lasbordes et al. (2026)), the models used in this paper appear to have already been trained on reasoning data in the target language. If so, why is layer swapping still necessary? Alternatively, would training a language model and a reasoning model separately and then merging them be better than training a single model on both language and reasoning abilities? The authors need experimental evidence, or at least should cite existing evidence to justify this choice, and should state it clearly in the paper.
- Missing details on data construction. For example, translation templates, quality control, training data sources, and construction procedures are not described in either the main text or the appendix.
- The method does not outperform other existing methods.
- The writing is not well standardized. There is a half-page blank in the main text, and the equations, tables, and other elements in the appendix have formatting issues.
Questions for authors
I think weakness 1 and weakness 2 are crucial.
Response to the authors
I appreciate the authors' response to my questions, and for running the 8B free-generation evaluation on MGSM-Rev2, clarifying that the earlier cache note came from a stopped German pilot, and updating Section 6 and Appendix D.1 with such candor. The new generation results resolve our factual questions and clarify both the strengths and the core boundary of SUTURE. On French, the selected shallow window [13,15] improves Numeric EM by +6.0 percentage points over the host (95.2% vs.\ 89.2%, 95% CI [2.0,10.0]) and Integer EM by +13.6 points (88.8% vs.\ 75.2%), while staying in French across all 247 evaluable traces (98.8% Target ID, 0.0% English-classified text segments) and avoiding the 55 length-cap truncations suffered by the wider published [13,22] swap (65.6% Numeric EM). However, on Chinese, the selected window [20,31] fails the language-preservation goal---switching to English on 225 of 248 complete traces (9.2% Target ID, 95.2% containing English-classified text segments) with no statistically significant Numeric EM gain over the host (+0.4 percentage points, 95% CI [−3.2,4.0]). Crucially, because [20,31] is already selected under the uncalibrated b=1 threshold (not only under the fitted b=0.012119 slope), this confirms that a first-response-token log-probability proxy (φlang) cannot detect mid-generation language collapse when late decoder layers are replaced. Furthermore, constrained first-order weight attribution selects the exact same windows ([13,15] and [20,31]), matching the synthetic-stack finding where \textsc{Suture} and weight attribution are statistically indistinguishable on gradient-trained experts (pHolm=0.632), and the 1.5B generation accuracy correlations (−0.018, 0.129) remain inconclusive. As a final clarification question: if you restrict the 8B candidate search to exclude late language-commitment layers (e.g., ℓ≤20), which Chinese window does SUTURE select, and does it still coincide with weight attribution? In recognition of the completed generation runs and transparent reporting, we have raised our soundness score from 2 to 3, and our overall rating from 2 to 3.