- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SLOA (Sequential Local Operator Alignment), a training-free framework for merging transformer models. Its argument has three layers:
- The object being merged should be the local functional operator, not a parameter tensor. For an attention head, the functionally meaningful objects are the composed operators θ_QK = W_Q W_Kᵀ/√d_k and θ_VO = W_V W_O, rather than W_Q, W_K, W_V, W_O themselves (§2, Example 1, Eq. 1). Independently averaging the factors introduces multiplicative and cross terms that depend on an arbitrary invertible R.
- Sequential alignment. Operators are merged one at a time in topological order; at step m, the operator being merged is evaluated on the local input Ẑ^(m) produced by the partially merged model, while the target remains the local output A(Z^(m); θ_t) of the original task model on its own input (Eq. 2, §3.4). Each step is therefore corrective with respect to upstream merging error.
- Decoupling the operator from its factorization → rank as a capacity control. The merged operator θ_merge is first solved for in the ambient space R^{d×d}, and only afterwards factorized at a chosen rank r, e.g. W̄_Q = U_r(τΣ_r)^{1/2}; r = d_k preserves the original architecture, while r > d_k expands capacity (§3.5, §D.5).
The unified objective combines behavior matching with a data-dependent operator geometry (a Mahalanobis regularizer) (Eq. 5). For fixed local inputs it is quadratic in θ, so every operator admits a closed-form solution (§D.1–D.4): QK uses the Kronecker structure plus CG (Eq. 76–78); VO uses rank-constrained least squares with an augmented design matrix (Eq. 85–89, Friedland–Torokhti); MLP becomes ridge regression; and LayerNorm decouples into a 2×2 linear system.
Experiments cover ViT-B/32 and ViT-B/16 (8/14/20 tasks), RoBERTa-base (7 GLUE tasks), and three LLaMA-2-7B LoRA experts (math/code/instruction-following, VO composition only), together with LoRA-ViT experiments, ablations on calibration samples / rank / regularization strength / error accumulation / components, and analyses of merging and inference cost.
Critical strengths
1. The problem is well identified, and the method is natural and self-consistent. The observation that "the parameterization is not the functional unit" genuinely holds inside attention (the QK/OV circuit view of Elhage et al. 2021), and turning it into a solvable merging algorithm is a reasonable and valuable step. Unifying LayerNorm, linear maps, QK, and VO under a single "behavior map + data-dependent metric" template (Table 1, Eq. 5) is cleanly presented.
S2. The derivations are correct. I checked Appendix D item by item: (i) the multiplicative factor ((1+c)(1+c⁻¹)/4) in Eq. (1) equals 1 only at c = 1; (ii) the QK normal equations ∇θ‖XᵀθX−Y‖²_F = 2Xᵀ(XθXᵀ−Y)X, the symmetry of R = ẐᵀẐ, the identity vec(RθR) = (R⊗R)vec(θ), and the definitions of H and C are all consistent; (iii) for the rank-constrained VO solution, after stacking Λ_t^{1/2} as additional rows of the design matrix, I independently verified θ* = (Z̃ᵀZ̃)^{−1/2}·trunc_r[(Z̃ᵀZ̃)^{−1/2}Z̃ᵀỸ] via the isometry-plus-rank-r-truncation argument, which matches the general Friedland–Torokhti solution; (iv) the argument that "dividing by the constant T does not change the solution" holds because trunc_r is homogeneous of degree one; (v) the identity that Λ^{1/2}ΔΛ^{1/2} corresponds to the Mahalanobis metric with covariance Λ⊗Λ holds; (vi) the factorization in Eq. (7)/(108) satisfies W̄_QW̄_Kᵀ/τ = θ^r (with τ = √r the softmax logits are exactly Xθ^r Xᵀ, so the choice is self-consistent); (vii) the LayerNorm 2×2 system and its Cramer solution are correct. I found no mathematical error.
S3. The breadth and rigor of the experiments exceed comparable work. Four backbones, four task counts, fully fine-tuned and LoRA experts, 10+ baselines (including recent ones: TSV-M, Iso-C/Iso-CTS, RegMean++, KnOTS-TIES), per-task tables, disclosure of the calibration coverage relative to the training set (Tables 5–7), use of the training split only, averaging over 3 seeds, and honest cost tables (Tables 10–15). I re-computed several averages and they are internally consistent: the 20-task average of SLOA (r=64) in Table 19 is indeed 80.06 (RegMean++ 74.51, a margin of 5.55); the values 0.7906 / 0.7979 / 0.7139 in Table 16 can all be reproduced from the seven tasks; so can 48.17 / 48.64 / 49.19 / 46.46 in Table 4.
S4. In the architecture-preserving setting, the claims are supported by the evidence. At r = d_k (no head widening) SLOA still beats every baseline: ViT-B/32 20 tasks 80.06 vs 74.51; ViT-B/16 20 tasks 82.12 vs 76.64; GLUE 0.7906 vs 0.7139; LLaMA 48.17 vs 46.46. The claim that the gains do not require rank expansion is therefore justified, and rank expansion is honestly positioned as an accuracy–inference-cost trade-off.
S5. The error-accumulation analysis genuinely supports the sequential claim. I zoomed into Figure 6(a): for every rank, the per-block mean relative L2 error of SLOA is below RegMean, RegMean++, and Iso-CTS after roughly block 4 (e.g., at block 10, SLOA r=256 ≈ 61% vs RegMean++ ≈ 70%, RegMean ≈ 77%, Iso-CTS ≈ 87%), and the stated peak-to-starting-value ratios of 1.65–1.69× versus 1.87×/2.08× match the figure. This is not a figure that cherry-picks a flattering metric.
S6. The limitations are stated honestly. Reliance on calibration data, the greedy rather than end-to-end objective, higher merging cost than data-free methods, the extra inference cost of r > d_k, and the fact that QK is not composed under RoPE are all acknowledged.
Critical weaknesses
The overall quality of the paper is very high, but I have doubts about the paper's most central contribution, "operator composition." Its independent gain is quite small in the ablation, and the paper discusses this insufficiently. Both Figure 6(b) and Figure 8 show that, under ViT-B/32, 20 tasks, the gap between Sequential (Isolated) and full SLOA is not large, and the gain relative to RegMean++ mainly comes from SLOA's own sequential/cross-state matching correction mechanism, rather than the paper's unique operator composition. In other words, the paper's unique point of "merging on composite operators" contributes only a modest improvement. This does not match the weight of the core claim in the introduction that "model merging should be performed at local functional operators" (Sec 1, last paragraph, page 2, manuscript line 101). Please provide more experimental evidence to support the gain of the core claim.
Second, Example 1 proves that "independent averaging is not equivariant to factor reparameterization," which is correct, but experts fine-tuned from the same source will not really differ by a random invertible R. The paper does not answer how much functional error the factor ambiguity of QK/VO actually causes on real experts. There is a missing causal chain between the argument in Sec 2 of the paper and the benefits reported in Sec 4 of the paper: Sec 4 only reports final accuracy, and no experiment links the "functional deviation caused by factor ambiguity" to the actual benefits. It is suggested to add experiments.
Questions for authors
Q1. How much of the gain is attributable to operator composition? Figures 6(b) and 8 (ViT-B/32 multi-task setting) show that Sequential (Isolated) and full SLOA perform comparably, and that the isolated baseline can even match or exceed full SLOA at small calibration budgets. The margin over RegMean++ therefore appears to come mainly from SLOA's sequential/cross-state matching rather than from merging the composed QK/VO operators. Since the central claim is that "model merging should be performed at local functional operators" (Sec 1), could the authors provide a formal multi-seed ablation across the evaluated task counts that isolates the operator-composition gain with confidence intervals, and explain why this gain is modest?
Q2. Can the factorization-ambiguity motivation be linked to the reported gains? Example 1 shows that independently averaging the factors is not invariant under (W_Q, W_K) ↦ (W_Q R, W_K R^{−⊤}) (and analogously for the VO factors), yet experts fine-tuned from a shared initialization are not actually related by a random invertible R. Since Sec 4 reports only final accuracy, it remains unclear how much functional error the QK/VO factorization ambiguity actually causes on real experts. Could the authors (a) measure this functional deviation directly (e.g., via the relative functional error of Fig. 2), and/or (b) provide a controlled experiment in which a baseline receives an equivalent factor reparameterization and degrades while SLOA remains unchanged?
Composition ablation (8/14/20 tasks) and reparameterization robustness
We thank the reviewer for the thorough review. Both questions concern how much operator composition contributes; we answer them with new experiments below, and we have marked all changes to the paper in blue.
W1/Q1: Isolating the gain from operator composition vs. sequential correction
To isolate the gain of composition, SLOA (Sequential only) applies SLOA's own sequential solver, including LayerNorm and bias corrections, to WQ, WK, WV, and WO separately, with the same data (N=256), regularization, and block order; only composition differs (ViT-B/32, 3 seeds; Δ = gain over SLOA (Sequential only), 95% CI):
At r=dk, the gain is small at 8 tasks and grows at 14 and 20 tasks. Composition also enables rank expansion, which adds most of the gain: at r=128, SLOA beats SLOA (Sequential only) by +1.40 points on 20 tasks and on every single task, by up to +6.2 (GTSRB) and +5.9 (EMNIST) [Table 11, App. F.9]. SLOA's per-block error is also at or below that of SLOA (Sequential only) at every depth, on ViT-B/32 and ViT-B/16 [Figure 10, App. F.9], so composition also lowers functional error throughout the network.
Why the gain at r=dk is modest. As the reviewer notes in Q2, experts fine-tuned from a shared initialization remain close to a common factorization. On such experts, a factor-wise solution that already benefits from SLOA's sequential correction approximates the composed solution well. Composition, however, does not rely on this assumption: in answering Q2 below, a small function-preserving perturbation of the experts' parameters (σ=0.1) costs SLOA (Sequential only) 9.3 points, while SLOA is unchanged. Composition therefore provides robustness to the experts' factorization, enables rank expansion, and gives larger gains as more tasks are merged.
We thank the reviewer for the questions and have revised the motivation (Section 2) to better highlight the importance of operator composition and its limitations.
W2/Q2: Linking factorization ambiguity to the gains via a controlled reparameterization
Following the reviewer's suggestion, we ran the controlled experiment.
Setup. ViT-B/32, 20 tasks, 3 seeds. We reparameterize each fine-tuned expert independently, per attention head:
(WQ,WK)↦(WQR, WKR−⊤),(WV,WO)↦(WVS, S−1WO),
with R,S=I+σN, where N is Gaussian. This leaves the function unchanged.
Key comparison: SLOA (Sequential only) vs. SLOA r=64. Both use the same data, sequential correction, and architecture. The only difference: SLOA (Sequential only) aligns WQ, WK, WV, WO separately; SLOA aligns WQWK⊤ and WVWO.
Avg. accuracy (%) over 20 tasks, mean over 3 seeds (Figure 11 in Appendix F.10 of the revised paper shows the same results as a plot for better understanding):
SLOA changes by at most 0.08 points, within seed noise. All factor-wise methods degrade as σ grows: TSV-M and Iso-CTS already lose 4–9 points at σ=0.01, and all reach chance level (10%) at σ=1. Simple Avg., Iso-C, and Fisher behave the same; we report all methods in [Table 12, Appendix F.10].
So the robustness comes from operating on the composed operator. It does not come from using calibration data or from sequential correction: at σ=0.1, RegMean and RegMean++ (which use data) lose about 10 points, and sequential correction alone loses 9.3 points, while SLOA with the composed operator remains unaffected.
We once again thank the reviewer for their comments and positive scores, and we would be glad to discuss any further suggestions.