acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Transformer Merging via Sequential Local Operator Alignment

Abstract

Training-free model merging aims to combine multiple fine-tuned models into a single model without further optimization on labeled data. Yet, in transformers, independently merging individual layers can affect a shared attention computation because the query–key and value–output operators depend on composed matrices, overlooking the functional structure. Moreover, when merging earlier components, downstream components receive different activations than they do in the original model, thus, the merged and original execution paths no longer match. In this paper, we introduce Sequential Local Operator Alignment, a training-free method that merges transformers along the execution path of the partially merged model. Our method uses calibration data to estimate the local behavior of each functional component, aligns operators sequentially under the intermediate activation of the partially merged model, and subsequently factorizes the merged operators back into valid transformer parameters. We empirically show that this sequential step reduces error accumulation across layers. Furthermore, the proposed operator factorization step enables rank expansion, providing a principled mechanism for increasing multi-task capacity. We demonstrate that our approach generalizes across modalities, model scales, and varying numbers of tasks, from CLIP and RoBERTa to billion-parameter LLMs, and further extends naturally to the merging of LoRA-fine-tuned models. The results indicate improvements over strong merging baselines without requiring rank expansion, while optional expansion provides a further accuracy–inference-cost trade-off.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
model merging, training-free model merging, transformers, attention mechanism, parameter symmetry, task vectors, multi-task learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Yya5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SLOA (Sequential Local Operator Alignment), a training-free framework for merging transformer models. Its argument has three layers:

  1. The object being merged should be the local functional operator, not a parameter tensor. For an attention head, the functionally meaningful objects are the composed operators θ_QK = W_Q W_Kᵀ/√d_k and θ_VO = W_V W_O, rather than W_Q, W_K, W_V, W_O themselves (§2, Example 1, Eq. 1). Independently averaging the factors introduces multiplicative and cross terms that depend on an arbitrary invertible R.
  2. Sequential alignment. Operators are merged one at a time in topological order; at step m, the operator being merged is evaluated on the local input Ẑ^(m) produced by the partially merged model, while the target remains the local output A(Z^(m); θ_t) of the original task model on its own input (Eq. 2, §3.4). Each step is therefore corrective with respect to upstream merging error.
  3. Decoupling the operator from its factorization → rank as a capacity control. The merged operator θ_merge is first solved for in the ambient space R^{d×d}, and only afterwards factorized at a chosen rank r, e.g. W̄_Q = U_r(τΣ_r)^{1/2}; r = d_k preserves the original architecture, while r > d_k expands capacity (§3.5, §D.5).

The unified objective combines behavior matching with a data-dependent operator geometry (a Mahalanobis regularizer) (Eq. 5). For fixed local inputs it is quadratic in θ, so every operator admits a closed-form solution (§D.1–D.4): QK uses the Kronecker structure plus CG (Eq. 76–78); VO uses rank-constrained least squares with an augmented design matrix (Eq. 85–89, Friedland–Torokhti); MLP becomes ridge regression; and LayerNorm decouples into a 2×2 linear system.

Experiments cover ViT-B/32 and ViT-B/16 (8/14/20 tasks), RoBERTa-base (7 GLUE tasks), and three LLaMA-2-7B LoRA experts (math/code/instruction-following, VO composition only), together with LoRA-ViT experiments, ablations on calibration samples / rank / regularization strength / error accumulation / components, and analyses of merging and inference cost.

Critical strengths

1. The problem is well identified, and the method is natural and self-consistent. The observation that "the parameterization is not the functional unit" genuinely holds inside attention (the QK/OV circuit view of Elhage et al. 2021), and turning it into a solvable merging algorithm is a reasonable and valuable step. Unifying LayerNorm, linear maps, QK, and VO under a single "behavior map + data-dependent metric" template (Table 1, Eq. 5) is cleanly presented.

S2. The derivations are correct. I checked Appendix D item by item: (i) the multiplicative factor ((1+c)(1+c⁻¹)/4) in Eq. (1) equals 1 only at c = 1; (ii) the QK normal equations ∇θ‖XᵀθX−Y‖²_F = 2Xᵀ(XθXᵀ−Y)X, the symmetry of R = ẐᵀẐ, the identity vec(RθR) = (R⊗R)vec(θ), and the definitions of H and C are all consistent; (iii) for the rank-constrained VO solution, after stacking Λ_t^{1/2} as additional rows of the design matrix, I independently verified θ* = (Z̃ᵀZ̃)^{−1/2}·trunc_r[(Z̃ᵀZ̃)^{−1/2}Z̃ᵀỸ] via the isometry-plus-rank-r-truncation argument, which matches the general Friedland–Torokhti solution; (iv) the argument that "dividing by the constant T does not change the solution" holds because trunc_r is homogeneous of degree one; (v) the identity that Λ^{1/2}ΔΛ^{1/2} corresponds to the Mahalanobis metric with covariance Λ⊗Λ holds; (vi) the factorization in Eq. (7)/(108) satisfies W̄_QW̄_Kᵀ/τ = θ^r (with τ = √r the softmax logits are exactly Xθ^r Xᵀ, so the choice is self-consistent); (vii) the LayerNorm 2×2 system and its Cramer solution are correct. I found no mathematical error.

S3. The breadth and rigor of the experiments exceed comparable work. Four backbones, four task counts, fully fine-tuned and LoRA experts, 10+ baselines (including recent ones: TSV-M, Iso-C/Iso-CTS, RegMean++, KnOTS-TIES), per-task tables, disclosure of the calibration coverage relative to the training set (Tables 5–7), use of the training split only, averaging over 3 seeds, and honest cost tables (Tables 10–15). I re-computed several averages and they are internally consistent: the 20-task average of SLOA (r=64) in Table 19 is indeed 80.06 (RegMean++ 74.51, a margin of 5.55); the values 0.7906 / 0.7979 / 0.7139 in Table 16 can all be reproduced from the seven tasks; so can 48.17 / 48.64 / 49.19 / 46.46 in Table 4.

S4. In the architecture-preserving setting, the claims are supported by the evidence. At r = d_k (no head widening) SLOA still beats every baseline: ViT-B/32 20 tasks 80.06 vs 74.51; ViT-B/16 20 tasks 82.12 vs 76.64; GLUE 0.7906 vs 0.7139; LLaMA 48.17 vs 46.46. The claim that the gains do not require rank expansion is therefore justified, and rank expansion is honestly positioned as an accuracy–inference-cost trade-off.

S5. The error-accumulation analysis genuinely supports the sequential claim. I zoomed into Figure 6(a): for every rank, the per-block mean relative L2 error of SLOA is below RegMean, RegMean++, and Iso-CTS after roughly block 4 (e.g., at block 10, SLOA r=256 ≈ 61% vs RegMean++ ≈ 70%, RegMean ≈ 77%, Iso-CTS ≈ 87%), and the stated peak-to-starting-value ratios of 1.65–1.69× versus 1.87×/2.08× match the figure. This is not a figure that cherry-picks a flattering metric.

S6. The limitations are stated honestly. Reliance on calibration data, the greedy rather than end-to-end objective, higher merging cost than data-free methods, the extra inference cost of r > d_k, and the fact that QK is not composed under RoPE are all acknowledged.

Critical weaknesses

The overall quality of the paper is very high, but I have doubts about the paper's most central contribution, "operator composition." Its independent gain is quite small in the ablation, and the paper discusses this insufficiently. Both Figure 6(b) and Figure 8 show that, under ViT-B/32, 20 tasks, the gap between Sequential (Isolated) and full SLOA is not large, and the gain relative to RegMean++ mainly comes from SLOA's own sequential/cross-state matching correction mechanism, rather than the paper's unique operator composition. In other words, the paper's unique point of "merging on composite operators" contributes only a modest improvement. This does not match the weight of the core claim in the introduction that "model merging should be performed at local functional operators" (Sec 1, last paragraph, page 2, manuscript line 101). Please provide more experimental evidence to support the gain of the core claim.

Second, Example 1 proves that "independent averaging is not equivariant to factor reparameterization," which is correct, but experts fine-tuned from the same source will not really differ by a random invertible R. The paper does not answer how much functional error the factor ambiguity of QK/VO actually causes on real experts. There is a missing causal chain between the argument in Sec 2 of the paper and the benefits reported in Sec 4 of the paper: Sec 4 only reports final accuracy, and no experiment links the "functional deviation caused by factor ambiguity" to the actual benefits. It is suggested to add experiments.

Questions for authors

Q1. How much of the gain is attributable to operator composition? Figures 6(b) and 8 (ViT-B/32 multi-task setting) show that Sequential (Isolated) and full SLOA perform comparably, and that the isolated baseline can even match or exceed full SLOA at small calibration budgets. The margin over RegMean++ therefore appears to come mainly from SLOA's sequential/cross-state matching rather than from merging the composed QK/VO operators. Since the central claim is that "model merging should be performed at local functional operators" (Sec 1), could the authors provide a formal multi-seed ablation across the evaluated task counts that isolates the operator-composition gain with confidence intervals, and explain why this gain is modest?

Q2. Can the factorization-ambiguity motivation be linked to the reported gains? Example 1 shows that independently averaging the factors is not invariant under (W_Q, W_K) ↦ (W_Q R, W_K R^{−⊤}) (and analogously for the VO factors), yet experts fine-tuned from a shared initialization are not actually related by a random invertible R. Since Sec 4 reports only final accuracy, it remains unclear how much functional error the QK/VO factorization ambiguity actually causes on real experts. Could the authors (a) measure this functional deviation directly (e.g., via the relative functional error of Fig. 2), and/or (b) provide a controlled experiment in which a baseline receives an equivalent factor reparameterization and degrades while SLOA remains unchanged?

1 comment
AuthorsOpenReview ↗

Composition ablation (8/14/20 tasks) and reparameterization robustness

We thank the reviewer for the thorough review. Both questions concern how much operator composition contributes; we answer them with new experiments below, and we have marked all changes to the paper in blue.

W1/Q1: Isolating the gain from operator composition vs. sequential correction

To isolate the gain of composition, SLOA (Sequential only) applies SLOA's own sequential solver, including LayerNorm and bias corrections, to WQ\mathbf{W}_Q, WK\mathbf{W}_K, WV\mathbf{W}_V, and WO\mathbf{W}_O separately, with the same data (N=256N = 256), regularization, and block order; only composition differs (ViT-B/32, 3 seeds; Δ\Delta = gain over SLOA (Sequential only), 95% CI):

TasksSLOA (Sequential only)SLOA r=64r = 64 (=dk= d_k)SLOA r=128r = 128
884.88 ± 0.1284.89 (+0.01 [−0.09, +0.11])85.50 (+0.61 [+0.33, +0.90])
1481.53 ± 0.1081.94 (+0.41 [+0.20, +0.62])82.60 (+1.07 [+0.74, +1.40])
2079.63 ± 0.2280.06 (+0.43 [+0.28, +0.58])81.03 (+1.40 [+1.15, +1.65])

At r=dkr = d_k, the gain is small at 8 tasks and grows at 14 and 20 tasks. Composition also enables rank expansion, which adds most of the gain: at r=128r = 128, SLOA beats SLOA (Sequential only) by +1.40 points on 20 tasks and on every single task, by up to +6.2 (GTSRB) and +5.9 (EMNIST) [Table 11, App. F.9]. SLOA's per-block error is also at or below that of SLOA (Sequential only) at every depth, on ViT-B/32 and ViT-B/16 [Figure 10, App. F.9], so composition also lowers functional error throughout the network.

Why the gain at r=dkr = d_k is modest. As the reviewer notes in Q2, experts fine-tuned from a shared initialization remain close to a common factorization. On such experts, a factor-wise solution that already benefits from SLOA's sequential correction approximates the composed solution well. Composition, however, does not rely on this assumption: in answering Q2 below, a small function-preserving perturbation of the experts' parameters (σ=0.1\sigma = 0.1) costs SLOA (Sequential only) 9.3 points, while SLOA is unchanged. Composition therefore provides robustness to the experts' factorization, enables rank expansion, and gives larger gains as more tasks are merged.

We thank the reviewer for the questions and have revised the motivation (Section 2) to better highlight the importance of operator composition and its limitations.

W2/Q2: Linking factorization ambiguity to the gains via a controlled reparameterization

Following the reviewer's suggestion, we ran the controlled experiment.

Setup. ViT-B/32, 20 tasks, 3 seeds. We reparameterize each fine-tuned expert independently, per attention head:

(WQ,WK)↦(WQR, WKR−⊤),(WV,WO)↦(WVS, S−1WO),(\mathbf{W}_Q, \mathbf{W}_K) \mapsto (\mathbf{W}_Q \mathbf{R},\ \mathbf{W}_K \mathbf{R}^{-\top}), \qquad (\mathbf{W}_V, \mathbf{W}_O) \mapsto (\mathbf{W}_V \mathbf{S},\ \mathbf{S}^{-1} \mathbf{W}_O),

with R,S=I+σN\mathbf{R}, \mathbf{S} = \mathbf{I} + \sigma \mathbf{N}, where N\mathbf{N} is Gaussian. This leaves the function unchanged.

Key comparison: SLOA (Sequential only) vs. SLOA r=64r = 64. Both use the same data, sequential correction, and architecture. The only difference: SLOA (Sequential only) aligns WQ\mathbf{W}_Q, WK\mathbf{W}_K, WV\mathbf{W}_V, WO\mathbf{W}_O separately; SLOA aligns WQWK⊤\mathbf{W}_Q \mathbf{W}_K^\top and WVWO\mathbf{W}_V \mathbf{W}_O.

Avg. accuracy (%) over 20 tasks, mean over 3 seeds (Figure 11 in Appendix F.10 of the revised paper shows the same results as a plot for better understanding):

Methodσ=0\sigma = 0σ=0.01\sigma = 0.01σ=0.05\sigma = 0.05σ=0.1\sigma = 0.1σ=1\sigma = 1
Task Arithmetic60.6160.6160.4017.1910.00
TIES62.7762.9159.6110.1510.16
TSV-M73.6769.7010.2910.1010.11
Iso-CTS73.4064.1710.2210.1010.30
RegMean72.2372.2571.2361.9810.17
RegMean++74.5574.5173.3365.1510.16
SLOA (Sequential only)79.6679.6178.6370.3710.19
SLOA, r=64r = 64 (=dk= d_k)80.1380.1380.1380.1380.05
SLOA, r=128r = 12881.1481.1581.1681.1781.09

SLOA changes by at most 0.08 points, within seed noise. All factor-wise methods degrade as σ\sigma grows: TSV-M and Iso-CTS already lose 4–9 points at σ=0.01\sigma = 0.01, and all reach chance level (10%) at σ=1\sigma = 1. Simple Avg., Iso-C, and Fisher behave the same; we report all methods in [Table 12, Appendix F.10].

So the robustness comes from operating on the composed operator. It does not come from using calibration data or from sequential correction: at σ=0.1\sigma = 0.1, RegMean and RegMean++ (which use data) lose about 10 points, and sequential correction alone loses 9.3 points, while SLOA with the composed operator remains unaffected.

We once again thank the reviewer for their comments and positive scores, and we would be glad to discuss any further suggestions.

Reviewer_ZiVJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sequential Local Operator Alignment (SLOA), a training-free, calibration-based framework for merging multiple task-specific Transformers into a single model. The method addresses two limitations of conventional merging: (1) independently merging attention projection matrices may disrupt their underlying functional computations, and (2) merging layers independently ignores activation shifts caused by previously merged components. SLOA addresses these issues by merging composed QK and VO attention operators and sequentially aligning local operator outputs using activations from the partially merged model. It further introduces data-dependent regularization to stabilize alignment and SVD-based factorization with optional rank expansion to increase multi-task capacity. Experiments on CLIP ViTs, RoBERTa, and LoRA-adapted models, including LLaMA-2-7B, demonstrate improvements over several existing merging baselines. Ablation studies investigate the contributions of sequential alignment, QK/VO operator composition, and rank expansion.

Critical strengths
  1. The paper provides a clear theoretical motivation for merging composed QK/VO operators rather than individual attention projection matrices, showing how independent weight merging can disrupt functionally equivalent attention computations. This operator-level formulation also enables flexible rank expansion, offering a meaningful mechanism for controlling multi-task capacity.

  2. SLOA demonstrates consistent improvements over the evaluated baselines across vision and language tasks, including ViT, RoBERTa, and LoRA-adapted LLaMA-2-7B. Importantly, substantial gains are achieved without rank expansion (e.g., 80.06% vs. 74.51% for RegMean++ on 20-task ViT-B/32), indicating that the improvements are not solely attributable to increased model capacity. The experiments also suggest that the method is particularly effective as the number of merged tasks increases.

Critical weaknesses
  1. The core sequential correction mechanism closely resembles prior approaches that match outputs from partially merged models to those of the original experts. In particular, IterIS (CVPR 2025) already formulates a similar cross-state regression objective and iteratively updates merged-model activations. Although SLOA extends this principle to composed QK/VO operators, the paper does not sufficiently distinguish its sequential alignment contribution from these established ideas.

  2. For RoPE-based LLMs, the proposed QK operator formulation is not directly applicable, so the LLaMA-2-7B experiments rely on VO-only composition. However, without an isolated sequential alignment baseline, it remains unclear whether VO composition provides any additional benefit on these architectures or whether the reported gains primarily come from sequential correction. This limits the evidence for the effectiveness and generalizability of the proposed operator-level formulation.

Questions for authors

Prior work, such as IterIS and FeatCal, has already explored sequential correction using activations from partially merged models to match the original experts' outputs. Given this overlap, is the primary novelty of SLOA the formulation and merging of composed QK/VO operators, rather than the sequential alignment mechanism itself?

6 comments
AuthorsOpenReview ↗

Comment 1 of 2 (W1/Q: novelty vs. IterIS and FeatCal)

(Comment 1 of 2)

We thank the reviewer for their valuable comments. Below, we clarify how SLOA differs from IterIS and FeatCal, add a direct comparison with IterIS, and add the requested LLaMA ablation. We have marked all changes to the paper in blue.

W1 and Q: How SLOA differs from IterIS and FeatCal, and its primary novelty.

We thank the reviewer for pointing us to IterIS and FeatCal; the revision now discusses and credits both (Sections 3 and 5). Key differences:

[1] Chen, Hongxu et. al. "Iteris: Iterative inference-solving alignment for lora merging." In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4829-4838. IEEE, 2025.

IterIS [1] aligns the merged model to the experts using the merged model's own activations: it solves for each linear layer and repeats this over the whole model, each time refreshing the merged-model inputs. However, [1] does not perform the sequential step as in SLOA. We list key differences between the works below:

Sequential alignment vs. iterative refinement. IterIS iterates over the whole model. It starts from the expert’s own activations and solves all layers. It then runs inference on the resulting merged model, replaces the merged-side activations, and solves all layers again, for a fixed number of iterations (6–24 in the authors' configurations). SLOA, in contrast, makes a single pass in topological order at the operator level. Each operator is solved against the partially merged model, in which the upstream operators, including those earlier in the same block, are already final.

This distinction is important because SLOA explicitly accounts for the changes introduced by previously merged operators and not repeatedly refining all layers.

Operator-level vs. layer-wise alignment. IterIS merges only the linear layers, each separately. SLOA defines a local operator for every functional component, namely LayerNorm affine maps, linear maps, and the QK and VO operators of self-attention, and aligns each of them. Moreover, this operator-level formulation naturally supports rank expansion, allowing SLOA to improve accuracy beyond the original model rank without retraining, a capability not explored in IterIS.

Empirical comparison. To assess how these methodological differences affect performance, we compare SLOA vs. IterIS (20 tasks, fully fine-tuned experts):

MethodViT-B/32ViT-B/16
IterIS76.31 ± 0.0479.59 ± 0.26
SLOA, r = 64 (standard architecture)80.06 ± 0.1882.12 ± 0.26
SLOA, r = 12881.03 ± 0.1283.43 ± 0.09

We see that SLOA outperforms IterIS by 3.8-4.7 points on ViT-B/32 and by 2.5-3.8 points on ViT-B/16, already at the standard architecture.

[2] Gu, Yanggan, et. al. "FeatCal: Feature calibration for post-merging models." NeurIPS 2026.

While FeatCal [2] and SLOA both aim to improve merging performance, they take different approaches. Specifically, FeatCal is a post-hoc calibration method that starts from an existing merged model (e.g., obtained through Task Arithmetic) and adjusts its representations. In contrast, SLOA constructs the merged model directly from the expert models, without requiring an existing merged model as a starting point.

Also, the notion of sequential correction in FeatCal is different from SLOA’s. FeatCal proceeds sequentially across transformer blocks, but within each block, all operators are calibrated in parallel from a single forward pass through the uncalibrated block, so the MLP is fit to features produced by the initial (uncorrected) attention. SLOA’s sequential merging, in contrast, is operator-based: within each block, each operator is solved on inputs produced by the already-merged upstream operators in the same block, so the MLP is fit to the output of the attention that SLOA has already solved. This allows SLOA to correct merging error within each block, not only between blocks.

FeatCal could also be applied after SLOA. We are running SLOA + FeatCal and will report it during the discussion if compute allows, otherwise in the final version.

In Section 5 of the revision (Calibration-based training-free merging), we discuss how sequential/progressive merging is used in related work and how our operator-level alignment differs from it.


To reiterate, our main contribution is to view merging as aligning the functional operators that act on the input, rather than combining individual weight matrices. SLOA defines a local operator for each functional component of the transformer: LayerNorm affine maps, linear maps, and the QK and VO operators of attention, and solves each in closed form. Sequential alignment follows from the same view: since these operators act one after another, hence they are aligned one after another. Rank expansion is further building on top of that -- since a solved operator can be factorized at any rank and empirically, a higher rank gives a more accurate merge.

AuthorsOpenReview ↗

Comment 2 of 2 (W2: LLaMA ablation)

(Comment 2 of 2)

W2: VO composition adds benefits beyond sequential alignment on LLaMA-2-7B.

We thank the reviewer for suggesting this ablation. We have now added this comparison in the revision, and denoted it as SLOA-Seq, which uses the same sequential alignment, hyperparameters, and calibration data (N = 256 samples per task) as SLOA-VO, but merges each projection separately instead of the composed VO operator. Their performances are as follows:

MethodGSM8KHumanEvalIFEvalAvg.
RegMean++55.9529.2754.1646.46
SLOA-Seq (sequential alignment, no composition)56.3327.4458.9647.58
SLOA-VO, r = 12858.6128.0557.8648.17
SLOA-VO, r = 25659.5127.4458.9648.64
SLOA-VO, r = 38459.5127.4460.6349.19

We observe that sequential alignment alone improves over RegMean++ (47.58 vs 46.46), and VO composition improves further at the standard architecture (48.17). Again, composition also enables rank expansion: because the rank is chosen after the composed operator is solved, it can exceed d_k, which is not possible when the projections are merged separately. With rank expansion, SLOA-VO reaches 49.19 at r = 384 (+1.6 over SLOA-Seq). Thus, the gains on LLaMA do not come from sequential correction alone, and these results suggest that the framework remains effective on RoPE-based LLMs even when composition is applied to VO only.


Paper changes (in blue): W1: IterIS and FeatCal added to the related work (Section 5) and to Additional remarks (ii) (Section 3); contribution clarified in the introduction. W2: SLOA-Seq added to Table 4 and discussed in Section 4.3.

The revision now states that progressive matching is shared with ProDistill, IterIS, and FeatCal, and that SLOA's contribution is operator-level alignment. We would like to ask the reviewer if they think that this positions SLOA's novelty clearly for readers? Would the paper benefit from a more detailed technical comparison with IterIS in the appendix, or is there another analysis that would make the distinction clearer?

We thank the reviewer once again for the insightful comments, which helped us improve the paper, and we would be glad to discuss any further suggestions.

Thank you for the detailed rebuttal and additional experiments. The new comparisons with IterIS and the controlled ablations substantially strengthen the empirical validation of SLOA.

My remaining concern is primarily about the novelty of the Sequential Only component, which accounts for most of the performance improvement. Existing sequential merging methods have already explored progressive activation updates and cross-state correction. Could the authors clarify what is novel about the algorithmic design of SLOA's Sequential Only component compared with existing sequential merging strategies, particularly regarding its operator-wise processing within Transformer blocks?

Clarifying this distinction would help me better assess the methodological contribution and could support increasing my rating to 3 (Weak Acceptance).

Reviewer_sN5rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Sequential Local Operator Alignment (SLOA) to solve functional misalignment and error accumulation in training-free transformer model merging. They do this by sequentially aligning composed attention operators (QK, and VO) using calibration data along the partially merged model's execution path, followed by a factorization step. They empirically evaluate their method on Vision (ViT) and language models (RoBERTa, Llama-2) and show that SLOA improves multi-task accuracy while enabling controllable capacity through rank expansion.

Critical strengths
  1. By composing the local operators (QK, VO), they bypass parameterization ambiguity common in isolated weight matrices merging.
  2. The paper is well written and easy to follow
  3. Separating operator alignment from factorization allows the model to increase its attention head rank beyond d_k, providing a training-free lever to expand multi-task capacity
  4. Their method outperforms the baseline methods on 20 vision tasks and Language benchmarks such as GLUE, GSM8k, and HumanEval.
Critical weaknesses
  1. The RoPE embeddings in Llama-2-7b model causes problems with their core algorithm, causing them to fall back to isolated weight merging.
  2. The latency degrades when expanding rank of the attention heads in the ViT models.
Questions for authors
  1. What is the actual wall-clock merging time, peak memory usage for LLM models like Llama-2-7B? How does this scale with larger models?
  2. How does the throughput measure for LLMs when expanding rank?
  3. Expanding the head rank to non-standard dimensions breaks standard tile sizes in optimized kernles like FlashAttention. Did you benchmark using optimized inference engines or using native Pytorch?
5 comments
AuthorsOpenReview ↗

W1–W2: RoPE scope and rank expansion as an optional trade-off (1/2)

(Comment 1 of 2)

We thank the reviewer for the constructive feedback. Please find below detailed answers to the concerns and questions raised (we have marked all new ablations in the revised paper in blue).

W1: Extending operator composition to RoPE-based attention.

We thank the reviewer for this helpful comment and agree that extending our method to address QK factorization ambiguities in models with rotary position embeddings (RoPE) is a valuable direction. Under RoPE, the effective QK operator depends on the relative token position through WQRΔWK⊤W_Q R_\Delta W_K^\top, where RΔR_\Delta denotes the relative-position rotation. For full-rank projections, a change of basis WQ↦WQA, WK↦WKA−⊤W_Q \mapsto W_Q A, \ W_K \mapsto W_K A^{-\top} preserves this operator only when AA commutes with RΔR_\Delta for every relative position. For standard RoPE applied across the full head dimension with distinct frequencies, the admissible transformations are independent scaled rotations within the prescribed two-dimensional frequency subspaces, with matching rotations and reciprocal scalings on the query and key sides. This reduces the continuous QK ambiguity from O(d2)O(d^2) to O(d)O(d) degrees of freedom per head, where dd is the head dimension. Given this substantial reduction in QK reparameterization freedom, we expect misalignment arising from these ambiguities to be less pronounced in RoPE-based models, although this expectation remains to be validated empirically. The relative-position dependence also prevents directly applying our current formulation based on a single composed QK matrix. A straightforward extension that jointly optimizes WQW_Q and WKW_K to match the position-dependent scores would generally yield a nonconvex objective, to which our existing closed-form solution does not directly apply. We are actively investigating efficient ways to address the remaining RoPE-compatible ambiguities as follow-up work. Our operator composition still applies to the VO circuit, which RoPE does not affect. On LLaMA-2-7B, SLOA therefore does not fall back to isolated weight merging: VO is merged in composed form, and WQ\mathbf{W}_Q and WK\mathbf{W}_K are aligned sequentially on the partially merged model. Our new ablation (Table 4, Section 4.3; see also our response to Reviewer ZiVJ, (W2: LLaMA ablation)) shows that VO composition improves over sequential alignment alone (48.17 vs. 47.58), reaching 49.19 with rank expansion.

We can state the scope of composition under RoPE (VO only) in the introduction, but we wonder whether it would be better to discuss it in the limitations section (as done for now in revised manuscript). Do you have any preference, as a reader, between these two options to improve the paper’s scope?

W2: Rank expansion and latency

Thanks for this point. We think SLOA should be evaluated from two complementary perspectives:

(i) SLOA's main gains come at the original rank, with no change to inference cost and latency degradation. At r = d_k, the merged model has exactly the architecture of every baseline, and it already outperforms the strongest baseline in both vision and language:

BenchmarkStrongest baselineSLOA (r = d_k)Gain
ViT-B/32, 20 tasks74.51 (RegMean++)80.06+5.6
ViT-B/16, 20 tasks78.50 (Iso-CTS)82.12+3.6
RoBERTa, GLUE71.39 (RegMean++)79.06+7.7

(ii) Rank expansion then adds an option that weight-space merging does not offer: practitioners can exchange inference speed for further accuracy without any retraining. Table 19 in Appendix G.4 (ViT-B/16, 20 tasks) quantifies this trade-off:

Deployed modelAccuracy (%)Throughput (img/s)Latency (ms/img)
Best baseline (original architecture)78.50≈32570.307
SLOA, r = 64 (original architecture)82.1232610.307
SLOA, r = 12883.432712 (−17%)0.369
SLOA, r = 25683.741999 (−39%)0.500

Following this, we suggest a practical guideline:

  • Latency-critical deployment: use r = d_k. Inference cost is identical across all baselines, and SLOA is already the most accurate merge.
  • Accuracy-critical deployment: use r = 2·d_k, which captures most of the benefit of expansion.
  • LLMs: at r = 2·d_k, decoding on LLaMA-2-7B is only 5% slower (Q2).

At every rank, the result is still a single model, which is considerably cheaper to serve than the individual fine-tuned models it replaces.

We have added the discussion and this guidance for practitioners in Appendix G.4 - - thanks so much for this comment. It has made the paper more useful from a practitioner's perspective.

AuthorsOpenReview ↗

Q1–Q3: merge time and memory, throughput, and FlashAttention (2/2)

(Comment 2 of 2)

Q1: What is the actual wall-clock merging time and peak memory usage for LLM models like Llama-2-7B? How does this scale with larger models?

We measured merge time and peak GPU memory for SLOA and all baselines in our LLaMA-2-7B setting, which merges three LoRA experts (math, code, and instruction-following) into a single model. All methods were run under identical conditions: a full 32-layer merge, a single A100-40GB GPU, and N = 256 calibration samples per task for the calibration-based methods.

MethodMerge timePeak GPU memoryAvg.
Task Arithmetic / TIES / Weight Avg.< 1 min14–15 GB33.0–39.2
Iso-C / TSV-M / Iso-CTS / KnOTS-TIES21 / 59 / 70 / 78 min7–16 GB37.1–39.5
RegMean / RegMean++9 / 13 min10 GB44.5 / 46.5
SLOA-VO r = 128 / 256 / 38433 / 33 / 34 min37 GB48.2 / 48.6 / 49.2

SLOA takes longer than RegMean++ because it computes a composite operator per attention head. In absolute terms, this is a one-time cost for a single 40 GB GPU, faster than three of the four SVD-based baselines, and achieves the highest accuracy. Rank expansion has almost no effect on merging time. Merging time can also be reduced with multiple GPUs: within each block, the Gram statistics are sums over calibration samples and can be accumulated in parallel across devices, and the per-head operator can be solved independently of one another in multiple GPUs.

Scaling. SLOA processes one transformer block at a time, so merging time grows linearly with depth. Peak GPU memory does not grow with depth. Per block, cost grows with hidden size (Gram accumulation and the closed-form solution scale with width). For wider models than LLaMA-2-7B, memory usage can be reduced by using a smaller calibration batch, fewer calibration samples, or a shorter calibration sequence length, without changing the algorithm.

We have added the full table with merging time and peak GPU memory as Table 18 in Appendix G.3.

Q2: Throughput with rank expansion on LLMs.

Thanks for this question. Following the reviewer's comment, we measured the inference throughput of the merged LLaMA-2-7B at each rank. We report two speeds: prefill, the rate at which the model reads the input prompt, and decode, the rate at which it generates new tokens. All measurements use a single A100-40GB GPU in bf16 with PyTorch's built-in attention kernels (SDPA).

r_ovParametersPrefill tokens/sDecode tokens/sAvg.
128 (standard)6.74B13,97661848.17
2567.81B (+16%)12,264 (−12%)585 (−5%)48.64
3848.89B (+32%)10,879 (−22%)517 (−16%)49.19

Rank expansion widens only the V/O projections, so the overhead grows in proportion to the added width, and at r = 256, generation is only 5% slower. For comparison, serving the three experts separately requires either three models or per-query adapter routing; the rank-expanded SLOA model remains a single network.

We have added this table as Table 20 in Appendix G.4.

Q3: Non-standard head sizes and optimized kernels (FlashAttention).

We benchmarked in native PyTorch with its fused attention kernels (SDPA), not in serving engines. To answer the tile-size concern directly, we profiled which attention kernel actually runs:

  • r = d_k: The merged model has the standard architecture, so FlashAttention and any serving engine apply unchanged.
  • ViT: SLOA widens Q, K, V and O together, so all heads keep equal Q/K/V sizes (64 → 128 → 256), which are within the range that FlashAttention supports. With profiling, we confirm that FlashAttention runs at every rank.
  • LLaMA-2-7B: only V/O are widened (Q/K stay at 128 because of RoPE). For r > 128, PyTorch uses its fused memory-efficient kernel rather than FlashAttention; the throughput reported in Q2 already includes this.

Native support for expanded V/O heads in serving engines is an engineering extension that we leave to future work.

We add this in the Reproducibility Statement of the paper.


To reiterate, SLOA's gains hold at the standard architecture with no inference overhead, its merging cost is a one-time cost, and rank expansion is an accuracy–speed trade-off.

We thank the reviewer once again for the insightful comments, which helped us improve the paper, and we look forward to discussing any further suggestions.

Dear Reviewer,

Thank you again for your review. As the discussion period is coming to an end, we wanted to kindly check whether our response has addressed your concerns, in particular on RoPE-based attention, the latency of rank expansion, and merging time and memory on LLaMA-2-7B.

If anything remains unclear or you have further questions, we would be glad to discuss and provide additional results.

Best regards,

The Authors

Comments on the paper

1 comment
Area_Chair_T5pfOpenReview ↗

Author- Reviewer Discussion

Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.