acceptodds
Under review as a conference paper at ICLR 2027

SWIN: Seed Weight Integration for LLM Fine-Tuning

Abstract

Fine-tuning large language models exhibits high variance across random seeds, forcing repeated runs or expensive ensembles to reach top performance. We introduce SWIN (Seed Weight INtegration), a training-free, geometry-aware fusion method that integrates models fine-tuned with different seeds without increasing inference cost, motivated by the largely low-rank nature of fine-tuning updates and the emergence of complementary, task-relevant directions across seeds. SWIN first computes per-layer low-rank SVDs and forms basis-invariant projection operators to estimate a shared left/right consensus subspace. In that shared coordinate system it decouples each seed’s core update into orientation and scale via polar decomposition, aggregates orientations with a Procrustes projection and averages symmetric scales, and reconstructs a single low-rank update. By preserving matrix geometry and avoiding brittle element-wise averaging, SWIN effectively merges complementary, task-relevant directions while suppressing redundant spectral tails. Across mathematical reasoning, commonsense reasoning, code generation, and RL benchmarks, SWIN consistently matches or exceeds the best individual seed and outperforms element-wise baselines (e.g., improving MetaMath and AIME by up to +4.0 and +9.9 absolute points respectively) with no extra training or inference overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Fine-Tuning, Model Merge

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_8ouGOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SWIN, a training-free, geometry-aware fusion method for integrating models fine-tuned with different seeds. It uses two stages: (i) estimating a shared subspace through a closed-form consensus objective over projection operators, followed by (ii) structure-preserving aggregation using polar decomposition and an orthogonality-constrained projection, with no additional inference-time cost. The paper evaluates SWIN against other baseline approaches in both full fine-tuning and reinforcement learning settings and shows that it outperforms them.

Critical strengths
  • The paper makes a clear case for the method through four empirical observations about single-task model fusion: same-basin mode connectivity, low-rank redundancy, complementary specialization across seeds, and weak correspondence between the SVD bases of different seeds.

  • The method follows naturally from those observations. Rather than assuming that seed-specific SVD bases align, it first estimates a shared subspace through optimization, then uses polar decomposition to separate orientation from scale.

  • The evaluation covers several benchmarks and compares against a range of baselines. The proposed method achieves the best results on most of the reported tasks.

Critical weaknesses
  1. The empirical evidence does not yet support some of the paper’s broader claims about seeds. Figure 1 shows that one pair of same-task checkpoints can be linearly interpolated without a substantial GSM8K/MATH accuracy drop. That is useful evidence, but it does not establish that checkpoints from different seeds generally share a low-loss basin: the figure reports accuracy rather than loss, and it is unclear how the pair was selected. Likewise, Figure 2 uses two seeds to motivate the claim that no seed consistently dominates across subtasks. Results across more seeds and seed pairs would make both claims more convincing.

  2. The paper could do more to distinguish SWIN from related work. Qin et al. (EMNLP 2022 https://arxiv.org/abs/2210.14102) studied mode connectivity in fine-tuned pretrained language models and should be discussed alongside the paper’s basin claim. Task Singular Vectors (Gargiulo et al., CVPR 2025 https://arxiv.org/abs/2412.00081) is also relevant to the SVD-based method. It would help establish the contribution with a clearer account of what carries over from these methods and what is specific to SWIN.

Questions for authors
  1. How were the two seeds in Figure 1 selected? Could you show interpolation results for more seed pairs, including training or validation loss as well as GSM8K/MATH accuracy?

  2. How were the seeds in Figure 2 selected? Across a larger pool, is it still true that no individual seed consistently outperforms the others on the subtasks?

  3. Lines 205–208 interpret low maximum similarity between vectors in �0��0� and �1��1� as rotational ambiguity. How do you rule out the alternative that the two sets of vectors span substantially different subspaces?

  4. Why is the RL evaluation limited to DAPO training of DeepSeek-R1-Distill-Qwen-1.5B? Would the conclusions hold for another model or RL training setup?

  5. Could you clarify which components of SWIN overlap with the methods described in Appendix D, and which are new? A comparison with Task Singular Vectors, or an explanation of why it is not an appropriate baseline for same-task fusion, would be helpful. (https://arxiv.org/abs/2412.00081)

  6. Could you revise Figure 2’s presentation? The small arrows beneath the x-axis are unexplained, and the lines identified in the legend as using the left and right y-axes are hard to locate. Lines 162–168 also need formatting fixes; Tables 2 and 15 should mark tied or best results consistently.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8ouG

Thanks for your constructive feedback, we carefully address your concerns below.

W1, Q1 & Q2. Figures 1 and 2 each rely on one pair of seeds, Figure 1 reports accuracy rather than loss, and the seed selection is unclear. How were the seeds selected, and do the claims hold for more seed pairs (including loss) and a larger pool of seeds?

Thank you for this suggestion. We agree that the evidence in Figures 1 and 2 is based on one pair of seeds each, and that Figure 1 reports accuracy rather than loss. We address this from two aspects.

(1) Connectivity (Figure 1). Figure 1 interpolates between two LLaMA2-7B seeds fine-tuned on MetaMathQA, randomly selected from the three seeds used in Table 3; this pair is also included in the ten pairs below. To remove the dependence on a single pair, we now interpolate between all ten pairs of five LLaMA2-7B seeds fine-tuned on MetaMathQA (42, 87, 100, 13, and 57; the same seeds as in our response to Reviewer TFiH) at five equally spaced points (γ=0,0.25,0.5,0.75,1\gamma = 0, 0.25, 0.5, 0.75, 1 in Eq. 2). At each point, we measure the training loss (token-level negative log-likelihood of the responses) on 2,000 randomly sampled MetaMathQA training examples and evaluate GSM8K and MATH accuracy.

Seed pair (γ=1 / γ=0)Metricγ=10.750.50.25γ=0
42 / 87Loss (×100)5.265.275.275.225.17
GSM8K66.168.668.268.267.0
MATH20.521.320.220.420.8
42 / 100Loss (×100)5.265.525.355.295.25
GSM8K66.166.067.768.566.6
MATH20.520.421.120.820.2
42 / 13Loss (×100)5.265.525.846.106.25
GSM8K66.167.667.068.167.2
MATH20.520.820.319.619.1
42 / 57Loss (×100)5.265.515.836.086.22
GSM8K66.166.567.867.667.1
MATH20.521.121.020.218.5
87 / 100Loss (×100)5.175.405.295.265.25
GSM8K67.066.667.567.766.6
MATH20.820.420.520.920.2
87 / 13Loss (×100)5.175.445.806.096.25
GSM8K67.067.668.868.167.2
MATH20.820.220.319.919.1

In every pair, the training loss ($\times$100) at the interior points exceeds the line connecting the two endpoint losses by at most 0.30, about 5% of the endpoint loss, and GSM8K and MATH accuracy at the interior points is never more than 0.3 points below the lower of the two endpoints. Thus, all evaluated seed pairs are linearly connected without a substantial loss barrier or accuracy drop, consistent with Figure 1. The table shows six of the ten pairs; Appendix J.6 reports all ten.

(2) Seed specialization (Figure 2). Figure 2 shows seeds 87 and 100 of LLaMA3-8B fine-tuned on Commonsense-170k, randomly selected from the three seeds in Table 4. Table 4 already reports all three seeds: seed 42 has the highest average accuracy, seed 87 leads on HellaSwag, and seed 100 leads on WinoGrande and ARC-C, so the best seed varies across subtasks among these three. To test this on a larger pool, we fine-tune two additional LLaMA3-8B seeds (13 and 57) on Commonsense-170k under the same settings, and report the accuracy of all five seeds on each subtask in the same format as Table 4, where the results of seeds 42, 87, and 100 are taken from Table 4 and bold marks the best seed on each subtask, including ties.

SeedBoolQPIQASIQAHellaSWinoGARC-CARC-EOBQAAvg.
Full FT4275.789.179.894.586.580.191.088.285.6
Full FT8775.788.079.495.185.578.590.186.484.8
Full FT10075.488.479.293.386.680.490.184.884.8
Full FT1375.788.279.595.285.678.690.387.085.0
Full FT5775.588.979.593.686.880.790.786.485.3

In the pool of five seeds, the best seed still varies across subtasks: seeds 42, 13, and 57 each rank first on at least one subtask, seed 87 ties for first on BoolQ, and no seed ranks first on all eight. Seed 42 ranks first on five subtasks and has the highest average, while seed 13 leads on HellaSwag and seed 57 on WinoGrande and ARC-C.

Figure 1 will retain the original seed pair. Section 3.1 will summarize the ten-pair results and refer to Appendix J.6 for the summary figure and full loss and accuracy curves. A figure with all evaluated seeds will be added to the appendix and referenced in the caption of Figure 2. We will also state how the seeds were selected, and limit both claims to the evaluated models and seeds: the seed pairs are connected by linear paths without a substantial loss barrier, and the best seed varies across subtasks.

AuthorsOpenReview ↗

Response to Reviewer 8ouG - Continue

W2 & Q5. Clarify the relation to Qin et al. (2022) and Task Singular Vectors (TSV): which components of SWIN overlap with prior methods and which are new? Compare with TSV or explain why it is not an appropriate baseline.

Thank you for pointing us to these works. Our contribution is a merging scheme for same-task seeds that selects a fixed-rank consensus subspace capturing the directions shared across seeds, and fuses the seeds in it with a new rule that aggregates orientation and scale separately. We clarify the relation to these works from three aspects.

(1) Mode connectivity (Qin et al., 2022). Qin et al. show for T5 that fully fine-tuned minima are linearly connected under different data orders and initialization seeds, whereas minima of different tasks are not. Our Figure 1 and Section 3.2 are consistent with these findings and extend the setting to LLMs fine-tuned on generative reasoning tasks. Our method builds on this property, and we will cite their work in Section 3.1 and present Figure 1 as consistent evidence in our setting.

(2) What carries over and what is new. Like most merging methods, SWIN operates on task vectors (deltas), as in Task Arithmetic, Model Soup, TIES, and DARE, and computes layer-wise SVDs of the updates, as in TSV, KnOTS, and Core Space. Its two core stages differ from these methods as follows.

StagePrior methodsSWIN
Shared subspaceUnion of the task subspaces at full rank, for a lossless representation (Core Space); SVD of the concatenated updates (KnOTS)Fixed-rank consensus of the seed subspaces, keeping the directions shared across seeds
Fusion in the shared spaceExisting rules such as TA or TIES on the aligned components (KnOTS, Core Space); orthogonalized concatenation (TSV)Procrustes mean of the polar orientations and average of the PSD scales

Both stages are designed for checkpoints of the same task, whereas KnOTS, Core Space, and TSV target merging models of different tasks (and KnOTS and Core Space are designed for LoRA updates). In particular, TSV uses Procrustes to decorrelate the singular vectors of different tasks, whereas SWIN uses it to average the orientations of seeds of the same task.

(3) TSV as a baseline. As discussed in (2), TSV is designed to reduce interference between different tasks. For seeds of the same task, the singular vectors largely span the same subspace (Q3), so for SVD-based merging the main difficulty is basis misalignment rather than task interference (Section 3.2). TSV is still applicable to this setting, and Table 14 in Appendix E compares it with SWIN on the same three LLaMA2-7B/MetaMathQA checkpoints: TSV reaches 68.1/19.8 on GSM8K/MATH, and SWIN reaches 71.0/22.1. We will move this comparison into the main text.

We will add this discussion to Section 2 and Appendix D in the revised version.

AuthorsOpenReview ↗

Response to Reviewer 8ouG - Continue

Q3. How do you rule out that the low vector similarity (lines 205–208) reflects different subspaces rather than rotation?

Thank you for this question. Reviewer V7dE raised the same question (Q1), and we summarize the analysis here. We agree that Figure 3 alone does not rule out this alternative, since low row-wise maxima can arise either from rotations within a shared subspace or from different subspaces. To distinguish the two, we compute the principal angles between the retained subspaces of each pair of seeds, whose cosines are the singular values of Ui⊤UjU_i^\top U_j, for all projection matrices of three models (LLaMA2-7B, LLaMA3-8B, and Gemma-2-9B, each fine-tuned on MetaMathQA with three seeds). The table uses the retained rank r=256r=256 of Figure 3. Two random rr-dimensional subspaces of Rm\mathbb{R}^m have mean squared cosine r/mr/m, which serves as a reference. We report medians over matrices and seed pairs, with 10th–90th percentiles in brackets.

ModelBest vector match (Fig. 3 statistic)Mean cos⁡2θ\cos^2\theta / random baselineMedian principal angle
LLaMA2-7B0.19 [0.15, 0.26]10.7×10.7\times [8.0, 20.7]35∘35^\circ [18, 47]
LLaMA3-8B0.23 [0.18, 0.34]11.7×11.7\times [3.0, 35.3]28∘28^\circ [15, 40]
Gemma-2-9B0.15 [0.13, 0.22]8.5×8.5\times [4.1, 20.1]41∘41^\circ [20, 54]

Across the three models, each singular vector's best match in the other seed has cosine 0.15–0.23, yet the two subspaces overlap 8.5–11.7$\times$ more than random subspaces. The same holds at the 50% rank used by SWIN: the median principal angle is 19∘19^\circ–27∘27^\circ, well below the 45∘45^\circ or more expected for random subspaces at this rank. We also examine the eigenvalues of the averaged projector used by Stage 1: a direction present in all three seeds has an eigenvalue near 1, and a direction present in only one seed (and orthogonal to the other seeds' subspaces) has about 1/3. At 50% rank, the median retained eigenvalue is 0.89–0.95, so most retained directions are largely shared by all seeds.

The seeds therefore span largely shared subspaces in different bases, so the low maxima in Figure 3 mainly reflect basis ambiguity. The principal angles are not zero, so genuine subspace differences also exist; Stage 1 handles them by taking the consensus of the seed projectors, which is invariant to the choice of basis within each subspace.

We will revise lines 205–208 to state that Figure 3 measures the correspondence between individual singular vectors, and support the rotational-ambiguity interpretation with the principal-angle analysis in Section 3.5, with details in the appendix.

Reviewer_V7dEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SWIN (Seed Weight Integration), a training-free, geometry-aware model merging method designed to integrate language models fine-tuned on the same task with different random seeds. The method first estimates shared low-rank subspaces using basis-invariant projection operators, then performs structure-preserving fusion through polar decomposition, Procrustes-based orientation aggregation, and scale averaging. Experiments across mathematical reasoning, commonsense reasoning, code generation, and reinforcement learning demonstrate improvements over individual fine-tuned models and existing merging baselines, including Model Soup, TIES, DARE, and KnOTS. The paper also provides analyses of low-rank redundancy, seed-specific complementarity, and the contributions of different fusion components.

Critical strengths
  1. The paper investigates same-task, multi-seed model merging, an important setting that has received relatively less attention than multi-task merging.
  2. This paper introduces a structured fusion framework based on consensus subspace estimation, polar decomposition, and Procrustes aggregation, providing an alternative to conventional element-wise weight merging.
  3. The proposed method achieves improvements across multiple model architectures and tasks, including mathematical reasoning, commonsense reasoning, code generation, and RL-trained models, without additional inference overhead.
Critical weaknesses
  1. Weak alignment between individual SVD basis vectors does not necessarily imply destructive interference in direct weight averaging. The connection between basis misalignment and the proposed fusion mechanism requires stronger evidence.
  2. The individual contributions of Procrustes orientation aggregation, scale averaging, and update magnitude are not sufficiently isolated.
  3. Although SWIN introduces no additional inference cost, its offline merging requires large per-layer SVD operations, without quantitative analysis of merging time or peak memory consumption.

If the authors can address the above questions, I will consider raising the score.

Questions for authors
  1. How does the observed weak correspondence between individual SVD basis vectors demonstrate destructive interference in element-wise weight averaging? Could the authors provide principal-angle or projection-based analyses to distinguish basis ambiguity from genuine subspace misalignment?

  2. Could the authors provide more fine-grained ablations to separately evaluate Procrustes orientation aggregation, positive-semidefinite scale averaging, and the effects of update magnitude?

  3. What are the wall-clock merging time and peak GPU memory requirements of SWIN compared with Model Soup, TIES, DARE, and KnOTS, particularly when retaining 50% of the full matrix rank?

5 comments
AuthorsOpenReview ↗

Response to Reviewer V7dE

Thanks for your constructive feedback, we carefully address your concerns below.

W1 & Q1. Weak alignment between individual SVD basis vectors does not by itself show destructive interference in element-wise weight averaging. Provide stronger evidence, such as principal-angle or projection-based analyses, to separate basis ambiguity from genuine subspace misalignment.

Thank you for these comments, and we apologize for the confusion caused by our wording.

Figure 3 and element-wise averaging. The reviewer is correct that the weak alignment between individual SVD basis vectors (Figure 3) does not by itself demonstrate destructive interference in element-wise weight averaging, since entry-wise averaging of the full updates ΔWi\Delta W_i does not involve the SVD bases. Figure 3 shows the rotational ambiguity of per-seed SVD bases (Sections 3.5 and 4.3; see also Appendix J). Our concern was that combining seeds without alignment can weaken information, which applies directly when seeds are combined through their SVD factors: even the sign ambiguity of the SVD, uσv⊤=(−u)σ(−v)⊤u\sigma v^\top=(-u)\sigma(-v)^\top, allows two seeds to represent the same component with opposite vectors, whose average vanishes. Sections 4.4 and 5.1 extended this wording to element-wise averaging without making this distinction.

To provide the stronger evidence requested, we follow the reviewer's suggestion in Q1 and examine, from the perspectives of principal angles and projections, whether the weak vector alignment comes from different subspaces or from different bases of the same subspace.

(1) Principal-angle analysis. For retained orthonormal bases Ui,UjU_i,U_j of two seeds, the singular values of Ui⊤UjU_i^\top U_j are the cosines of the principal angles between the two subspaces, and we treat the right bases in the same way. We compute them for all projection matrices of the three models (224 for LLaMA2-7B and LLaMA3-8B, 294 for Gemma-2-9B) at the retained rank r=256r=256 of Figure 3. Two random rr-dimensional subspaces of Rm\mathbb{R}^m have mean squared cosine r/mr/m, which serves as a reference. We report medians over matrices and seed pairs, with 10th–90th percentiles in brackets.

ModelBest vector match (Fig. 3 statistic)Mean cos⁡2θ\cos^2\theta / random baselineMedian principal angle
LLaMA2-7B0.19 [0.15, 0.26]10.7×10.7\times [8.0, 20.7]35∘35^\circ [18, 47]
LLaMA3-8B0.23 [0.18, 0.34]11.7×11.7\times [3.0, 35.3]28∘28^\circ [15, 40]
Gemma-2-9B0.15 [0.13, 0.22]8.5×8.5\times [4.1, 20.1]41∘41^\circ [20, 54]

Across the three models, the best match of each singular vector in another seed has a median cosine of only 0.15–0.23, yet the two subspaces overlap 8.5–11.7$\times$ more than random subspaces. The same holds at the 50% rank used by SWIN: the median principal angle is 19∘19^\circ–27∘27^\circ, well below the 45∘45^\circ or more expected for random subspaces at this rank.

(2) Projection-based analysis. We also examine the eigenvalues of the averaged projector Pˉ=1N∑iUiUi⊤\bar P=\frac{1}{N}\sum_i U_iU_i^\top used by Stage 1. A direction present in all seeds receives an eigenvalue near 1, and a direction present in only one seed (and orthogonal to the other seeds' subspaces) receives about 1/31/3. At 50% rank, the median retained eigenvalue is 0.89–0.95 across the three models, so most consensus directions are largely shared by all three seeds.

Both analyses lead to the same conclusion. The weak vector correspondence mainly reflects basis ambiguity: individual singular vectors do not match across seeds, whereas the subspaces they span are largely shared. Genuine subspace differences also exist, since the principal angles are not zero, but they are limited. This confirms the description in Section 3.2: the seeds express largely the same subspace in different coordinate systems.

Connection to the fusion mechanism. This observation motivates both stages of SWIN. Because fine-tuning updates are largely low-rank (Section 3.3), SWIN fuses the seeds within a shared low-rank subspace. Since individual singular vectors cannot be matched across seeds, this subspace cannot be built by selecting or matching basis vectors, so Stage 1 estimates it from basis-invariant projectors. Since the subspaces are similar but not identical, Stage 1 takes their consensus rather than the subspace of any single seed. Expressing all seeds in this common basis yields cores Si=U∗⊤ΔWiV∗S_i=U^{\ast\top}\Delta W_iV^{\ast} that are directly comparable, which allows Stage 2 to aggregate them with structure-preserving operations. The benefit of aggregating these cores by orientation and scale is confirmed by the controlled ablation in W2 & Q2.

In the revised version, we will add these analyses to Section 3.5 and the appendix, and revise Sections 4.4 and 5.1 to use "cancellation" and "destructive interference" only for combining seeds in misaligned coordinates.

AuthorsOpenReview ↗

Response to Reviewer V7dE - Continue

W2 & Q2. The contributions of Procrustes orientation aggregation, scale averaging, and update magnitude are not isolated; provide finer ablations of these three components.

Thank you for this suggestion. We agree that Table 11 did not isolate these components, since it replaced all of Stage 2 with vanilla averaging at once. We now provide the requested fine-grained ablation. Starting from Stage 1 + Vanilla Avg (the first row of Table 11), which uses the mean of the projected seed cores (the averaged core), we add Procrustes orientation aggregation, PSD scale averaging, and update magnitude one at a time, so that each step changes a single component and the last step gives full SWIN. All configurations use the same checkpoints, Stage 1 bases, rank, merging hyperparameters as in Table 3, and decoding settings. Since all four configurations are re-run with the same evaluation pipeline, the first row differs from Table 11 (68.9/21.0); we will update Table 11 with these numbers. In the table, Δ\Delta is the change relative to the previous row.

ConfigurationGSM8KMATHΔ\Delta GSM8KΔ\Delta MATH
Stage 1 + Vanilla Avg66.720.4––
+ Procrustes orientation aggregation68.520.9+1.8+0.5
+ PSD scale averaging69.421.2+0.9+0.3
+ Update magnitude (= SWIN)71.121.6+1.7+0.4

(1) Procrustes orientation aggregation. The orientation of the averaged core is obtained after the cores are summed, so it is weighted by the seeds' scales. This step replaces it with the orthogonal matrix closest to all seed orientations (Eq. 11), which is determined by the seed orientations alone, and keeps the scale unchanged. It improves accuracy by 1.8/0.5 points (GSM8K/MATH).

(2) PSD scale averaging. The scale of the averaged core also depends on how much the seed orientations disagree. This step replaces it with the average of the seeds' own PSD scale factors, rescaled to the norm of the previous scale, so that only the shape of the scale changes while the orientation and the norm are kept fixed. It improves accuracy by a further 0.9/0.3 points.

(3) Update magnitude. Averaging the cores directly can shrink the merged update when the seed orientations disagree, whereas in SWIN the norm of the merged core is that of the averaged seed scale and does not depend on orientation disagreement. This step removes the rescaling of step (2), so that only the norm changes, which gives full SWIN. It improves accuracy by a further 1.7/0.4 points.

Added in this order, all three components contribute, with Procrustes orientation aggregation and the update magnitude giving the largest gains, and together they account for the 4.4/1.2-point gain of SWIN over Stage 1 + Vanilla Avg.

In the revised version, we will extend Table 11 with this ablation and report the contribution of each component in Section 5 and Appendix C.

AuthorsOpenReview ↗

Response to Reviewer V7dE - Continue

W3 & Q3. The cost of the offline merge is not quantified; report the merging time and peak GPU memory of SWIN against Model Soup, TIES, DARE, and KnOTS at 50% rank.

Thank you for this constructive comment. We agree that the merging cost should be quantified. We first note that this cost is incurred only once, offline: SWIN outputs a single checkpoint with the original architecture, so its inference and storage costs are the same as those of one model. We measured the wall-clock merging time and peak GPU memory of SWIN at 50% of full matrix rank and of Model Soup, TIES, DARE, and KnOTS on the same three LLaMA2-7B checkpoints, using a single NVIDIA H100 (80 GB). The results are shown in the following table.

MethodWall time (s)Peak GPU memory (GiB)
Model Soup12.52.47
DARE12.52.47
TIES150.83.69
KnOTS-TIES14574.64
SWIN (50% rank)17842.78

All methods use their configurations from the paper, process one weight matrix at a time on the same GPU, and are timed excluding disk I/O. From the table, we observe the following.

(1) Merging time. Element-wise methods take seconds to minutes, while the two SVD-based methods take tens of minutes. For each weight matrix, SWIN computes one SVD per seed, two decompositions for the consensus bases, and N+1N{+}1 SVDs of r×rr\times r matrices for the polar factors and the Procrustes step. Merging the three LLaMA2-7B checkpoints at 50% rank takes 1784 s, which is 22% longer than KnOTS-TIES (1457 s), the other SVD-based method.

(2) Peak memory. Since every method processes one weight matrix at a time, peak GPU memory is determined by the largest layer rather than by the model size. SWIN's peak of 2.78 GiB is close to that of Model Soup (2.47 GiB) and below those of TIES (3.69 GiB) and KnOTS-TIES (4.64 GiB); for example, KnOTS decomposes the concatenation of all seeds' updates, whereas SWIN decomposes each seed's update separately.

We will add this analysis to the appendix of the revised version.

Reviewer_TFiHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SWIN, a training-free method for merging large language models that are fine-tuned on the same task using different random seeds. The proposed method finds a good single projector to approximate the projectors created by different seeds. After that, for the core matrix, the polar decomposition and the orthogonality-constrained projection are applied to get the single core matrix for the merge. Experiments cover tasks including mathematical reasoning, commonsense reasoning, code generation, and reinforcement learning and show outstanding performance of the proposed method.

Critical strengths
  1. The paper proposed an offline method to merge a few models trained on the same task with different seeds. The method estimated the shared subspace for checking the projectors and performed structure-preserving aggregation.
  2. Results show improvements across several models and tasks. Ablation studies are provided.
Critical weaknesses
  1. In Stage 1, the proposed method finds the shared subspace by finding the best projector that approximates all projectors. So, retained singular directions all receive equal weight, and the consensus objective favors directions shared across seeds. It remains unclear how well this preserves useful seed-specific information. A comparison with weighted subspace estimation would help justify this choice.

  2. There is a similar issue in Stage 2. Orientation aggregation projects the mean of orientations into the orthogonal matrix space. The scales for each direction are discarded in the orientation. Therefore, a low-importance direction also contributes the same level of importance during the aggregation. The paper should clarify when this aggregation is appropriate and whether scale-weighted alternatives improve robustness.

Questions for authors
  1. How sensitive is Stage 2 to using the left rather than the right polar decomposition?
  2. Are the reported gains consistent across different groups of random seeds? It would be helpful to repeat the merging experiment with several seed combinations.
  3. The presentation could be more focused on the proposed method. Section 3 could retain the key motivations while shortening background discussions and avoiding repetition in other parts of the paper.
5 comments
AuthorsOpenReview ↗

Response to Reviewer TFiH

Thanks for your constructive feedback, we carefully address your concerns below.

W1. Stage 1 weights all retained directions equally; does this lose useful seed-specific information? Compare with weighted subspace estimation.

Thank you for this comment. We agree that the consensus objective favors directions shared across seeds, and we address how this affects useful seed-specific information from two aspects.

(1) How seed-specific information is preserved. Seed-specific information within the shared directions is preserved: Stage 1 only determines the shared subspace, and each seed's own update along it, including its magnitudes, is retained in the cores Si=U∗⊤ΔθiV∗S_i=U^{\ast\top}\Delta\theta_iV^\ast (Eq. 9) and included in the fusion of Stage 2. Directions that appear in only one seed are kept only as the rank allows: a direction contained in kk of the NN seeds (and orthogonal to the others) has an eigenvalue of about k/Nk/N in Pˉ\bar P, so at a fixed rank it is ranked after the shared directions. A direction found in only one seed may reflect useful specialization or seed-specific noise, which cannot be distinguished from that seed alone, whereas a direction found in several seeds is reproducible.

(2) Comparison with weighted subspace estimation. Following your suggestion, we compare SWIN with a singular-value-weighted Stage 1. Weighting by singular values allows a strong direction of a single seed to rank above weakly shared directions, so this comparison tests whether keeping such directions improves the merged model. For the rank-rr SVD of seed ii's update, let UiU_i and ViV_i contain the retained singular vectors and σij\sigma_{ij} the corresponding singular values. We set

wij=rσij∑k=1rσik,Di=diag⁡(wi1,…,wir),w_{ij}=\frac{r \sigma_{ij}}{\sum_{k=1}^{r}\sigma_{ik}}, \qquad D_i=\operatorname{diag}(w_{i1},\ldots,w_{ir}),

so that the weights sum to rr within each seed, and take the top rr eigenvectors of ∑iUiDiUi⊤\sum_i U_iD_iU_i^\top and ∑iViDiVi⊤\sum_i V_iD_iV_i^\top as the shared bases. Uniform weighting (SWIN) corresponds to Di=IrD_i=I_r. Both variants use the same LLaMA2-7B/MetaMathQA checkpoints, rank, Stage 2, merging hyperparameters as in Table 3, and decoding settings. To directly measure how well useful seed-specific information is preserved, we also report accuracy on the questions that exactly one of the three seeds answers correctly before merging.

Stage 1Overall accuracy GSM8KOverall accuracy MATHQuestions solved by one seed only GSM8KQuestions solved by one seed only MATH
Uniform (SWIN)71.021.544.231.5
Singular-value weighted70.421.543.632.4

Relative to uniform Stage 1, singular-value weighting changes overall accuracy by −0.6-0.6/0.00.0 points (GSM8K/MATH) and accuracy on the single-seed questions by −0.6-0.6/+0.9+0.9 points. Thus, singular-value weighting shows no consistent advantage in preserving seed-specific information, and we keep uniform weighting as the default.

We will add this comparison and the discussion of the weighting choice to Section 4.3 and Appendix C in the revised version.

AuthorsOpenReview ↗

Response to Reviewer TFiH - Continue

W2. Stage 2 aggregates orientations with equal weight regardless of scale; when is this appropriate, and do scale-weighted alternatives improve robustness?

Thank you for this comment. We agree that the Procrustes step weights the orientations of all directions equally, and we address when this is appropriate and whether scale weighting helps from three aspects.

(1) Scales are separated, not discarded. The polar decomposition Si=OiHiS_i=O_iH_i separates each core into an orientation OiO_i and a scale HiH_i. The Procrustes step aggregates the orientations only, while the scales are averaged separately into Hˉ=1N∑iHi\bar H=\frac1N\sum_iH_i, and the merged core is OPHˉO_P\bar H. Each direction therefore enters the merged update with its averaged scale, so a low-scale direction contributes little to the merged update. However, since the fused orientation is estimated jointly over all directions, noisy orientations of low-scale directions can still affect it, and we test whether this affects the merged model in (3).

(2) When equal weighting is appropriate. SWIN targets checkpoints fine-tuned on the same task from the same initialization. Since no single seed ranks first on all subtasks (Figure 2), SWIN treats the seeds symmetrically in orientation aggregation, so that a seed with larger updates does not dominate the fused orientation, while the importance of each direction is still reflected in the merged update through the averaged scale, as discussed in (1). Whether the scale of a direction also indicates a seed's specific strength, in which case scale weighting would help, is an empirical question that we test in (3). Scale weighting can also be more appropriate when the merged models are fine-tuned on different tasks, since the scale of each direction may then reflect task-specific importance.

(3) Comparison with a scale-weighted alternative. Following your suggestion, we compare SWIN with a scale-weighted orientation rule, Ow∈arg⁡min⁡O⊤O=Ir∑i∥(O−Oi)Hi1/2∥F2,O_{\mathrm{w}}\in\arg\min_{O^\top O=I_r}\sum_i \bigl\Vert(O-O_i)H_i^{1/2}\bigr\Vert_F^2, which penalizes disagreement more along directions with larger scale. Its solution is the orthogonal polar factor of ∑iOiHi=∑iSi\sum_iO_iH_i=\sum_iS_i, i.e., the orientation of the averaged core, whereas SWIN uses the orthogonal polar factor of ∑iOi\sum_iO_i. Both rules use the same averaged scale Hˉ\bar H, so the two merged cores have identical singular values and differ only in orientation. We measure robustness by how much the merged model is affected by low-scale directions, comparing the two rules as more such directions are retained (higher rank). Specifically, we compare them on the three seeds of Table 3 at retained ranks of 25%, 50% (default), 75%, and 100% of the full rank, with the same Stage 1, merging hyperparameters as in Table 3, and decoding settings. All experiments in this response use the same checkpoints and evaluation pipeline, but different tables come from separate evaluation runs, so each comparison is made within one table.

SettingEqual weighting (SWIN) GSM8KEqual weighting (SWIN) MATHScale weighting GSM8KScale weighting MATH
Rank 25%68.920.569.120.7
Rank 50% (default)71.821.871.921.8
Rank 75%71.321.371.321.6
Full rank71.021.471.321.6

Across all retained ranks, scale weighting changes accuracy by at most 0.3 points relative to equal weighting, including at full rank where the most low-scale directions are retained. Thus, in the same-task setting, scale weighting does not noticeably improve robustness. We therefore keep equal weighting as the default, which treats the seeds symmetrically, and will note scale weighting as an alternative for merging models fine-tuned on different tasks.

We will add this discussion and comparison to Section 4.4 and Appendix C in the revised version.

AuthorsOpenReview ↗

Response to Reviewer TFiH - Continue

Q1. How sensitive is Stage 2 to the left versus right polar decomposition?

Thank you for this question. We address it from two aspects.

(1) Relation between the two forms. Each core admits a right and a left polar decomposition with the same orientation, Si=OiHiR=HiLOiS_i=O_iH_i^R=H_i^LO_i, where the right scale acts on the shared input coordinates and the left scale on the shared output coordinates. Applying the left-polar rule to a weight matrix is equivalent to applying the right-polar rule to its transpose, so neither side is preferred a priori, and we follow the standard right form (Eq. 10). The two rules give the same merged core when the seed orientations agree, and differ only through the disagreement among them.

(2) Empirical sensitivity. We compare the two rules on the LLaMA2-7B and LLaMA3-8B checkpoints fine-tuned on MetaMathQA (three seeds each), with the same Stage 1 bases, rank, and decoding settings, computing the Procrustes orientation once per layer and reusing it for both rules.

Stage 2 fusionLLaMA2-7B GSM8KLLaMA2-7B MATHLLaMA3-8B GSM8KLLaMA3-8B MATH
Right polar (SWIN)71.221.681.633.3
Left polar70.521.781.533.6

Left-polar fusion changes accuracy by −0.7-0.7/+0.1+0.1 points (GSM8K/MATH) on LLaMA2-7B and by −0.1-0.1/+0.3+0.3 points on LLaMA3-8B relative to right-polar fusion. Thus, Stage 2 is not sensitive to this choice, and we keep the right polar decomposition as the default.

We will add this discussion and comparison to Appendix C in the revised version.

Q2. Are the gains consistent across different seed combinations?

Thank you for this suggestion. Table 3 reports the three reference seeds and Table 9 varies the number of merged checkpoints, but neither varies which seeds are merged. To address this, we use five LLaMA2-7B checkpoints fine-tuned on MetaMathQA with different random seeds (42, 87, 100, 13, and 57) and the same training recipe, and evaluate SWIN on all ten three-seed combinations with the same merging hyperparameters as in Table 3. For each combination, we compare SWIN with the best of its three seeds on each benchmark, which is a strong reference since the best seed is selected on the test set separately for each benchmark. All numbers in this table, including the single seeds, come from one evaluation run, so they differ slightly from Table 3.

42871001357GSM8K best seedGSM8K SWINMATH best seedMATH SWIN
Single-seed accuracy:
66.1/20.567.0/20.866.6/20.267.2/19.167.1/18.5
✓✓✓67.071.020.821.5
✓✓✓67.270.420.821.3
✓✓✓67.170.220.821.5
✓✓✓67.270.720.520.7
✓✓✓67.170.020.521.5
✓✓✓67.270.120.520.8
✓✓✓67.270.020.821.1
✓✓✓67.171.220.821.8
✓✓✓67.270.720.821.3
✓✓✓67.270.020.221.8
Mean over combinations67.170.420.621.3
SWIN better than best seed10 / 1010 / 10

SWIN improves over the best single seed in all ten combinations on both GSM8K and MATH, with gains of 2.8 to 4.1 points on GSM8K and 0.2 to 1.6 points on MATH. We will include this table in the revised appendix.

Q3. Make the presentation more focused, especially Section 3.

Thank you for this suggestion. We agree that Section 3 can be more focused and that several points are repeated. We will keep the key motivations and shorten the background discussion as follows.

(1) Section 3. We will keep each observation to its key evidence and conclusion. Section 3.1 will keep Figure 1 with a short conclusion, and the additional interpolation results will be placed in the appendix. Section 3.2 will keep only the distinction that the multi-seed regime is dominated by basis misalignment rather than task interference, with the numerical comparison moved to Appendix G. Sections 3.3 and 3.4 will each be shortened to the main result (Table 1 and Figure 2). Section 3.5 will keep Figure 3 with a short summary of the principal-angle analysis requested by the other reviewers, and the detailed subspace analysis will be placed in the appendix.

(2) Removing repetition. The discussion of Figure 3 currently appears in Sections 3.5, 4.2, and 4.3; we will keep it in Section 3.5 and only refer to it in Section 4. We will also state the four observations only once, and present the two-stage overview of SWIN only once in Section 4.2.

Comments on the paper

1 comment
Area_Chair_bP41OpenReview ↗

Please respond to the rebuttal

Dear Reviewers,

The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.