- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SWIN, a training-free, geometry-aware fusion method for integrating models fine-tuned with different seeds. It uses two stages: (i) estimating a shared subspace through a closed-form consensus objective over projection operators, followed by (ii) structure-preserving aggregation using polar decomposition and an orthogonality-constrained projection, with no additional inference-time cost. The paper evaluates SWIN against other baseline approaches in both full fine-tuning and reinforcement learning settings and shows that it outperforms them.
Critical strengths
-
The paper makes a clear case for the method through four empirical observations about single-task model fusion: same-basin mode connectivity, low-rank redundancy, complementary specialization across seeds, and weak correspondence between the SVD bases of different seeds.
-
The method follows naturally from those observations. Rather than assuming that seed-specific SVD bases align, it first estimates a shared subspace through optimization, then uses polar decomposition to separate orientation from scale.
-
The evaluation covers several benchmarks and compares against a range of baselines. The proposed method achieves the best results on most of the reported tasks.
Critical weaknesses
-
The empirical evidence does not yet support some of the paper’s broader claims about seeds. Figure 1 shows that one pair of same-task checkpoints can be linearly interpolated without a substantial GSM8K/MATH accuracy drop. That is useful evidence, but it does not establish that checkpoints from different seeds generally share a low-loss basin: the figure reports accuracy rather than loss, and it is unclear how the pair was selected. Likewise, Figure 2 uses two seeds to motivate the claim that no seed consistently dominates across subtasks. Results across more seeds and seed pairs would make both claims more convincing.
-
The paper could do more to distinguish SWIN from related work. Qin et al. (EMNLP 2022 https://arxiv.org/abs/2210.14102) studied mode connectivity in fine-tuned pretrained language models and should be discussed alongside the paper’s basin claim. Task Singular Vectors (Gargiulo et al., CVPR 2025 https://arxiv.org/abs/2412.00081) is also relevant to the SVD-based method. It would help establish the contribution with a clearer account of what carries over from these methods and what is specific to SWIN.
Questions for authors
-
How were the two seeds in Figure 1 selected? Could you show interpolation results for more seed pairs, including training or validation loss as well as GSM8K/MATH accuracy?
-
How were the seeds in Figure 2 selected? Across a larger pool, is it still true that no individual seed consistently outperforms the others on the subtasks?
-
Lines 205–208 interpret low maximum similarity between vectors in and as rotational ambiguity. How do you rule out the alternative that the two sets of vectors span substantially different subspaces?
-
Why is the RL evaluation limited to DAPO training of DeepSeek-R1-Distill-Qwen-1.5B? Would the conclusions hold for another model or RL training setup?
-
Could you clarify which components of SWIN overlap with the methods described in Appendix D, and which are new? A comparison with Task Singular Vectors, or an explanation of why it is not an appropriate baseline for same-task fusion, would be helpful. (https://arxiv.org/abs/2412.00081)
-
Could you revise Figure 2’s presentation? The small arrows beneath the x-axis are unexplained, and the lines identified in the legend as using the left and right y-axes are hard to locate. Lines 162–168 also need formatting fixes; Tables 2 and 15 should mark tied or best results consistently.
6 comments
Response to Reviewer 8ouG - Continue
W2 & Q5. Clarify the relation to Qin et al. (2022) and Task Singular Vectors (TSV): which components of SWIN overlap with prior methods and which are new? Compare with TSV or explain why it is not an appropriate baseline.
Thank you for pointing us to these works. Our contribution is a merging scheme for same-task seeds that selects a fixed-rank consensus subspace capturing the directions shared across seeds, and fuses the seeds in it with a new rule that aggregates orientation and scale separately. We clarify the relation to these works from three aspects.
(1) Mode connectivity (Qin et al., 2022). Qin et al. show for T5 that fully fine-tuned minima are linearly connected under different data orders and initialization seeds, whereas minima of different tasks are not. Our Figure 1 and Section 3.2 are consistent with these findings and extend the setting to LLMs fine-tuned on generative reasoning tasks. Our method builds on this property, and we will cite their work in Section 3.1 and present Figure 1 as consistent evidence in our setting.
(2) What carries over and what is new. Like most merging methods, SWIN operates on task vectors (deltas), as in Task Arithmetic, Model Soup, TIES, and DARE, and computes layer-wise SVDs of the updates, as in TSV, KnOTS, and Core Space. Its two core stages differ from these methods as follows.
| Stage | Prior methods | SWIN |
|---|---|---|
| Shared subspace | Union of the task subspaces at full rank, for a lossless representation (Core Space); SVD of the concatenated updates (KnOTS) | Fixed-rank consensus of the seed subspaces, keeping the directions shared across seeds |
| Fusion in the shared space | Existing rules such as TA or TIES on the aligned components (KnOTS, Core Space); orthogonalized concatenation (TSV) | Procrustes mean of the polar orientations and average of the PSD scales |
Both stages are designed for checkpoints of the same task, whereas KnOTS, Core Space, and TSV target merging models of different tasks (and KnOTS and Core Space are designed for LoRA updates). In particular, TSV uses Procrustes to decorrelate the singular vectors of different tasks, whereas SWIN uses it to average the orientations of seeds of the same task.
(3) TSV as a baseline. As discussed in (2), TSV is designed to reduce interference between different tasks. For seeds of the same task, the singular vectors largely span the same subspace (Q3), so for SVD-based merging the main difficulty is basis misalignment rather than task interference (Section 3.2). TSV is still applicable to this setting, and Table 14 in Appendix E compares it with SWIN on the same three LLaMA2-7B/MetaMathQA checkpoints: TSV reaches 68.1/19.8 on GSM8K/MATH, and SWIN reaches 71.0/22.1. We will move this comparison into the main text.
We will add this discussion to Section 2 and Appendix D in the revised version.
Response to Reviewer 8ouG - Continue
Q3. How do you rule out that the low vector similarity (lines 205–208) reflects different subspaces rather than rotation?
Thank you for this question. Reviewer V7dE raised the same question (Q1), and we summarize the analysis here. We agree that Figure 3 alone does not rule out this alternative, since low row-wise maxima can arise either from rotations within a shared subspace or from different subspaces. To distinguish the two, we compute the principal angles between the retained subspaces of each pair of seeds, whose cosines are the singular values of , for all projection matrices of three models (LLaMA2-7B, LLaMA3-8B, and Gemma-2-9B, each fine-tuned on MetaMathQA with three seeds). The table uses the retained rank of Figure 3. Two random -dimensional subspaces of have mean squared cosine , which serves as a reference. We report medians over matrices and seed pairs, with 10th–90th percentiles in brackets.
| Model | Best vector match (Fig. 3 statistic) | Mean / random baseline | Median principal angle |
|---|---|---|---|
| LLaMA2-7B | 0.19 [0.15, 0.26] | [8.0, 20.7] | [18, 47] |
| LLaMA3-8B | 0.23 [0.18, 0.34] | [3.0, 35.3] | [15, 40] |
| Gemma-2-9B | 0.15 [0.13, 0.22] | [4.1, 20.1] | [20, 54] |
Across the three models, each singular vector's best match in the other seed has cosine 0.15–0.23, yet the two subspaces overlap 8.5–11.7$\times$ more than random subspaces. The same holds at the 50% rank used by SWIN: the median principal angle is –, well below the or more expected for random subspaces at this rank. We also examine the eigenvalues of the averaged projector used by Stage 1: a direction present in all three seeds has an eigenvalue near 1, and a direction present in only one seed (and orthogonal to the other seeds' subspaces) has about 1/3. At 50% rank, the median retained eigenvalue is 0.89–0.95, so most retained directions are largely shared by all seeds.
The seeds therefore span largely shared subspaces in different bases, so the low maxima in Figure 3 mainly reflect basis ambiguity. The principal angles are not zero, so genuine subspace differences also exist; Stage 1 handles them by taking the consensus of the seed projectors, which is invariant to the choice of basis within each subspace.
We will revise lines 205–208 to state that Figure 3 measures the correspondence between individual singular vectors, and support the rotational-ambiguity interpretation with the principal-angle analysis in Section 3.5, with details in the appendix.
Response to Reviewer 8ouG
Thanks for your constructive feedback, we carefully address your concerns below.
Thank you for this suggestion. We agree that the evidence in Figures 1 and 2 is based on one pair of seeds each, and that Figure 1 reports accuracy rather than loss. We address this from two aspects.
(1) Connectivity (Figure 1). Figure 1 interpolates between two LLaMA2-7B seeds fine-tuned on MetaMathQA, randomly selected from the three seeds used in Table 3; this pair is also included in the ten pairs below. To remove the dependence on a single pair, we now interpolate between all ten pairs of five LLaMA2-7B seeds fine-tuned on MetaMathQA (42, 87, 100, 13, and 57; the same seeds as in our response to Reviewer TFiH) at five equally spaced points (γ=0,0.25,0.5,0.75,1 in Eq. 2). At each point, we measure the training loss (token-level negative log-likelihood of the responses) on 2,000 randomly sampled MetaMathQA training examples and evaluate GSM8K and MATH accuracy.
In every pair, the training loss ($\times$100) at the interior points exceeds the line connecting the two endpoint losses by at most 0.30, about 5% of the endpoint loss, and GSM8K and MATH accuracy at the interior points is never more than 0.3 points below the lower of the two endpoints. Thus, all evaluated seed pairs are linearly connected without a substantial loss barrier or accuracy drop, consistent with Figure 1. The table shows six of the ten pairs; Appendix J.6 reports all ten.
(2) Seed specialization (Figure 2). Figure 2 shows seeds 87 and 100 of LLaMA3-8B fine-tuned on Commonsense-170k, randomly selected from the three seeds in Table 4. Table 4 already reports all three seeds: seed 42 has the highest average accuracy, seed 87 leads on HellaSwag, and seed 100 leads on WinoGrande and ARC-C, so the best seed varies across subtasks among these three. To test this on a larger pool, we fine-tune two additional LLaMA3-8B seeds (13 and 57) on Commonsense-170k under the same settings, and report the accuracy of all five seeds on each subtask in the same format as Table 4, where the results of seeds 42, 87, and 100 are taken from Table 4 and bold marks the best seed on each subtask, including ties.
In the pool of five seeds, the best seed still varies across subtasks: seeds 42, 13, and 57 each rank first on at least one subtask, seed 87 ties for first on BoolQ, and no seed ranks first on all eight. Seed 42 ranks first on five subtasks and has the highest average, while seed 13 leads on HellaSwag and seed 57 on WinoGrande and ARC-C.
Figure 1 will retain the original seed pair. Section 3.1 will summarize the ten-pair results and refer to Appendix J.6 for the summary figure and full loss and accuracy curves. A figure with all evaluated seeds will be added to the appendix and referenced in the caption of Figure 2. We will also state how the seeds were selected, and limit both claims to the evaluated models and seeds: the seed pairs are connected by linear paths without a substantial loss barrier, and the best seed varies across subtasks.