- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the identifiability of task-related information from fine-tuned checkpoints in data-free model merging and proposes Invariant Task-Core Fusion (ITCF). The authors first show in a linear regression setting that different input covariance structures can produce identical final checkpoints, indicating that checkpoints cannot uniquely determine the relative importance of input directions. Motivated by this observation, ITCF constructs isotropic task cores from the dominant right-singular subspaces of task vectors and leverages their overlap to coordinate expert updates. An adaptive spectral regularization scheme further balances joint response matching with summed task updates. The method requires neither task data nor validation-based hyperparameter tuning. Experiments across multiple vision and language models, covering over 500 task compositions and up to 166 experts, demonstrate competitive performance, scalability, and computational efficiency.
Critical strengths
-
The paper investigates what task-related information can be reliably inferred from fine-tuned checkpoints. By characterizing the ambiguity of input covariance structures and deriving a minimax-optimal comparison geometry under the considered uncertainty, it provides an interesting theoretical perspective on data-free model merging and motivates the proposed isotropic task-core construction.
-
The proposed method combines task-core overlap with an adaptive spectral recovery scheme, requiring neither task data nor validation-based tuning. Experiments across multiple vision and language model families and up to 166 experts demonstrate competitive performance against strong baselines. The closed-form construction also offers substantial computational advantages over iterative merging methods, making the approach practically appealing.
Critical weaknesses
-
Proposition 1 proves that isotropic task cores minimize worst-case response-comparison distortion under a specific uncertainty set. However, this does not guarantee better multi-expert fusion or downstream performance, as also illustrated by the two-expert counterexample in Appendix B.3. The ablation results provide limited support for the practical benefit of isotropic weighting. In Table 4, ISO outperforms SV2 by only 0.052 percentage points (95% CI: [-0.155, +0.260]). ISO also performs worse on RoBERTa (-0.213 pp) and Qwen2.5-1.5B (81.82 vs. 81.91; Table 31). In contrast, the proposed recovery scheme improves over ordinary ridge by 0.277 pp when ISO is fixed. These results leave the contribution of the minimax-derived geometry to downstream performance unclear.
-
Although the paper discusses Iso-C (Marczak et al., 2025), it does not compare ITCF with either Iso-C or Iso-CTS. Both methods use task-vector subspaces and isotropic spectral transformations to mitigate task interference. They are therefore particularly relevant baselines. Without comparisons under matched checkpoints, task compositions, and comparable tuning budgets, it is difficult to assess whether ITCF offers a meaningful improvement over existing isotropic merging methods.
Questions for authors
Could the authors analyze whether a lower task-core response-matching loss corresponds to better preservation of individual task performance? It would be helpful to examine this relationship under different levels of task-core overlap and disagreement between expert updates. Such evidence would clarify when the proposed minimax-derived geometry is a useful proxy for downstream task preservation.
4 comments
To Reviewer uWGk
Dear Reviewer uWGk,
Thank you again for your thoughtful review and valuable suggestions.
We have submitted our detailed responses and uploaded the revised manuscript, incorporating additional theoretical results and experimental evidence addressing your concerns, including the matched-budget Iso-C/Iso-CTS comparisons and complete-network functional interventions.
As the discussion period draws to a close, we would greatly appreciate it if you could review these additions and share any remaining questions or concerns.
We would be happy to provide further clarifications during the remaining discussion period.
Thank you very much for your time and consideration.
Best regards, The Authors
Response to Authors
Thank you for the detailed rebuttal and supplementary experiments. The matched Iso-C/Iso-CTS comparisons and full-network interventions substantially address my earlier concerns and strengthen the empirical validation of ITCF. However, the isolated downstream benefit of the theoretically motivated isotropic geometry remains limited, while the recovery scheme and its interaction with geometry show clearer gains. Overall, I appreciate the additional evidence but maintain my rating of 3 (Weak Acceptance).
To Reviewer uWGk
Dear Reviewer uWGk,
Thank you very much for your thoughtful reassessment and constructive feedback.
We sincerely appreciate your recognition that the additional Iso-C/Iso-CTS comparisons and full-network interventions have substantially addressed your earlier concerns and strengthened the empirical validation of ITCF.
We also appreciate your thoughtful observations regarding the respective roles of isotropic geometry and recovery. Your feedback has been valuable in helping us further refine our understanding and presentation of the method.
Thank you again for your time, careful consideration, and positive assessment of our work.
Best regards, The Authors
Response to Reviewer uWGk
Dear Reviewer uWGk,
Thank you for your constructive review and for recognizing ITCF's theoretical foundation and practical efficiency. We have added full-response analysis, direct Iso-C/Iso-CTS comparisons, and network interventions across task-overlap and update-disagreement regimes.
1. From minimax comparison to a beneficial fusion decision.
Under its stated uncertainty model, Proposition 1 gives the minimax isotropic comparison metric Gi=Pi/ri on the retained task support. ITCF converts it into participation A=∑iGi and requested response B=∑iΔiGi, then computes the actual anchored update Mτ=M0+(B−M0A)QΛ(Λ2+τ2I)−1Q⊤. Its modewise weight λj2/(λj2+τ2) follows shared requests more strongly along well-supported directions and stays closer to the anchor where support is weak.
Theorem L.3 and Corollary L.4 establish a full-response success criterion for this actual output, allowing calibrated within-core inputs, off-core energy, and cross-covariance (sufficient condition ag−dS>∑iLi). Importantly, the factorial study also identifies where the empirical gains arise: recovery is +0.277/+0.323 and geometry–recovery interaction +0.161/+0.155 under setting/task weighting, alongside isolated geometry +0.052/−0.039. With the original scalar selector fixed, isotropic geometry additionally improves accuracy in all five ViT task–composition comparisons across three distinct tasks. Retaining the core correction improves task-native scores in 24 of 28 matched full-network interventions (p. 9, line 485; p. 75, lines 4008–4009).
Thus, the minimax geometry supplies a principled comparison structure, while anchored recovery and its interaction with that geometry deliver the principal pooled benefits of the actual fusion rule.
(p. 4, lines 188–215; p. 5, lines 218–260; p. 60, lines 3204–3222; p. 66, lines 3530–3555; p. 67, lines 3602–3617; p. 79, lines 4238–4254.)
2. Direct Iso-C and Iso-CTS comparisons.
We evaluated 22 matched task compositions and five training seeds under three validation-selection tracks. In the particularly informative Track B, the original checkpoint-only ITCF uses zero validation candidates while Iso-C and Iso-CTS receive 30 each; ITCF still leads by +0.568 (paired 95% CI [+0.511, +0.624]) and +0.350 ([+0.302, +0.397]) scaled mixed-native macro points, with 5/5 positive paired seed aggregates against each baseline.
Under matched 30/30/30 budgets in Track C, validation-selected ITCF leads by +0.753/+0.536 points, again 5/5 positive seed aggregates and 18/22 favorable task-composition means against each comparator.
ITCF leads both tuned isotropic baselines without validation and remains ahead under equal tuning budgets.
(p. 72, lines 3882–3887; p. 73, lines 3889–3918.)
3. Core response matching and preservation of individual-task ability.
Separately from the original-core retention experiment in Point 1, we compared controlled withdrawals of the task-support direction in 13 task contexts under two fixed network backgrounds, while holding expert untargeted parameters, task heads, inputs, and complementary directions fixed within each pair.
The measured core response-matching loss decreases in 26/26 background-conditioned comparisons; gold-label CE decreases in 19/26, and the task-specific readout improves in 17/26 (four ties, five decreases). By background, the matching-loss/CE co-improvement occurs in 9/13 (B0) and 10/13 (B1).
For RoBERTa CoLA, CE falls by 0.575/0.713 nats per example and MCC improves by 0.421/0.289; for ViT Cars, CE falls by 0.226/0.244 with accuracy gains of 6.25/7.81 percentage points. Classification uses native task metrics; T5 uses fixed gold-prefix token readouts.
We further stratified these same comparisons by projector overlap tr(GiGj) and task-projected own/foreign update-direction disagreement Cdir=1−cos(⋅,⋅), using within-family median strata. In the sum-anchor background, ViT's CE improvements span 2/2 low, 1/1 median, and 2/2 high-overlap cases; RoBERTa's high-disagreement group improves in 2/2 cases under each background.
These controlled withdrawals reveal a direct empirical relationship between task-core response matching and individual-task outcomes across distinct overlap and update-disagreement regimes. Together with the original-core retention interventions in Point 1, they provide complementary functional evidence for ITCF's task-core construction.
(p. 77, lines 4153–4157; p. 78, lines 4158–4209.)
We have uploaded the revised PDF, incorporating the additional theoretical results, experimental evidence, and clarifications discussed above. We sincerely hope these additions address your concerns and further demonstrate the scientific contribution of ITCF. Thank you again for your thoughtful feedback and consideration.