acceptodds
Under review as a conference paper at ICLR 2027

Loss-Aware Optimization for Model Merging in a Low-Dimensional Subspace

Abstract

Model merging aims to combine multiple task-specific models into a single model without joint retraining. Many existing methods construct merged updates from geometric or structural information, leaving their relation to downstream task losses largely indirect. To address this limitation, we formulate model merging as local optimization of the average supervised loss around a strong merged anchor. Using calibration subsets, we linearize each task output at the anchor and apply a second-order Taylor expansion to the resulting tangent loss, yielding a quadratic surrogate based on generalized Gauss-Newton curvature. This curvature captures the sensitivity of each task loss to perturbations along different parameter directions, distinguishing directions that are costly to change from those that can accommodate larger deviations. We further restrict optimization to a low-dimensional subspace spanned by structured components of merged updates from strong existing methods. This restriction retains the structural information encoded by these updates, regularizes the optimization by constraining the search space, and reduces the quadratic problem to a low-dimensional linear system with a closed-form solution. Our method can be integrated with a broad range of existing merging methods and jointly leverage updates from multiple sources. Extensive experiments on benchmarks demonstrate notable improvements over strong baselines across model architectures and task collections.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Multi-task Learning, Quadratic Approximation, Linear Subspace

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Qn8uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LossOpt uses labeled calibration data to refine a strong merged model. It partitions updates from merging methods into Transformer blocks and remaining-parameter directions, constructs a local generalized Gauss–Newton (GGN) quadratic loss, and solves a ridge-regularized linear system. CLIP and RoBERTa experiments examine performance, directions, curvature approximations, and calibration size.

Critical strengths
  1. Correcting residual task conflicts from a strong merged initialization is worthwhile. The anchor preserves an existing compromise, blockwise directions control flexibility, and projected GGN captures interactions between directions. Each choice serves a clear role.
  2. The multi-source configuration improves ESM in all nine vision settings, including 81.3% to 83.4% for ViT-B/32 with 20 tasks. Normalized GLUE performance also rises from 89.11 to 90.85 (Tables 1, 3; PDF pp. 7, 9). Curvature ablations and loss diagnostics support effectiveness in the tested settings.
Critical weaknesses
  1. GGN and ridge regularization are established tools, leaving the common anchor and structured source basis as the main methodological additions. However, the source ablation increases dimensionality from 7 to 14 to 21 (Table 4, PDF p. 9). It shows that expanding the basis helps, but does not distinguish complementary information retained by merging methods from additional flexibility. This limits the case for a merging-specific contribution.
  2. Figure 3 compares curvature approximations, and Figure 4 compares direct tangent-loss optimization using the same anchor and subspace (PDF pp. 8–9). However, direct nonlinear supervised-loss optimization remains only a conceptual comparator (Appendix B, Table 5, Eq. 6, pp. 15–16). The quadratic solver's advantage over ordinary supervised coefficient refinement is therefore unclear. The optimizer, stopping rule, and regularization selection for Figure 4's comparator are also incompletely specified.
  3. The paper explicitly excludes Pearson loss from its PSD guarantee (§4.6, PDF pp. 8–9). Algorithm 1 nevertheless checks only positive trace and sets ridge regularization from average curvature; this cannot ensure that an indefinite Hessian becomes positive definite. Eigenvalue checks, additional damping, and whether the returned solution minimizes the quadratic objective remain unspecified. This affects interpretation of the language results, not the PSD-GGN derivation.
Questions for authors
  1. What information motivates using merged updates as a directional prior? How does Table 4 distinguish source complementarity from the effect of additional coefficients?
  2. With the anchor, basis, and calibration data fixed, how does direct nonlinear supervised-loss optimization compare with LossOpt in performance and computation? What tuning budgets are used? Please also specify the existing tangent-loss comparator's optimization settings.
  3. For STS-B, do you check the regularized matrix's minimum eigenvalue or apply additional damping? If it remains indefinite, how is the returned solution defined and selected?
3 comments
AuthorsOpenReview ↗

Responses to Questions 1 and 2

We sincerely thank the reviewer for the thoughtful questions and insightful suggestions.

Q1.

Our motivation is to retain useful structure already identified by successful merging methods. Existing methods combine task updates using information from parameter, representation, or spectral spaces. Their merged updates therefore provide structured candidate directions, while the local supervised objective determines how these directions should be adjusted around a common merged anchor.

To further examine direction choice independently of coefficient count, we have added a comparison between two 21-dimensional bases on RoBERTa-Base. We compare the original WUDI/Iso-CTS/ESM block basis with a task-vector SVD basis. For each of the six selected blocks and the Rest group, we stack the eight individual task updates and retain the top three singular directions. Both configurations use the same ESM anchor and calibration data:

BasisDimensionValidation scoreTest score
Task-vector SVD basis2198.93790.225
WUDI/Iso-CTS/ESM2199.59490.846

The original basis improves the test score by 0.621 points at the same dimension. Together with the comparisons in Table 4, this supports the value of the selected directions beyond the number of coefficients.

Q2.

We have added a RoBERTa-Base comparison using the same ESM anchor, 21-dimensional WUDI/Iso-CTS/ESM basis, fixed task heads, and 1,000 calibration examples per task. The table reports mean normalized test scores, the number of candidate models considered for validation selection, and recorded computation times:

Method# candidatesTest scoreComputation time (min)
Direct nonlinear loss1290.300282.5
LossOpt1290.84632.7

LossOpt exceeds direct nonlinear optimization by 0.546 points. Regarding computational cost, direct nonlinear optimization requires repeated network forward and backward passes, totaling 282.5 minutes. LossOpt computes directional Jacobians once in 32.7 minutes, then solves a regularized 21×2121\times21 system for each candidate without further network differentiation. Both timings exclude validation and testing.

Direct tangent loss optimization. This comparator uses the same cached directional Jacobians and 12 regularization values. Each run starts from the corresponding LossOpt quadratic solution and uses L-BFGS with the following settings:

SettingValue
Learning rate1
Line searchStrong Wolfe
Maximum iterations / evaluations300 / 400
History size50
Gradient tolerance10−1010^{-10}
Change tolerance10−1210^{-12}

The selected candidate uses κ=0.001\kappa=0.001 and requires 284 loss and gradient evaluations. These evaluations use cached outputs and directional Jacobians, without further forward or backward passes through the nonlinear network.

AuthorsOpenReview ↗

Responses to Question 3

Q3.

We have explicitly checked the minimum eigenvalue of the aggregate coefficient-space Hessian before solving. For STS-B, the Pearson correlation objective can contribute negative curvature, but this does not necessarily make the aggregate Hessian indefinite.

Let H\mathbf{H} denote the symmetric Hessian averaged over the eight normalized task objectives. We set

λκ=max⁡{0,−λmin⁡(H)+10−8max⁡(1,max⁡i∣λi(H)∣)}+κ meani∣λi(H)∣,\lambda_\kappa = \max\{0,-\lambda_{\min}(\mathbf{H})+10^{-8}\max(1,\max_i\vert\lambda_i(\mathbf{H})\vert)\}+\kappa\,\mathrm{mean}_i\vert\lambda_i(\mathbf{H})\vert,

and solve (H+λκI)η^κ=−g(\mathbf{H}+\lambda_\kappa\mathbf{I})\widehat{\boldsymbol{\eta}}_\kappa=-\mathbf{g}.

The first term supplies an additional shift when needed to ensure positive definiteness; the second controls regularization. The solver explicitly verifies that λmin⁡(H+λκI)>0\lambda_{\min}(\mathbf{H}+\lambda_\kappa\mathbf{I})>0.

Thus, it returns a unique quadratic minimizer, not a stationary point from an indefinite system.

Here, I is the identity matrix and g is the coefficient-space gradient. The first term, delta, supplies an additional shift when needed to ensure positive definiteness; the second controls regularization. The solver explicitly verifies that the minimum eigenvalue of H + lambda_kappa I is positive. Thus, it returns a unique quadratic minimizer, not a stationary point from an indefinite system. The measured minimum eigenvalues are:

MatrixMinimum eigenvalue
Weighted STS-B contribution−4.7606×10−4-4.7606\times10^{-4}
Aggregate Hessian H\mathbf{H}4.2645×10−54.2645\times10^{-5}
Regularized aggregate, selected κ=10−3\kappa=10^{-3}1.1106×10−41.1106\times10^{-4}
Regularized aggregate, smallest κ=3×10−4\kappa=3\times10^{-4}6.3170×10−56.3170\times10^{-5}

The aggregate Hessian is already positive definite, so no additional positive definiteness shift was needed. The selected κ=10−3\kappa=10^{-3} gives λκ=6.8416×10−5\lambda_\kappa=6.8416\times10^{-5}, and all 12 tested regularized systems were positive definite. We select among their solutions using the validation score and evaluate the selected model once on the test set, obtaining a mean normalized test score of 90.846.

I appreciate the author providing additional experiments, which have addressed some of my concerns. My primary reservation remains that the current evidence is insufficient to determine to what extent the performance gains stem from combining the directions provided by different merging methods. Consequently, I will maintain my current rating.

Reviewer_fLuxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LossOpt, which refines a merged model by optimizing a GGN-based quadratic surrogate of the average supervised loss within a low-dimensional subspace built from existing merging updates. Results on CLIP and RoBERTa benchmarks show consistent improvements over the anchor methods.

Critical strengths
  1. The derivation is clear, connecting tangent linearization, GGN curvature, and a closed-form ridge solution.
  2. The coefficient-space computation involves directional Jacobians and a linear system of at most 21×21.
  3. The curvature ablation and the comparison with direct tangent-loss optimization in Fig. 4 are informative.
Critical weaknesses
  1. The comparison does not account for differences in data and tuning budgets.
    Most baselines in Table 1 construct updates without calibration data, whereas LossOpt uses labeled examples from each task and validation data for κ and block selection. The comparison should distinguish data and tuning budgets and include a supervised baseline with the same budget.

  2. The benefit over simple supervised fine-tuning remains unclear.
    A natural competitor is missing from the experiments: fine-tuning the anchor on the same calibration set (full FT or LoRA) with the same validation budget. If this baseline matches LossOpt, the accuracy–cost benefit of the GGN machinery needs clarification; if it underperforms, that would support the proposed constraints.

  3. The construction of the basis directions is insufficiently explored.
    The study feels incomplete regarding the construction of the basis directions. The framework already estimates the quantities needed to construct its own directions (e.g., the top eigenvectors of Ĝ, or blockwise task vectors), yet it relies entirely on directions borrowed from existing merging methods, without analyzing or comparing any data-driven alternative.

  4. The robustness of the reported improvements is not established.
    The main results are reported without repeated runs or uncertainty estimates, while the smallest margin is only 0.2 points (ViT-L/14, 8 tasks). Repeating experiments across random seeds, including calibration-set sampling and any other stochastic components, and reporting mean ± standard deviation would help assess whether the gains are reliable.

Questions for authors

see Weaknesses

3 comments
AuthorsOpenReview ↗

Responses to Weaknesses 1 and 2

We sincerely thank the reviewer for the thoughtful questions and insightful suggestions.

W1.

We would like to distinguish labeled calibration data from validation-based tuning. Data-free construction does not necessarily mean tuning-free evaluation: existing merging methods also use validation data to select the global update scale in θ0+λΔ\boldsymbol{\theta}_0+\lambda\boldsymbol{\Delta}.

Our method does not require extensive additional tuning. The block ablations present exploratory results; we do not select blocks strictly by rank. We generally use the final six Transformer blocks, consistent with the expectation that later blocks are more influential.

For regularization, the main experiments use only a four-value grid. Moreover, the supplementary three-seed experiments fix κ=0.01\kappa=0.01 on both ViT-B/32 and ViT-L/14 with fourteen tasks, without selecting it separately for each run. All runs outperform their corresponding ESM anchors. These results show that a fixed regularization setting can yield effective corrections without repeated ridge tuning. (The results are reported in our response to Q4.)

Nevertheless, LossOpt uses labeled calibration data beyond what data-free merging requires. Data-assisted methods such as ESM and RegMean also use task inputs, although their merging statistics do not require ground truth labels. Methods using labeled data also exist, including the empirical Fisher baseline reported in MaTS.

On the eight task benchmark, the reported average test accuracies are:

MethodViT-B/32ViT-L/14
RegMean82.0%89.6%
ESM88.4%94.8%
Empirical Fisher66.3%82.5%
LossOpt (T+I+E)89.1%95.0%

More importantly, we have varied the basis, objective, and curvature while keeping the calibration data and anchor fixed within each comparison. Together, these controlled comparisons show that performance depends not only on access to data, but also on the choice of directions, objective, and curvature structure.

W2.

As discussed in our response to Question 1, the original and supplementary experiments support the contribution of method design beyond calibration data alone.

We have conducted the suggested supervised fine-tuning of the ESM anchor on ViT-B/32 with fourteen tasks. We trained a shared rank-one LoRA adapter on the same seed-0 calibration samples, applying it to the query and value projections in the final six Transformer blocks. The anchor and task-specific heads remain frozen, and the same adapter is shared across all fourteen tasks.

Under the corresponding task loss normalization, the results are:

MethodFree parametersTest accuracy
LossOpt2185.1271%
Anchor + LoRA18,43287.8061%

LoRA exceeds LossOpt by 2.6790 percentage points and also outperforms the other merging baselines reported for this configuration. It optimizes approximately 878 times as many variables. LoRA performs 500 Adam updates with repeated network forward and backward passes, whereas LossOpt estimates local statistics and solves a small regularized linear system. This avoids iterative adapter training. For storage, the LoRA parameters occupy approximately 72 KiB in FP32, with additional gradient and optimizer states during training. LossOpt optimizes only 21 coefficients.

These results position LossOpt as a structured correction with few optimization variables, rather than a universally more accurate replacement for supervised fine-tuning.

AuthorsOpenReview ↗

Responses to Weaknesses 3 and 4

W3.

We have expanded the basis study by adding a task-vector SVD basis that does not rely on directions produced by existing merging rules, and evaluated it on both ViT-B/32 and RoBERTa-Base.

On ViT-B/32 with fourteen tasks, we stack the fourteen individual task updates without centering for each of the six selected blocks and the Rest group, retaining the top three singular directions per group. This yields 21 directions in total. We match each group's total squared basis norm to that of the original T+I+E directions. Both bases use the same ESM anchor, calibration samples, raw task-averaged cross-entropy objective, and κ=0.01\kappa=0.01:

ConfigurationDimensionTest accuracy
ESM anchor--84.4713%
Task-vector SVD basis2185.0193%
Original T+I+E basis2185.3832%

On RoBERTa-Base, we construct a 21-dimensional task-vector SVD basis from the eight individual task updates and compare it with the original WUDI/Iso-CTS/ESM basis, denoted by W+I+E. Both use the same ESM anchor and calibration samples. The mean normalized test scores are:

ConfigurationDimensionTest score
ESM anchor--89.110
Task-vector SVD basis2190.225
Original W+I+E basis2190.846

In both settings, the SVD basis improves upon the ESM anchor, while the original basis performs better at the same dimension. These results show that LossOpt remains effective with alternative directions, and basis construction matters beyond the number of coefficients.

Regarding the suggested eigenvectors of G^\widehat{\mathbf{G}}, we compute directional Jacobian products JB\mathbf{J}\mathbf{B} and assemble the small projected curvature matrix G^∈Rr×r\widehat{\mathbf{G}}\in\mathbb R^{r\times r} without explicitly constructing the full parameter GGN matrix. Its eigenvectors only rotate or restrict the existing subspace; they do not introduce directions outside it.

W4.

We have added repeated calibration sampling experiments on both ViT-B/32 and ViT-L/14 with fourteen tasks. For each configuration, we use seeds 0, 1, and 2 while fixing the experts, source directions, data splits, ESM anchor, and relative ridge parameter κ=0.01\kappa=0.01.

Test accuracies are reported as mean ±\pm sample standard deviation across the three runs:

Backbone, 14 tasksLossOptLocal ESM anchor
ViT-B/32(85.3607±0.0314)(85.3607\pm0.0314)%83.7713%
ViT-L/14(91.9289±0.0142)(91.9289\pm0.0142)%91.2602%

All three seeds yield improvements over the corresponding ESM anchor in both configurations. The mean gains are 1.5894 percentage points on ViT-B/32 and 0.6687 points on ViT-L/14, with small observed variation across calibration draws.

Thank you for the detailed response and additional experiments. However, the new results have further reinforced my concerns. LossOpt achieves substantially lower accuracy than a rank-one LoRA baseline, which raises further doubts about the practical advantages and necessity of the proposed method. Therefore, I have decided to maintain my original rating.

Reviewer_BTC6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a theoretically motivated and computationally efficient framework for loss-aware model merging using a GGN-based quadratic surrogate and closed-form optimization in a low-dimensional subspace. Despite its clean formulation and broad baseline comparisons, major concerns remain regarding novelty, particularly the omission of closely related works, the lack of trust-region constraints for the local approximation, and potentially unfair comparisons against training-free baselines due to additional labeled calibration data. Insufficient reporting of tuning procedures and statistical variability further weakens the empirical evidence.

Critical strengths

(1) Introduces an anchor-centered GGN formulation for loss-aware model merging.

(2) Provides a well-justified convex surrogate under appropriate assumptions.

(3) Reduces optimization to a small linear system with a closed-form solution.

(4) Directly optimizes supervised task loss rather than proxy objectives.

(5) Compares against competitive recent model-merging baselines and presents clear mathematical derivations and a flexible plug-in framework applicable to existing merging methods.

Critical weaknesses

(1) The proposed formulation overlaps with closely related works, which are neither cited nor compared, making the originality difficult to assess.

(2) The closed-form correction lacks trust-region or step-size constraints, potentially violating the validity of the local approximation.

(3) The method uses labeled calibration data, while several baselines are training-free, making performance gains difficult to attribute solely to the proposed approach.

(4) The paper lacks statistical significance analysis, calibration-size ablations, and computational cost measurements.

(5) Experiments focus exclusively on CLIP vision models, despite broader claims about applicability across architectures.

(6) Important implementation details, particularly the subspace basis construction and anchor selection, are insufficiently specified.

(7) The method does not address potential shared-knowledge over-accumulation or justify its fixed-basis capacity allocation.

(8) The computational and accuracy trade-offs against iterative data-driven merging methods remain unexplored.

Questions for authors
  1. How does the proposed GGN-based formulation fundamentally differ from closely related methods, particularly regarding curvature-based optimization in low-dimensional subspaces?

  2. How do the authors ensure that the closed-form correction remains within the region where the local tangent approximation is valid, given the absence of trust-region or step-size constraints?

  3. Since the proposed method uses 2,000 labeled calibration examples per task, how can the authors demonstrate that the improvements arise from the proposed objective rather than additional labeled data or tuning?

  4. How sensitive are the results to calibration-set size, random seeds, and basis construction? Can the authors provide ablation studies and statistical significance analysis?

  5. What are the actual computational and memory costs compared with training-free and iterative data-driven merging methods?

  6. Does the additive correction amplify shared-knowledge over-accumulation as the number of merged tasks increases?

  7. Can the authors provide evidence that the method generalizes beyond CLIP-based vision models?

4 comments
AuthorsOpenReview ↗

Responses to Questions 1 and 2

We sincerely thank the reviewer for the thoughtful questions and insightful suggestions.

Q1.

Our contribution is not a new curvature estimator or linear solver, but their formulation as a structured correction of an existing merged model.

First, existing curvature-based methods, such as Fisher merging and MaTS, construct matching objectives using statistics evaluated at individual fine-tuned models. For LossOpt, we have instead evaluated all task gradients and curvatures at a common merged anchor. Changing the anchor therefore changes the local objective itself. On ViT-B/32 and ViT-L/14 with eight tasks, the comparison with published average test accuracies is:

MethodViT-B/32ViT-L/14
Diagonal Fisher merging66.3%82.5%
MaTS82.6%90.2%
LossOpt (T+I+E)89.1%95.0%

Second, we have adopted a flexible basis construction using, for example, block components of existing merged updates. This allows LossOpt to jointly leverage complementary directions identified by empirically strong merging methods, such as Iso-CTS and ESM.

Third, we have retained the complete projected GGN, including directional interactions, whereas Fisher merging and MaTS use diagonal or isotropic approximations. We have examined these interactions through a curvature ablation on ViT-B/32 with fourteen tasks. Figure 3 (right) reports the following test accuracies:

FullDiagonalIsotropic
85.376%84.472%83.879%

These results support the benefit of retaining directional interactions.

Q2.

We have used ridge regularization to control the correction without imposing an explicit trust region. In the PSD-GGN setting, for λ>0\lambda>0, the solution satisfies ∥η^λ∥2≤∥g^∥2λ\Vert\widehat{\boldsymbol{\eta}}_{\lambda}\Vert_2 \leq \frac{\Vert\widehat{\mathbf{g}}\Vert_2}{\lambda}

∥Bη^λ∥2≤∥B∥2∥g^∥2λ.\Vert\mathbf{B}\widehat{\boldsymbol{\eta}}_{\lambda}\Vert_2 \leq \frac{\Vert\mathbf{B}\Vert_2\Vert\widehat{\mathbf{g}}\Vert_2}{\lambda}. These bounds characterize how regularization controls the correction magnitude. We have also proved in Proposition 1 of the Appendix that a nonzero correction defines a descent direction for the original loss, guaranteeing improvement for a sufficiently small step.

We have further examined the local approximation empirically in Figure 9 of Appendix C.3. The original nonlinear loss decreases at the implemented step s=1s=1 in both examined configurations. We have also selected the regularization strength by evaluating the nonlinear model on validation data. These checks provide empirical support.

AuthorsOpenReview ↗

Responses to Questions 3 and 4

Q3.

Our experiments support the contribution of the proposed optimization design beyond access to labeled calibration data. We have examined this from three perspectives.

We have evaluated different labeled calibration budgets on ViT-B/32 with fourteen tasks. Table 2 reports:

500 examples/task2,000 examples/task
85.392%85.376%

Thus, comparable performance is obtained with one quarter of the labeled calibration data in this configuration.

We have also compared different objectives using the same labeled calibration data, ESM anchor, and 21-dimensional subspace on RoBERTa-Base. Figure 4 shows that LossOpt achieves a higher test score than direct tangent loss optimization despite a lower validation score, supporting the contribution of the quadratic surrogate with the labeled data held fixed. Furthermore, we have conducted a curvature ablation under the same calibration setting. Figure 3 (right) shows that full projected curvature outperforms diagonal and isotropic approximations, supporting the benefit of modeling directional interactions.

We have additionally compared our results with published results for Fisher merging and MaTS, which also use calibration data. On the eight task benchmark, the average test accuracies are:

MethodViT-B/32ViT-L/14
Fisher merging66.3%82.5%
MaTS82.6%90.2%
LossOpt (T+I+E)89.1%95.0%

Q4.

We have conducted additional experiments on calibration set size, random seeds, and basis construction using ViT-B/32 with fourteen tasks.

First, we have evaluated calibration budgets of 500, 1,000, 2,000, and 3,000 examples per task. Table 2 reports:

Examples/task5001,0002,0003,000
Test accuracy85.392%85.316%85.376%85.383%

The spread of only 0.076 percentage points indicates limited sensitivity to calibration set size in this setting.

Second, we have conducted three runs with calibration sampling seeds 0, 1, and 2, using 2,000 examples per task:

Seed 0Seed 1Seed 2
85.3832%85.3248%85.3741%

Mean ±\pm sample standard deviation: (85.3607±0.0314)(85.3607\pm0.0314)%.

This small variation indicates limited sensitivity to the tested sampling seeds. All three runs improve upon the ESM anchor at 84.4713%. Under a normality assumption for variation across calibration draws, the exploratory 95% Student's tt confidence interval for the mean gain is [0.8113,0.9675][0.8113,0.9675] percentage points.

Third, the original basis ablations in Figure 3 and Table 4 examine block selection, inclusion of the Rest direction, and source combinations. Combining all three sources performs best among the tested source configurations.

We have now added an alternative task-vector SVD basis. For each of the six selected blocks and the Rest group, we retain the top three singular directions from the 14 individual task updates. The corresponding test accuracies are:

ConfigurationTest accuracyGain (percentage points)
ESM anchor84.4713%--
Task-vector SVD basis85.0193%+0.5480+0.5480 over ESM
Original T+I+E basis85.3832%+0.3639+0.3639 over SVD

Thus, the correction remains effective with an alternative basis, while directions from existing merging methods perform better in this comparison.

Together, these results show limited observed sensitivity to calibration budget and sampling seeds in this setting, while confirming that basis construction affects the achievable improvement.

AuthorsOpenReview ↗

Responses to Question 5

Q5.

We have reported computational costs and runtime results across all nine vision configurations.

BackboneTasksDeriv./solveSelection/eval.Core total
ViT-B/3280:09:430:10:130:19:56
ViT-B/32140:20:010:11:320:31:33
ViT-B/32200:23:320:22:170:45:49
ViT-B/1680:31:010:17:130:48:14
ViT-B/16140:53:510:26:551:20:46
ViT-B/16201:16:470:41:401:58:27
ViT-L/1481:48:420:46:452:35:27
ViT-L/14143:09:331:19:234:28:56
ViT-L/14204:30:302:13:186:43:48

On ViT-B/32 with fourteen tasks and 2,000 calibration examples per task, we have also compared LossOpt with direct tangent and nonlinear loss optimization using the same anchor, 21-dimensional subspace, calibration data, and task normalization. Rounded runtimes are:

LossOpt (main): 31 min Tangent: 68 min Nonlinear: 82 min

Regarding memory, LossOpt stores and solves a projected system rather than a full parameter Hessian. With 21 directions, the curvature matrix contains only 441 entries, approximately 3.45 KiB in FP64. Its storage and dense solution costs are O(r2)\mathcal O(r^2) and O(r3)\mathcal O(r^3), respectively.

AuthorsOpenReview ↗

Responses to Questions 6 and 7

Q6.

The additive form does not require additional accumulation of every source update. We have formulated LossOpt as a signed correction around an existing merged anchor, with zero coefficients recovering that anchor. For each parameter group, the absolute ESM coefficient is 1+η^E,g⋆1+\widehat\eta_{\mathrm{E},g}^{\star}, whereas the additional source coefficients are η^I,g⋆\widehat\eta_{\mathrm{I},g}^{\star} and η^T,g⋆\widehat\eta_{\mathrm{T},g}^{\star}. Consequently, the correction can attenuate an anchor component or counteract a source direction, rather than only adding more of it. These coefficients are optimized jointly using the projected curvature.

We have examined this behavior in the supplementary coefficient results for ViT-B/32 with fourteen tasks:

SourceCoefficient observation
Task ArithmeticAll seven coefficients are negative.
ESMFive of seven absolute coefficients are below one.

For example, the Rest coefficients, ordered as ESM, Iso-CTS, and Task Arithmetic, are:

ESMIso-CTSTask Arithmetic
Anchor100
Optimized0.78510.2232−0.1501-0.1501

These results demonstrate that the learned correction includes attenuation and subtraction, rather than uniform accumulation.

We have also evaluated increasing task counts while keeping the coefficient dimension fixed at 21 and optimizing the average task loss. Table 1 shows improvements over ESM in all nine configurations. The reported accuracy gains, in percentage points, from eight to twenty tasks are:

Backbone8 tasks20 tasks
ViT-B/320.72.1
ViT-B/160.61.5
ViT-L/140.21.2

Together, these results support selective correction rather than indiscriminate accumulation as the number of tasks increases.

Q7.

We have evaluated LossOpt beyond CLIP on RoBERTa-Base using eight GLUE tasks: CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, and RTE. We merge only the shared encoder parameters, retaining the task-specific heads, and use 1,000 labeled training examples per task for calibration.

MethodScoreMethodScoreMethodScore
Zero-shot41.70TIES-Merging64.70Iso-CTS82.24
Fine-tuned100.00Task Arithmetic w/ DARE63.70ESM89.11
Weight Averaging52.60TIES-Merging w/ DARE65.60LossOpt (W+I+E)90.85
Task Arithmetic67.80WUDI-Merging85.34

LossOpt (W+I+E) combines directions from WUDI-Merging, Iso-CTS, and ESM. The complete merging results are summarized below:

ESM directions alone improve the anchor, while combining all three sources performs best, exceeding the strongest baseline, ESM, by 1.74 points and the strongest two-source configuration by 0.81 points. These results demonstrate that LossOpt is applicable beyond CLIP-based vision models to language models with both classification and regression tasks.

Comments on the paper

1 comment
Area_Chair_ED3dOpenReview ↗

Author- Reviewer Discussion

Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.