- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LossOpt uses labeled calibration data to refine a strong merged model. It partitions updates from merging methods into Transformer blocks and remaining-parameter directions, constructs a local generalized Gauss–Newton (GGN) quadratic loss, and solves a ridge-regularized linear system. CLIP and RoBERTa experiments examine performance, directions, curvature approximations, and calibration size.
Critical strengths
- Correcting residual task conflicts from a strong merged initialization is worthwhile. The anchor preserves an existing compromise, blockwise directions control flexibility, and projected GGN captures interactions between directions. Each choice serves a clear role.
- The multi-source configuration improves ESM in all nine vision settings, including 81.3% to 83.4% for ViT-B/32 with 20 tasks. Normalized GLUE performance also rises from 89.11 to 90.85 (Tables 1, 3; PDF pp. 7, 9). Curvature ablations and loss diagnostics support effectiveness in the tested settings.
Critical weaknesses
- GGN and ridge regularization are established tools, leaving the common anchor and structured source basis as the main methodological additions. However, the source ablation increases dimensionality from 7 to 14 to 21 (Table 4, PDF p. 9). It shows that expanding the basis helps, but does not distinguish complementary information retained by merging methods from additional flexibility. This limits the case for a merging-specific contribution.
- Figure 3 compares curvature approximations, and Figure 4 compares direct tangent-loss optimization using the same anchor and subspace (PDF pp. 8–9). However, direct nonlinear supervised-loss optimization remains only a conceptual comparator (Appendix B, Table 5, Eq. 6, pp. 15–16). The quadratic solver's advantage over ordinary supervised coefficient refinement is therefore unclear. The optimizer, stopping rule, and regularization selection for Figure 4's comparator are also incompletely specified.
- The paper explicitly excludes Pearson loss from its PSD guarantee (§4.6, PDF pp. 8–9). Algorithm 1 nevertheless checks only positive trace and sets ridge regularization from average curvature; this cannot ensure that an indefinite Hessian becomes positive definite. Eigenvalue checks, additional damping, and whether the returned solution minimizes the quadratic objective remain unspecified. This affects interpretation of the language results, not the PSD-GGN derivation.
Questions for authors
- What information motivates using merged updates as a directional prior? How does Table 4 distinguish source complementarity from the effect of additional coefficients?
- With the anchor, basis, and calibration data fixed, how does direct nonlinear supervised-loss optimization compare with LossOpt in performance and computation? What tuning budgets are used? Please also specify the existing tangent-loss comparator's optimization settings.
- For STS-B, do you check the regularized matrix's minimum eigenvalue or apply additional damping? If it remains indefinite, how is the returned solution defined and selected?
3 comments
Responses to Question 3
Q3.
We have explicitly checked the minimum eigenvalue of the aggregate coefficient-space Hessian before solving. For STS-B, the Pearson correlation objective can contribute negative curvature, but this does not necessarily make the aggregate Hessian indefinite.
Let denote the symmetric Hessian averaged over the eight normalized task objectives. We set
and solve .
The first term supplies an additional shift when needed to ensure positive definiteness; the second controls regularization. The solver explicitly verifies that .
Thus, it returns a unique quadratic minimizer, not a stationary point from an indefinite system.
Here, I is the identity matrix and g is the coefficient-space gradient. The first term, delta, supplies an additional shift when needed to ensure positive definiteness; the second controls regularization. The solver explicitly verifies that the minimum eigenvalue of H + lambda_kappa I is positive. Thus, it returns a unique quadratic minimizer, not a stationary point from an indefinite system. The measured minimum eigenvalues are:
| Matrix | Minimum eigenvalue |
|---|---|
| Weighted STS-B contribution | |
| Aggregate Hessian | |
| Regularized aggregate, selected | |
| Regularized aggregate, smallest |
The aggregate Hessian is already positive definite, so no additional positive definiteness shift was needed. The selected gives , and all 12 tested regularized systems were positive definite. We select among their solutions using the validation score and evaluate the selected model once on the test set, obtaining a mean normalized test score of 90.846.
I appreciate the author providing additional experiments, which have addressed some of my concerns. My primary reservation remains that the current evidence is insufficient to determine to what extent the performance gains stem from combining the directions provided by different merging methods. Consequently, I will maintain my current rating.
Responses to Questions 1 and 2
We sincerely thank the reviewer for the thoughtful questions and insightful suggestions.
Q1.
Our motivation is to retain useful structure already identified by successful merging methods. Existing methods combine task updates using information from parameter, representation, or spectral spaces. Their merged updates therefore provide structured candidate directions, while the local supervised objective determines how these directions should be adjusted around a common merged anchor.
To further examine direction choice independently of coefficient count, we have added a comparison between two 21-dimensional bases on RoBERTa-Base. We compare the original WUDI/Iso-CTS/ESM block basis with a task-vector SVD basis. For each of the six selected blocks and the Rest group, we stack the eight individual task updates and retain the top three singular directions. Both configurations use the same ESM anchor and calibration data:
The original basis improves the test score by 0.621 points at the same dimension. Together with the comparisons in Table 4, this supports the value of the selected directions beyond the number of coefficients.
Q2.
We have added a RoBERTa-Base comparison using the same ESM anchor, 21-dimensional WUDI/Iso-CTS/ESM basis, fixed task heads, and 1,000 calibration examples per task. The table reports mean normalized test scores, the number of candidate models considered for validation selection, and recorded computation times:
LossOpt exceeds direct nonlinear optimization by 0.546 points. Regarding computational cost, direct nonlinear optimization requires repeated network forward and backward passes, totaling 282.5 minutes. LossOpt computes directional Jacobians once in 32.7 minutes, then solves a regularized 21×21 system for each candidate without further network differentiation. Both timings exclude validation and testing.
Direct tangent loss optimization. This comparator uses the same cached directional Jacobians and 12 regularization values. Each run starts from the corresponding LossOpt quadratic solution and uses L-BFGS with the following settings:
The selected candidate uses κ=0.001 and requires 284 loss and gradient evaluations. These evaluations use cached outputs and directional Jacobians, without further forward or backward passes through the nonlinear network.