acceptodds
Under review as a conference paper at ICLR 2027

When Does Orthogonal Training Help Model Merging? Operator-Relative Disentanglement in Task Arithmetic

Abstract

Task arithmetic combines independently fine-tuned models through their parameter updates. Training methods and merge rules are usually assessed one component at a time, leaving open whether a training intervention benefits merge rules uniformly or changes their relative utility. We study orthogonality-regularized training under a stricter test: after allowing every merge rule to improve or deteriorate together, does training still change their gains over direct addition? We formalize this question as operator-relative disentanglement and test multiplicity-controlled differences between operator responses. Across image and language task suites, orthogonal training changes relative operator responses rather than producing only a common gain shift. Under our prespecified inference procedure, two vision backbones exhibit resolved preference reversals, a language suite exhibits resolved changes in relative gaps, and a larger vision backbone provides a null result under the primary metric. Reaggregating the same frozen outputs using worst-quartile rather than average utility also changes the response pattern. Shared-radius searches, fixed-map equal-norm audits, validation-only selection, and identity-inclusive deployment separate compatibility from global scale and endpoint quality. A local second-order analysis explains why Euclidean orthogonality alone cannot guarantee compatibility after a merge rule transforms the updates. These results motivate evaluating training, merging, and functional metrics as a coupled system.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model Merging, Task Arithmetic, Task-Vector Disentanglement, Training–Merging Compatibility, Orthogonal Regularization, Operator-Relative Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UmNwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether orthogonality-regularized fine-tuning (OrthoReg) changes the relative effectiveness of different model-merging operators, rather than merely producing a common improvement across operators. The authors introduce an operator-relative compatibility measure and evaluate training–operator interactions using Fixed Depth, LiNeS, TIES, and TSV-Merge. Experiments on three CLIP ViT backbones and Flan-T5-base show significant operator-dependent responses in two vision backbones and the language setting, while ViT-L/14 yields a null result under the primary metric. Additional analyses examine metric dependence, update scaling, and functional curvature.

Critical strengths
  • The paper evaluates multiple merging operators across different training procedures, architectures, and task domains. The paired experimental design and additional robustness analyses provide a relatively comprehensive empirical investigation.

  • The distinction between final merged-model performance and incremental gains over direct addition is useful. The authors also report null results and conduct additional analyses to examine the effects of evaluation metrics and scaling.

Critical weaknesses
  • The finding that different merging operators respond differently to changes in training is somewhat expected, given that these operators manipulate task vectors in fundamentally different ways. The common-shift null hypothesis is restrictive, and rejecting it does not necessarily establish a sufficiently surprising insight.

  • TSV-Merge remains the best-performing operator across all evaluated settings in the main experiments. Therefore, the reported preference reversals do not change the optimal operator-selection decision. It remains unclear how the proposed perspective would lead to better practical outcomes.

  • The main experiments allow different validation-selected configurations and update radii. Notably, the fixed-depth interaction changes sign when configurations and absolute update norms are held fixed. This raises concerns about whether the observed interactions reflect intrinsic operator compatibility or differences in scaling and hyperparameter selection.

  • The hierarchical bootstrap resamples tasks within runs, although the same tasks are shared across runs. This resampling scheme may underestimate uncertainty associated with task composition. Furthermore, the experiments use only three independent training runs, limiting the reliability of statistical inference.

  • A statistically significant operator-response contrast does not necessarily establish a statistically significant ranking reversal. Some reported reversals involve very small performance differences (e.g., 0.16 percentage points between Fixed Depth and LiNeS under OrthoReg on ViT-B/32), and uncertainty in the within-condition rankings is not established.

  • Section 3 is difficult to follow due to the dense notation and insufficient explanation of how the different components relate to each other. In particular, Sections 3.2–3.4 move abruptly between empirical hypotheses, local curvature analysis, and scale normalization without clearly connecting them. Moreover, the relationship between the theoretical analysis of loss-based interference and the empirical accuracy-based interaction remains unclear. A more intuitive explanation and a clearer presentation of the overall framework would substantially improve readability.

Questions for authors
  • Given that TSV-Merge remains the best-performing operator in every evaluated setting, what concrete practical benefit does operator-relative disentanglement provide? Can the authors demonstrate a case where this perspective improves operator selection, computational efficiency, or final merging performance?

  • Can the authors repeat the statistical analysis using a bootstrap procedure that preserves the crossed run–task structure? Do the main conclusions, including the reported significant operator-response contrasts, remain valid under this alternative inference procedure?

Reviewer_MgKYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether a training method that makes task updates orthogonal (OrthoReg) improves all merging rules uniformly, or changes which of them work best. To answer this, it compares the gain of each merging rule over plain addition of the task vectors, before and after orthogonal training, and uses a bootstrap to check whether the differences between rules go beyond a common shift. Experiments use three CLIP ViTs on eight vision tasks and Flan-T5 on GLUE, with four families of merging rules (a depth-weighting probe, LiNeS, TIES and TSV-Merge) and three runs. The authors find significant differences between rules on two backbones and on the language suite, a null result on the largest backbone, and a change in the overall picture when evaluating the worst case instead of the average. A local second-order analysis argues that Euclidean orthogonality alone does not guarantee compatibility after merging.

Critical strengths

The question is interesting and underexplored: most work studies either training-side methods or merge-side methods in isolation, and the interaction between the two is rarely analyzed. Looking at gains relative to a re-tuned identity merge within each training geometry is a sensible way to frame it.

Critical weaknesses

(1) The evidence does not support the scope of the claims, and the findings are unclear.

(1a) The "training" factor is a single method, OrthoReg, at one regularization strength (lambda = 10). OrthoReg enforces orthogonality in the Euclidean sense, which is exactly what the paper's own analysis (Sec. 3.3) argues is insufficient.

(1b) Methods that induce task-wise orthogonality through functional or curvature information around the pretrained model (e.g., [a], [b]) are never tested, although they are the natural test of the paper's own mechanism.

(1c) As a result, the title ("When does orthogonal training help...") and the abstract promise more than is delivered: no condition under which orthogonal training helps emerges.

(1d) What is shown is that the common-shift null is rejected on B/32, B/16 and the language suite and not on L/14. It is not clear which results are findings, which are descriptive, and which promises of the title and abstract are actually fulfilled.

(1e) Presentation compounds this: the paper is very hard to follow, and several paragraphs are difficult to read because of what appears to be AI-assisted writing (consistent with the authors' AI use statement, which mentions drafting and polishing). The text is dense and over-qualified (heavy hedging, many defined terms, repeated statements of what is not claimed, etc.).

[a] Yoshida, K., Naraki, Y., Horie, T., Yamaki, R., Shimizu, R., Saito, Y., ... & Naganuma, H. (2025, May). Mastering task arithmetic: tau Jp as a key indicator for weight disentanglement. In International Conference on Learning Representations (Vol. 2025, pp. 29289-29313).

[b] Porrello, A., Buzzega, P., Dangel, F., Sommariva, T., Salami, R., Bonicelli, L., and Calderara, S. Dataless weight disentanglement in task arithmetic via Kronecker-factored approximate curvature. In The Fourteenth International Conference on Learning Representations, 2026.

(2) Normalized utilities hide absolute performance. Gains are reported on normalized utility, whose denominators (specialist accuracies) differ between training geometries; for ViT-L/14 the OrthoReg identity merge reaches 109.24% normalized utility. Table 5 gives raw accuracy only for identity and TSV-Merge. From the raw and normalized identity values (e.g., 74.56 vs 96.06 on ViT-B/32), the OrthoReg specialists appear to be roughly 10 to 13 points less accurate than Standard ones, if I compute correctly. If so, "more mergeable" may partly reflect less-trained specialists, and smaller gains for interference-resolving operators may reflect less interference to resolve. Absolute accuracies are needed to interpret the central contrasts.

Questions for authors
  1. Please report absolute accuracy (per backbone, averaged over tasks) of the single-task specialists and of all merged models (identity, fixed depth, LiNeS, TIES, TSV-Merge) under both training regimes. Do the conclusions hold when specialist accuracy is matched, e.g., by choosing λ?
  2. Do the conclusions change with a non-Euclidean (curvature- or functional-metric-based) orthogonality regularizer, which your Sec. 3.3 suggests is the relevant notion?
  3. What concrete answer does the paper give to "when does orthogonal training help"?
Reviewer_PsieOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a training-side intervention for task arithmetic changes the relative usefulness of downstream merge operators, rather than shifting all operator gains together. It defines compatibility as the gain of a validation-selected operator family over an identity merge reselected within the same training geometry. The training–operator interaction is the OrthoReg-minus-Standard difference in compatibility. Pairwise contrasts between operators’ interactions are tested against a “common-shift” null with simultaneous bootstrap intervals, and metric contrasts recompute the same quantities under a different task aggregation of identical outputs.

Critical strengths
  1. The question is under-explored and correctly framed. Training-side and merge-side methods are usually evaluated in isolation, and the paper makes the useful point that an absolute improvement from a training intervention is not the same as an improvement in each operator’s marginal value. The comparison of Table 5 with Table 1 shows concretely that the two can diverge.

  2. The experimental protocol is careful and transparent. It uses a within-geometry reselected identity comparator, a shared radius grid, paired runs, validation-only selection with a single test evaluation, disclosed search budgets, and an identity-inclusive deployment view. Null results on ViT-L/14 are reported as nulls rather than reframed as evidence, which is commendable.

Critical weaknesses
  1. The training factor is a single intervention, while the claims concern “training geometry.” Training has only two levels, standard fine-tuning and OrthoReg. The λ sweep is limited to ViT-B/32 and, as the authors acknowledge, is not an independent geometry. Yet the abstract and conclusion generalize to source training and task-vector disentanglement.

  2. The intervention is not characterized, and neither orthogonality nor disentanglement is measured. OrthoReg penalizes Σ‖ΔWTΔW−I‖F2Σ‖ΔWᵀΔW − I‖²_F, a target of identity rather than cI. This drives every singular value of each update toward one, i.e., toward full-rank updates of fixed, large norm, but the main text describes the intervention only as an “internal orthogonal structure.” Moreover, although “disentanglement” appears in the title, the weight-disentanglement error of Ortiz-Jimenez et al. (2023) is never computed, and Section 3.1 concedes that compatibility is not a certificate of weight disentanglement.

Questions for authors
  1. Can you add at least one additional training regime and show whether the operator-response heterogeneity persists? Natural candidates are TTA or attention-only fine-tuning with and without OrthoReg, both available in the OrthoReg codebase, or a non-orthogonal regularizer.

  2. For all backbones and both regimes, can you report ‖ΔWTΔW−I‖F‖ΔWᵀΔW − I‖_F, singular-value spectra, update norms, cross-task cosines and single-task accuracies? Can you also include a control matched in update norm and specialist accuracy but without the orthogonality penalty, to show how much of Δint survives?

  3. Can you report the weight-disentanglement error for both regimes, ideally in an operator-specific form, and show whether Δint tracks it? Without such a measurement, what justifies the term “operator-relative disentanglement”?

  4. Why do the language operator definitions and radius grid differ from the vision ones? Do the language conclusions hold with vision-identical operators, with STS-B excluded, and with extended grids that remove the boundary selections in both domains?

  5. How do you rule out the substitution account, and would an effect-size or equivalence formulation of the common-shift null change the conclusions? Which contrasts remain resolved under studentized max-t simultaneous intervals, including on ViT-L/14, and with the absolute-norm audit also anchored to the OrthoReg identity norm?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.