- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether orthogonality-regularized fine-tuning (OrthoReg) changes the relative effectiveness of different model-merging operators, rather than merely producing a common improvement across operators. The authors introduce an operator-relative compatibility measure and evaluate training–operator interactions using Fixed Depth, LiNeS, TIES, and TSV-Merge. Experiments on three CLIP ViT backbones and Flan-T5-base show significant operator-dependent responses in two vision backbones and the language setting, while ViT-L/14 yields a null result under the primary metric. Additional analyses examine metric dependence, update scaling, and functional curvature.
Critical strengths
-
The paper evaluates multiple merging operators across different training procedures, architectures, and task domains. The paired experimental design and additional robustness analyses provide a relatively comprehensive empirical investigation.
-
The distinction between final merged-model performance and incremental gains over direct addition is useful. The authors also report null results and conduct additional analyses to examine the effects of evaluation metrics and scaling.
Critical weaknesses
-
The finding that different merging operators respond differently to changes in training is somewhat expected, given that these operators manipulate task vectors in fundamentally different ways. The common-shift null hypothesis is restrictive, and rejecting it does not necessarily establish a sufficiently surprising insight.
-
TSV-Merge remains the best-performing operator across all evaluated settings in the main experiments. Therefore, the reported preference reversals do not change the optimal operator-selection decision. It remains unclear how the proposed perspective would lead to better practical outcomes.
-
The main experiments allow different validation-selected configurations and update radii. Notably, the fixed-depth interaction changes sign when configurations and absolute update norms are held fixed. This raises concerns about whether the observed interactions reflect intrinsic operator compatibility or differences in scaling and hyperparameter selection.
-
The hierarchical bootstrap resamples tasks within runs, although the same tasks are shared across runs. This resampling scheme may underestimate uncertainty associated with task composition. Furthermore, the experiments use only three independent training runs, limiting the reliability of statistical inference.
-
A statistically significant operator-response contrast does not necessarily establish a statistically significant ranking reversal. Some reported reversals involve very small performance differences (e.g., 0.16 percentage points between Fixed Depth and LiNeS under OrthoReg on ViT-B/32), and uncertainty in the within-condition rankings is not established.
-
Section 3 is difficult to follow due to the dense notation and insufficient explanation of how the different components relate to each other. In particular, Sections 3.2–3.4 move abruptly between empirical hypotheses, local curvature analysis, and scale normalization without clearly connecting them. Moreover, the relationship between the theoretical analysis of loss-based interference and the empirical accuracy-based interaction remains unclear. A more intuitive explanation and a clearer presentation of the overall framework would substantially improve readability.
Questions for authors
-
Given that TSV-Merge remains the best-performing operator in every evaluated setting, what concrete practical benefit does operator-relative disentanglement provide? Can the authors demonstrate a case where this perspective improves operator selection, computational efficiency, or final merging performance?
-
Can the authors repeat the statistical analysis using a bootstrap procedure that preserves the crossed run–task structure? Do the main conclusions, including the reported significant operator-response contrasts, remain valid under this alternative inference procedure?