- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper predicts interference occurring when adding another task vector to a fine-tuned model using the directional derivative. It predicts changes in output, shifts in predicted classes, and task-specific accuracy, reporting that linearizing around the receiving model that gets updated yields better accuracy than doing so around the pre-trained initialization point. Based on this, it proposes LR-Merge, which utilizes linearized outputs and pseudo-labels from teacher models. It evaluates calibration over existing merging results, performance with scarce calibration data, and the cost of iterative coefficient searches across CLIP-based vision models and selected language models.
Critical strengths
-
The controlled experiment that changes direction at a fixed distance provides evidence that simple parameter distance alone struggles to account for predictive performance. Repeating the difference in predictive performance between the initialization point and the receiving model across six independent training sets also enhances the reliability of the observation.
-
It consistently improved the accuracy of four existing merging methods across six independent training sets.
Critical weaknesses
-
It is not sufficiently clear what new insight the core comparison regarding the linearization point provides. Since the prediction target in Eq. (1) is the output change starting from theta_A, approximating it with the Jacobian at the same point is standard local Taylor expansion. Conversely, the comparison using J_A(theta_0)tau_B introduces an additional error from transferring a derivative calculated at a different point. Therefore, the nature of this comparison should be made clear when interpreting the lower predictive performance at theta_0. Prior work [3] have already shown that post-hoc linearizing a nonlinearly fine-tuned CLIP model around its initialization point fails to sufficiently reproduce the original model's performance. While the controlled experiment at a fixed distance and the measurement of initial rotation in this paper are useful, it should be made clearer what new explanations they provide beyond the known limitations of linearization at the initialization point.
-
Predicting merging performance using cross-task information alone can hardly be considered a novel contribution. SyMerge [1] already analyzes the relationship between cross-task performance and merging quality, while Demystifying Mergeability [2] predicts merging quality using model-pair metrics calculated prior to merging. Clearly differentiating your approach from these papers would significantly strengthen the paper.
-
Figure 1 is introduced before defining key notation such as z_A and J_A, making it hard to grasp the figure's meaning upon first reading. It would be better to explain the necessary symbols in the caption or move the figure after the problem formulation in Section 3. Table 1 outlines the purpose of each metric, but does not provide sufficient calculation details required for reproducibility. Specifically, the definition of damage rho and the mean drift cos should be clarified.
[1] SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation, ICML 2026
[2] Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success, ICML2026
[3] Task Arithmetic in the Tangent Space:Improved Editing of Pre-Trained Models, NeurIPS 2023
Questions for authors
-
Eq. (1) fixes the coefficient of task A to 1, whereas general two-task merging is represented as theta_0 + beta * tau_A + alpha * tau_B. Section 5 presents an extension optimizing all coefficients, but can you verify to what extent the predictive performance and conclusions in Section 3 hold when beta also varies?
-
What is the reason for excluding Stanford Cars and using CIFAR-10 instead within the standard 8-task setup from existing Task Arithmetic literature? Please explain how this change affects comparisons with prior work and the reported conclusions.
Thank you for the additional experiments. My concerns regarding the evaluation protocol and benchmarks are largely resolved. However, I remain unconvinced that the observed directional dependence and early Jacobian reorientation constitute a sufficiently substantial conceptual advance beyond standard local linearization. Therefore I will maintain my original score.