acceptodds
Withdrawn from ICLR 2027

Watch Where You Linearize: Predicting and Correcting Model-Merging Interference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging composes separately fine-tuned models by adding their task vectors to a shared backbone. In standard merging pipelines, the resulting functional interference is known only after candidate merges are built and evaluated. We show that a single directional derivative, the receiving model's output response to another task's vector, forecasts the interference before any candidate is evaluated, predicting logit drift, label flips, per-example damage, and per-task accuracy of the merge. The forecast works only in the receiver's local geometry: read at the shared pretrained initialization, the same derivative is nearly useless, because fine-tuning reorients the cross-task responses at the very beginning of training. Freezing the receiver-local responses yields a surrogate whose outputs are affine in the merge coefficients, and with it LR-Merge, a label-free correction that selects coefficients through a sequence of convex teacher-matching problems on task-specific unlabeled probes, never backpropagating through a candidate merged network. On vision backbones and language models, LR-Merge improves every strong base merger we test. Its advantage concentrates where merging is data-scarce or repeated. At low probe budgets it surpasses strong coefficient-optimization baselines given several times more data and compute, and the frozen responses are built once and reused, so each further coefficient setting or task subset costs seconds rather than a rebuild.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, task vectors, task arithmetic, interference, linearization, transfer learning, model editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uFHMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper predicts interference occurring when adding another task vector to a fine-tuned model using the directional derivative. It predicts changes in output, shifts in predicted classes, and task-specific accuracy, reporting that linearizing around the receiving model that gets updated yields better accuracy than doing so around the pre-trained initialization point. Based on this, it proposes LR-Merge, which utilizes linearized outputs and pseudo-labels from teacher models. It evaluates calibration over existing merging results, performance with scarce calibration data, and the cost of iterative coefficient searches across CLIP-based vision models and selected language models.

Critical strengths
  1. The controlled experiment that changes direction at a fixed distance provides evidence that simple parameter distance alone struggles to account for predictive performance. Repeating the difference in predictive performance between the initialization point and the receiving model across six independent training sets also enhances the reliability of the observation.

  2. It consistently improved the accuracy of four existing merging methods across six independent training sets.

Critical weaknesses
  1. It is not sufficiently clear what new insight the core comparison regarding the linearization point provides. Since the prediction target in Eq. (1) is the output change starting from theta_A, approximating it with the Jacobian at the same point is standard local Taylor expansion. Conversely, the comparison using J_A(theta_0)tau_B introduces an additional error from transferring a derivative calculated at a different point. Therefore, the nature of this comparison should be made clear when interpreting the lower predictive performance at theta_0. Prior work [3] have already shown that post-hoc linearizing a nonlinearly fine-tuned CLIP model around its initialization point fails to sufficiently reproduce the original model's performance. While the controlled experiment at a fixed distance and the measurement of initial rotation in this paper are useful, it should be made clearer what new explanations they provide beyond the known limitations of linearization at the initialization point.

  2. Predicting merging performance using cross-task information alone can hardly be considered a novel contribution. SyMerge [1] already analyzes the relationship between cross-task performance and merging quality, while Demystifying Mergeability [2] predicts merging quality using model-pair metrics calculated prior to merging. Clearly differentiating your approach from these papers would significantly strengthen the paper.

  3. Figure 1 is introduced before defining key notation such as z_A and J_A, making it hard to grasp the figure's meaning upon first reading. It would be better to explain the necessary symbols in the caption or move the figure after the problem formulation in Section 3. Table 1 outlines the purpose of each metric, but does not provide sufficient calculation details required for reproducibility. Specifically, the definition of damage rho and the mean drift cos should be clarified.

[1] SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation, ICML 2026

[2] Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success, ICML2026

[3] Task Arithmetic in the Tangent Space:Improved Editing of Pre-Trained Models, NeurIPS 2023

Questions for authors
  1. Eq. (1) fixes the coefficient of task A to 1, whereas general two-task merging is represented as theta_0 + beta * tau_A + alpha * tau_B. Section 5 presents an extension optimizing all coefficients, but can you verify to what extent the predictive performance and conclusions in Section 3 hold when beta also varies?

  2. What is the reason for excluding Stanford Cars and using CIFAR-10 instead within the standard 8-task setup from existing Task Arithmetic literature? Please explain how this change affects comparisons with prior work and the reported conclusions.

1 comment

Thank you for the additional experiments. My concerns regarding the evaluation protocol and benchmarks are largely resolved. However, I remain unconvinced that the observed directional dependence and early Jacobian reorientation constitute a sufficiently substantial conceptual advance beyond standard local linearization. Therefore I will maintain my original score.

Reviewer_S4RDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether directional derivatives evaluated at the model receiving a task-vector update can predict the resulting interference. It reports substantially better forecasts at fine-tuned receivers than at the pre-trained initialization. LR-Merge uses these derivatives to construct affine approximations of candidate logits and fit correction coefficients against teacher pseudo-labels, rebuilding the approximation between rounds.

Critical strengths
  1. Useful diagnostic evidence. The receiver versus pre-trained comparison is compelling in the primary vision setting: at merge strength 0.20.2, mean drift cosine is 0.9490.949 versus 0.1220.122 across six checkpoint sets. The matched-distance controls and training trajectories help establish that Euclidean distance alone does not explain forecast quality.
  2. Practical improvements over strong starting points. LR-Merge improves all four evaluated base mergers on every matched checkpoint set in the primary six-task experiment.
Critical weaknesses
  1. The contribution is incompletely positioned. The related-work discussion omits closely connected approaches: Jacobian-based interference regularization in τ\tauJp [1], curvature-based disentanglement in TAK [2], and transfer of linearized behavior to non-linear models in DELTA [3]. Their training-time access assumptions differ from those of post-hoc correction, but this relationship needs explanation. The paper should clarify what its anchor analysis adds and whether the observed gap is expected to persist for regularized or distilled checkpoints.
  2. The interpretation of receiver locality is broader than the evidence. The term initially means evaluating the derivative at the model being updated, then shifts to describing selected alternative anchors with favorable displacement directions. These should be distinct claims. The nine-anchor controls demonstrate directional dependence, but Appendix B acknowledges that the receiver-task projection is incomplete: the span-of-others anchor has a normalized projection of only 0.070.07 and still drops to a mean cosine of 0.6650.665. Likewise, claims that pre-trained responses carry almost no usable signal should be restricted to the studied regime. Table 5 reports pre-trained cosines of 0.23–0.680.23–0.68 for Pythia-410M and 0.53–0.540.53–0.54 in the two-pair Pythia-1.4B pilot. These results support receiver superiority while qualifying the broader language about pre-trained responses becoming obsolete.
  3. Key diagnostic quantities remain insufficiently specified. Operational descriptions of the four forecast metrics do not substitute for computational definitions. The paper should define the per-example damage scalar, cosine computation, evaluation population, and aggregation across pairs, probe seeds, and checkpoint sets. Accuracy and damage evaluation should state where ground-truth labels enter. Flip recall needs precision, flip prevalence, a no-change baseline, and treatment of pairs with no flips. Figure 2 also needs an explicit realized update and merge strength, confirmation of whether the source vector remains fixed at its final checkpoint, and uncertainty or individual trajectories. Table 7 provides useful replication but does not resolve these ambiguities.
  4. Benchmark identity and the scope of generalization are unclear. The primary suite includes CIFAR-10 and omits Cars, RESISC45, and SUN397; the declared eight-task extension adds the latter two but still omits Cars. Figure 4 instead labels its right panel as the canonical eight-task benchmark. A protocol table should resolve whether this panel uses different task membership and justify the suite choices. Larger-backbone corrections use one checkpoint set per backbone, and language correction uses one LoRA fine-tune with three probe/optimizer draws. These are useful pilots and should be distinguished prominently from independently replicated primary results. One nested ordering up to eight tasks and one task arrival provide limited support for broader scalability claims.
  5. Cost and retained-data assumptions need a fuller accounting. Exact response construction grows quadratically with task count, with an additional factor for parameter groups. Larger-backbone corrections take roughly 3.6003.600s and 29.00029.000s, making a main-text summary of accuracy, build time, solve time, peak memory, and cache storage important. Repeated-query savings are credible within a fixed anchor and response basis; new directions and re-anchoring require additional information or computation. The task-arrival experiment should specify whether old probes, cached teacher targets, teachers, or parameter-space derivative representations remain available. Separately, Appendix A.2 uses intermediate parameter instantiations and multiple derivative evaluations for path integration. Its near-exact forecasts need a cost comparison with direct evaluation and a clear distinction from single-anchor forecasting.
  6. Methodology presentation. The method section would benefit from introducing the coefficient vector and its reference model before the objective, then describing each round as forming the anchor, freezing responses, fitting coefficients, applying the update, and rebuilding responses. The grouped variant should first explain that task-vector pieces receive independent coefficients, then state the coefficient count and identify which experiments use it.

References:

  • [1] Yoshida, Kotaro, et al. "Mastering task arithmetic: $\tau $ jp as a key indicator for weight disentanglement." International Conference on Learning Representations. Vol. 2025. 2025.
  • [2] Porrello, Angelo, et al. "Dataless weight disentanglement in task arithmetic via kronecker-factored approximate curvature." arXiv preprint arXiv:2602.17385 (2026).
  • [3] Sommariva, Thomas, et al. "Distilling linearized behavior into non-linear fine-tuning for effective task arithmetic." arXiv preprint arXiv:2605.18993 (2026).
Questions for authors
  • How does the anchor gap change for checkpoints trained with Jacobian or curvature regularization, or distilled to preserve linearized behavior? Can the claims be explicitly scoped if such experiments are unavailable?
  • Can you provide exact metric definitions and the full Figure 2 protocol, including label access, source-vector construction, merge strength, aggregation, and uncertainty?
  • Which tasks appear in Figure 4’s canonical eight-task panel, and how does it relate to the suite declared elsewhere? Can you provide a clearly identified standard-benchmark comparison?
  • What information must remain available after deployment to add a new task direction and to re-anchor? What are the resulting data-retention, storage, memory, and time costs?
  • Can you summarize the diagnostic metrics and correction costs across backbones in the main text, clearly label the pilots, and either evaluate a larger established workload with multiple compositions or narrow the scalability claim?
Reviewer_Uaw5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper “Watch Where You Linearize: Predicting and Correcting Model-Merging Interference” studies how to forecast a merged model’s performance to accelerate the tuning of merging coefficients. It uses directional derivatives to characterize how a receiving model’s outputs respond to another task vector, enabling predictions of interference, logit drift, and example-level label flips before evaluating a merge candidate. The authors propose LR-Merge, a label-free method that selects scaling coefficients by solving a sequence of convex teacher-matching problems on unlabeled probes. Unlike prior methods such as AdaMerging, LR-Merge does not require backpropagation through merged models.

Critical strengths

[1] The work addresses an overlooked problem in the field of model merging, which is the high compute cost of tuning merging hyperparameters with the merge-then-eval pipeline. Using linear approximation to forecast a merged model’s performance is an interesting direction worth exploring. Additionally, the method generalizes to any merging technique.

[2] The authors discover that anchoring with a fine-tuned receiver model works but fails with the pre-trained model.

Critical weaknesses

[1] It is unclear if using a first-order approximation would work for autoregressive generative models such as LLMs, which is the setting of most interest and the one where tuning is actually very expensive.


[2] The authors compare against the outdated and inefficient implementation of Adamerging. There have been several reimplementations of the method, such as in [a], which shows that Adamerging can run in 10 mins for the 8-task setting, which is less than the time taken by LR-Merge as seen in Table 3.

References:

[a] Lee, Chanhyuk, et al. "Adarank: Adaptive rank pruning for enhanced model merging." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

[1] How does the method compare to more recent gradient-based methods like AdaRank [a] Merging?

[2] In Fig. 3A, is LW Adamerging being applied to tune layer-wise scaling coefficients for regMean, but its performance is lower than the regMean baseline, as seen in Table 3, which notes its performance is 87.49%?

[3] How do the methods perform when evaluated on the 14- and 20-task vision benchmark?

References:

[a] Lee, Chanhyuk, et al. "Adarank: Adaptive rank pruning for enhanced model merging." International Conference on Learning Representations. Vol. 2026. 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.