acceptodds
Under review as a conference paper at ICLR 2027

OBMerge: Optimal Brain Merge via Iterative Parameter Replacement

Abstract

Model merging combines task-specific fine-tuned models without joint training. Replacing some fine-tuned parameters with merged values can reduce conflicts between tasks, but deciding which parameters to replace requires estimating the loss change caused by each replacement. We introduce Optimal Brain Merge (OBMerge), an iterative model-merging framework based on parameter replacement. OBMerge adapts Optimal Brain Damage to model merging, using a Taylor expansion to estimate the loss change from replacing a fine-tuned parameter with its corresponding merged value, rather than setting it to zero as in pruning. Starting from an initial merge, OBMerge follows a score & replace re-merge loop. At each iteration, we replace low-scoring parameters with their corresponding merged values while retaining high-scoring task-specific parameters, then re-merge the modified models. We also compare replacement scores across objectives and task scopes. We further introduce a pipeline in which an LLM generates calibration data using task and dataset descriptions together with instruction templates. We use the resulting synthetic data for replacement scoring, merging, and selecting merging coefficients. Experiments on two vision and three language models show that OBMerge achieves the highest average accuracy among merging methods on four of the five models, and that synthetic calibration data can give results comparable to real calibration data. For example, on seven commonsense reasoning tasks with Llama-3.2-1B, OBMerge achieves , compared with for RegMean and for RegMean with synthetic calibration data.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Task Vector, Multi-task Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7rZuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OBMerge, an iterative model merging framework that adapts Optimal Brain Damage (OBD) from pruning to model merging, using Taylor expansion to estimate loss changes and decide which fine-tuned parameters should be replaced with corresponding merged values. The method follows a score-and-replace then re-merge loop, built primarily on top of RegMean and RegMean++ as base merging operators. The paper also derives and compares multiple replacement score variants under different loss functions and task scopes, and proposes an LLM-based pipeline to generate synthetic calibration data as a substitute for real data. Experiments across vision and language models show consistent accuracy improvements over baseline merging methods.

Critical strengths

The synthetic calibration data pipeline is well-motivated and addresses a meaningful practical bottleneck. Data-dependent merging methods typically require real calibration examples and incur storage costs for precomputed Gram matrices; this work shows that LLM-generated synthetic data from task descriptions and instruction templates can deliver comparable performance, which extends the applicability of merging methods to low-data or data-restricted scenarios.

The paper is clearly written and well-structured. The framework derivation, scoring variants, experimental setup, and ablation results are presented in a logical, easy-to-follow manner, with sufficient technical details and supplementary materials to support reproducibility.

Critical weaknesses

The technical contribution is largely incremental rather than conceptually novel. Adapting the OBD criterion from "setting parameters to zero" to "replacing parameters with merged values" is a natural and straightforward variant of the original pruning idea, and the overall framework is essentially a combination of existing components (OBD-style importance scoring + RegMean merging + iterative refinement) without fundamentally new insights into the mechanism of model merging. This limits the conceptual novelty and broader impact of the work.

The method is tightly coupled to reconstruction-based merging operators and lacks generalizability. As shown in the appendix, OBMerge delivers marginal gains with ACTMat/TSV-M and causes severe performance collapse with Iso-C/Iso-CTS in later iterations. This narrow compatibility undermines the framing of OBMerge as a general-purpose merging framework, and restricts its practical utility to a small subset of merging methods.

Questions for authors

The iterative replacement schedule and ratio choices currently lack principled guidance. The paper evaluates only a small set of linear schedules and selects the best one empirically, but provides no theory or heuristic for how to set replacement ratios or determine stopping criteria for unseen tasks and model architectures. How should practitioners configure these hyperparameters without exhaustive tuning?

It is unclear whether the computational overhead of multiple re-merging rounds is justified compared to a well-tuned single-pass baseline. Could you provide a direct comparison of computational cost (wall-clock time, memory usage) between OBMerge and tuned RegMean++, and quantify the performance gain per additional iteration round?

Have you investigated potential modifications to improve OBMerge’s compatibility with non-reconstruction-based merging operators? For example, would adjusting the scoring function, replacement schedule, or re-merging logic prevent the performance collapse observed with Iso-C and Iso-CTS?

3 comments
AuthorsOpenReview ↗

Author response to Reviewer 7rZu

Thank you for your comments.

W1: Contribution and replacement target.

To our knowledge, OBMerge is the first to use estimated objective changes to guide replacement with merged values. We also introduce an LLM-based pipeline to generate calibration data for scoring, merging, and coefficient selection. The fundamental difference is the replacement target: OBMerge replaces fine-tuned parameters with values from the previous merge, while OBIM and SA-Merging zero task-vector entries, restoring pre-trained parameter values.

OBIM uses a layer-wise MSE between pre-trained and fine-tuned outputs, with both models fixed. OBMerge instead scores replacement with values from the previous merge, using global task loss or local reconstruction loss under total or separate task scopes (Table 1 in the manuscript).

We evaluate OBIM's replacement with pre-trained values on ViT-B/32, holding RegMean, the fine-tuned models, calibration data, and layer-wise selection fixed. Note that OBIM's objective does not involve the evolving merged model, so it cannot directly guide multi-round refinement of the merge. Given the pre-trained parameter θp\theta_p and its original fine-tuned value θi,p0\theta_{i,p}^0, we rank parameters p=(r,c)p=(r,c) by Si,p=(θp−θi,p0)2[Xi0(Xi0)⊤]ccS_{i,p}=(\theta_p-\theta_{i,p}^0)^2[\boldsymbol{X}_i^0(\boldsymbol{X}_i^0)^\top]_{cc}. For each ρ\rho, we independently reset the lowest-scoring fraction of parameters in the original fine-tuned models and merge once. Accuracy drops from 82.74% to 72.07% at ρ=0.9\rho=0.9, versus 84.00% for OBMerge (Table 18).

Table 18. Replacement with pre-trained values on ViT-B/32: average test accuracy (%).

Replacement ratio ρ\rhoViT-B/32
0 (RegMean)82.74
0.182.73
0.282.69
0.382.61
0.482.43
0.582.07
0.681.43
0.780.18
0.877.47
0.972.07

SA-Merging multiplies connectivity-score gradients by the aggregate task vector and prunes low-scoring entries at a fixed per-round ratio. However, each round's merged model is obtained from the sum of the cumulatively pruned task vectors, whereas OBMerge obtains its merged model by re-merging fine-tuned models after replacing selected parameters with values from the previous merge. At each round, OBMerge starts from the original fine-tuned models and uses Taylor-based replacement scores to select an increasing fraction of low-scoring parameters for replacement.

We implemented SA-Merging following Algorithm 1 in the SA-Merging paper, as official code was not publicly available to us. We use SynFlow's reference code to compute its connectivity score (SynFlow reference implementation).

To compute each fine-tuned model's connectivity score, we run a full-model forward pass with continuous embeddings filled with ones and absolute weights, then sum the outputs (CLIP pooled features or LM logits). We use 50 tokens for ViT-B/32 and 257 for ViT-L/14, including the CLS token, and 256 tokens for CSR. We prune at p=0.2p=0.2 per round for T=10T=10 rounds, then add the final task-vector sum to the pre-trained parameters.

Table 19 and Appendix C.5 show that OBMerge outperforms both OBIM and SA-Merging on all four models. For example, on ViT-B/32, OBMerge reaches 84.00%, compared with 72.68% for OBIM and 39.44% for SA-Merging. On Llama-3.2-1B, OBMerge reaches 84.70%, compared with 61.41% for OBIM and 71.50% for SA-Merging.

Table 19. Merging results: average accuracy (%) on vision and CSR. SA-Merging uses our SynFlow-based implementation. Bold marks the best result per model.

MethodViT-B/32 (Vision)ViT-L/14 (Vision)Qwen3-0.6B (CSR)Llama-3.2-1B (CSR)
OBIM72.6883.8472.0161.41
SA-Merging39.4451.9677.8071.50
RegMean w/ OBMerge (LS2)84.0091.1683.9484.70
AuthorsOpenReview ↗

Response continued

W2, Q1, and Q3: Schedules and operator compatibility.

In Figure 2 of the manuscript, we evaluate six replacement schedules with different starting ratios and numbers of iterations. All six schedules, including the single-iteration schedule, improve on RegMean across the three language models. These results demonstrate that OBMerge works well across multiple schedules, so a simple schedule can yield good results. We leave optimal scheduling to future work.

The accuracy drop in later iterations with Iso-C/Iso-CTS might occur because re-merging adds the same task-vector updates multiple times. As more parameters are replaced, each modified fine-tuned model contains more values from the previous merge. Iso-C sums the resulting task vectors, while Iso-CTS constructs its common subspace from their sum, potentially amplifying these updates at each iteration. RegMean minimizes a closely related reconstruction objective, which may explain its more stable behavior.

To avoid performance deterioration, we select a model from 10 iterations using 100 COCO val2017 (Lin et al., 2014) image–caption pairs for vision and 100 fixed self-generated validation examples for each CSR task. We show the results in Table 20 (Appendix C.4). For example, on Qwen3-0.6B, we select iteration 6 for Iso-CTS with OBMerge and obtain 83.05% accuracy. Extending our proposed self-generated calibration method to generate task-specific synthetic images may improve model selection; we leave this to future work.

Table 20. Average test accuracy (%) on vision and CSR; parentheses indicate selected iterations.

MethodViT-B/32 (Vision)ViT-L/14 (Vision)Qwen3-0.6B (CSR)Llama-3.2-1B (CSR)
Iso-C80.4790.7681.1979.07
w/ OBMerge (LS2)81.69 (2)91.14 (2)83.24 (6)79.30 (3)
Iso-CTS78.9090.4379.1776.54
w/ OBMerge (LS2)81.82 (3)91.47 (3)83.05 (6)78.36 (4)

Q2: Merging cost.

We report end-to-end merging time and peak memory usage on ViT-B/32 in Table 21. End-to-end time includes initialization, Gram collection from 512 images/task, all merging rounds, and checkpoint saving. Ten rounds of OBMerge take 1.44× and 3.73× the time of a single merge for RegMean and RegMean++, respectively.

Figure 2 and Table 12 in the manuscript report per-iteration performance on language and vision models, respectively. For example, RegMean with OBMerge (LS2) on ViT-B/32 reaches 83.20%, 83.68%, 83.83%, and 83.95% at iterations 6–9, corresponding to gains of 0.48, 0.15, and 0.12 percentage points per additional iteration.

Table 21. ViT-B/32 merging cost: end-to-end time (s), peak memory (GiB), and average vision test accuracy (%). Times are means over two A40 runs; memory is the maximum across runs.

OperatorTotal roundsEnd-to-end (s)Peak memory (GiB)Acc. (%)
RegMean183.5214.8482.74
RegMean10119.9017.5684.01
RegMean++1106.9021.3984.55
RegMean++10398.4824.4785.31
AuthorsOpenReview ↗

Kind reminder

Thanks again for your helpful feedback. We have added experiments and clarifications addressing your concerns, including replacement-target ablations and comparisons with OBIM and SA-Merging, validation-based iteration selection for Iso-C/Iso-CTS, and merging-cost measurements.

We would appreciate it if you could take another look at our rebuttal and updated manuscript and reconsider your assessment in light of these results. We would be happy to clarify any remaining questions.

Thank you again for your time!

Reviewer_HkV4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OBMerge, an iterative model-merging framework that replaces selected parameters of task-specific fine-tuned models with their corresponding values from the current merged model, followed by re-merging. Replacement decisions are guided by Taylor-based estimates of loss changes. The framework is instantiated with RegMean and RegMean++ and evaluated on vision and language models. The paper also investigates using LLM-generated synthetic calibration data for replacement scoring, merging, and coefficient selection.

Critical strengths
  • With real calibration data, OBMerge improves the average performance of both RegMean and RegMean++ on all five evaluated vision and language models. These results suggest that the approach can provide useful improvements to reconstruction-based merging beyond a single model or task suite.

  • The score -> replace -> re-merge procedure is straightforward to understand and integrates naturally with the existing RegMean-based pipeline. In particular, under fixed input activations, LS2 exactly measures the local reconstruction-loss change caused by replacing a single parameter and can be computed from the diagonal of the activation Gram matrix.

Critical weaknesses
  • As described in the paper, OBIM already uses Taylor approximations to estimate the loss change associated with resetting fine-tuned parameters to their pretrained values. The principal differences introduced by OBMerge are using the current merged parameters as the replacement target and repeatedly updating scores, replacing parameters, and re-merging with RegMean or RegMean++. Although these extensions may be useful, the methodological advance appears incremental.

  • Figure 2 compares single-iteration and multi-iteration schedules, so the effect of iteration is not entirely unexamined. However, this comparison does not disentangle the contributions of changing the replacement target, updating the parameter rankings, performing repeated re-merging, and using a different merging operator. I would like to see incremental ablations connecting OBIM to OBMerge.

  • The main results use fixed default hyperparameters for the baselines, without validation-based tuning. However, Appendix D.6 shows that selecting merging coefficients can substantially improve several competing methods. Consequently, the main comparisons do not establish whether OBMerge remains advantageous against well-tuned baselines under comparable calibration-data and tuning budgets. Moreover, the default OBMerge configuration performs nine refinement iterations after the initial merge. Its benefit should therefore be assessed not only in terms of final accuracy but also relative to the additional computational cost. The paper should report end-to-end merging time and peak memory usage, including scoring, parameter selection and replacement, and re-merging.

  • The main results report point estimates without means and standard deviations across multiple seeds or confidence intervals. Several reported improvements are below one percentage point, making it difficult to determine whether they are robust to experimental randomness.

  • The replacement scores evaluate each parameter independently while holding the remaining parameters fixed. In practice, however, many parameters are replaced simultaneously. A collection of individually low-scoring replacements need not produce a small joint loss change because interactions between parameters are omitted. The authors acknowledge this limitation in Appendix B, but its practical magnitude is not quantified.

  • The language-model experiments are limited to Qwen3-0.6B, Llama-3.2-1B, and Llama-3.2-3B. Thus, the paper does not establish whether the performance gains and resource requirements remain favorable for models in the 7B class or larger. This concern is distinct from the performance–cost trade-off at the evaluated scales: it concerns whether the method remains feasible and effective as model size increases.

Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Author response to Reviewer HkV4

Thank you for your comments.

W1 and W2: Replacement target, objectives, and score contribution.

To our knowledge, OBMerge is the first to use estimated objective changes to guide replacement with merged values. We also introduce an LLM-based pipeline to generate calibration data for scoring, merging, and coefficient selection. The fundamental difference is the replacement target: OBMerge replaces fine-tuned parameters with values from the previous merge, while OBIM zeros task-vector entries, restoring pre-trained parameter values.

OBIM and OBMerge both use Taylor expansions, but their scoring objectives differ. OBIM uses a layer-wise MSE between pre-trained and fine-tuned outputs, with both models fixed. OBMerge instead estimates the objective change caused by replacing a fine-tuned parameter with its corresponding value from the previous merge. Building on this formulation, we develop a multi-iteration framework and extend scoring to global task loss and local reconstruction loss, each under total and separate task scopes, yielding four scoring objectives (Table 1 in the manuscript). OBMerge outperforms OBIM on all four models (Table 10).

Table 10. Merging results: average accuracy (%) on vision and CSR. Bold marks the best result per model.

MethodViT-B/32 (Vision)ViT-L/14 (Vision)Qwen3-0.6B (CSR)Llama-3.2-1B (CSR)
OBIM72.6883.8472.0161.41
RegMean w/ OBMerge (LS2)84.0091.1683.9484.70

We evaluate OBIM's replacement with pre-trained values on ViT-B/32, holding RegMean, the fine-tuned models, calibration data, and layer-wise selection fixed. Note that OBIM's objective does not involve the evolving merged model, so it cannot directly guide multi-round refinement of the merge. Given the pre-trained parameter θp\theta_p and its original fine-tuned value θi,p0\theta_{i,p}^0, we rank parameters p=(r,c)p=(r,c) by Si,p=(θp−θi,p0)2[Xi0(Xi0)⊤]ccS_{i,p}=(\theta_p-\theta_{i,p}^0)^2[\boldsymbol{X}_i^0(\boldsymbol{X}_i^0)^\top]_{cc}. For each ρ\rho, we independently reset the lowest-scoring fraction of parameters in the original fine-tuned models and merge once. Accuracy drops from 82.74% to 72.07% at ρ=0.9\rho=0.9, versus 84.00% for OBMerge (Table 11).

Table 11. Replacement with pre-trained values on ViT-B/32: average test accuracy (%).

Replacement ratio ρ\rhoViT-B/32
0 (RegMean)82.74
0.182.73
0.282.69
0.382.61
0.482.43
0.582.07
0.681.43
0.780.18
0.877.47
0.972.07

In Figure 2 of the manuscript, we compare single- and multi-iteration schedules and find that all tested schedules improve on RegMean across three language models. Appendix C.4 evaluates LS2 with RegMean, ACTMat, TSV-M, Iso-C, and Iso-CTS: RegMean remains stable and improves over the initial merge, while other operators can improve at intermediate iterations before losing accuracy in later iterations.

We compare random, distance, curvature, and LS2 scores under the separate local reconstruction objective, with identical merging operators and replacement schedules. LS2 performs best on four of five models (Table 12; Appendix C.6). On Llama-3.2-1B, LS2 improves the initial merge by 2.92 percentage points, compared with 0.72 for random replacement. On ViT-B/32, LS2 reaches 84.00%, compared with 82.64% for random replacement.

Table 12. Replacement-score ablation: average accuracy (%) on vision and CSR, and average MergeBench score. Bold marks the best result per model.

MethodViT-B/32 (Vision)ViT-L/14 (Vision)Qwen3-0.6B (CSR)Llama-3.2-1B (CSR)Llama-3.2-3B (MergeBench)
RegMean82.7490.5582.8881.7841.31
w/ OBMerge (Random)82.6490.4882.6082.5041.88
w/ OBMerge (Distance)75.4590.0484.2982.1341.39
w/ OBMerge (Curvature)32.224.9564.0658.4922.11
w/ OBMerge (LS2)84.0091.1683.9484.7043.34
AuthorsOpenReview ↗

Response continued

W3: Baseline tuning and merging cost.

We understand the reviewer's request for a tuned RegMean++ baseline as selecting the best non-diagonal scaling factor α\alpha. Following RegMean and RegMean++, we report sweeps over α∈0.1,0.2,…,0.9,0.95\alpha\in\\{0.1,0.2,\ldots,0.9,0.95\\}, excluding α=1\alpha=1 because these papers report a sharp accuracy drop at this value. Each coefficient is evaluated directly on the full test sets (Table 13; Appendix C.11). For RegMean++, the coefficients used in our manuscript, α=0.95\alpha=0.95 for vision and 0.9 for CSR, give the highest accuracy within this grid. For RegMean on Qwen3-0.6B, α=0.9\alpha=0.9 and 0.95 perform similarly (83.00% versus 83.03%).

OBMerge is complementary to coefficient tuning. In the manuscript's results, OBMerge (LS2) improves RegMean from 82.74% to 83.95% on ViT-B/32 at α=0.95\alpha=0.95, and RegMean++ from 83.30% to 84.15% on Qwen3-0.6B at α=0.9\alpha=0.9 (Table 2).

Table 13. RegMean and RegMean++ coefficient sensitivity: average test accuracy (%) on vision and CSR. Bold marks the best coefficient for each method and model.

MethodViT-B/32 (Vision)Qwen3-0.6B (CSR)
RegMean (α=0.1\alpha=0.1)74.2675.00
RegMean (α=0.2\alpha=0.2)76.4477.76
RegMean (α=0.3\alpha=0.3)77.7479.65
RegMean (α=0.4\alpha=0.4)78.8080.66
RegMean (α=0.5\alpha=0.5)79.5781.49
RegMean (α=0.6\alpha=0.6)80.3182.08
RegMean (α=0.7\alpha=0.7)81.0782.23
RegMean (α=0.8\alpha=0.8)81.7382.41
RegMean (α=0.9\alpha=0.9)82.3983.00
RegMean (α=0.95\alpha=0.95)82.7483.03
RegMean++ (α=0.1\alpha=0.1)75.6275.14
RegMean++ (α=0.2\alpha=0.2)77.6478.27
RegMean++ (α=0.3\alpha=0.3)78.9780.24
RegMean++ (α=0.4\alpha=0.4)80.0281.29
RegMean++ (α=0.5\alpha=0.5)80.9381.95
RegMean++ (α=0.6\alpha=0.6)81.7982.26
RegMean++ (α=0.7\alpha=0.7)82.5982.55
RegMean++ (α=0.8\alpha=0.8)83.4383.10
RegMean++ (α=0.9\alpha=0.9)84.2183.32
RegMean++ (α=0.95\alpha=0.95)84.5583.23

We report end-to-end merging time and peak memory usage on ViT-B/32 in Table 14. End-to-end time includes initialization, Gram collection from 512 images/task, all merging rounds, and checkpoint saving. Ten rounds of OBMerge take 1.44× and 3.73× the time of a single merge for RegMean and RegMean++, respectively.

Table 14. ViT-B/32 merging cost: end-to-end time (s), peak memory (GiB), and average vision test accuracy (%). Times are means over two A40 runs; memory is the maximum across runs.

OperatorTotal roundsEnd-to-end (s)Peak memory (GiB)Acc. (%)
RegMean183.5214.8482.74
RegMean10119.9017.5684.01
RegMean++1106.9021.3984.55
RegMean++10398.4824.4785.31

W4: Gains across calibration seeds.

We run experiments with three calibration seeds (0, 1, and 2) and report the mean and standard error in Table 15 (Appendix C.10). OBMerge improves on RegMean across all four models and all three seeds. For example, on ViT-B/32, mean accuracy increases from 82.80% to 84.01%, with a mean gain of 1.21 ± 0.07 percentage points.

Table 15. Calibration-seed comparison: mean ± SE of average test accuracy (%) over three calibration seeds. Gains are differences in percentage points; bold marks the higher mean accuracy per model.

MethodViT-B/32 (Vision)ViT-L/14 (Vision)Qwen3-0.6B (CSR)Llama-3.2-1B (CSR)
RegMean82.80 ± 0.0390.56 ± 0.0382.89 ± 0.0181.90 ± 0.06
w/ OBMerge (LS2)84.01 ± 0.0791.17 ± 0.0883.99 ± 0.1883.70 ± 0.55
Gain+1.21 ± 0.07+0.61 ± 0.05+1.09 ± 0.17+1.80 ± 0.59
AuthorsOpenReview ↗

Response continued

W5: Joint replacement effects.

We analyze the correlation between predicted replacement scores and actual loss changes after jointly replacing selected parameters and find generally high correlations (Table 16; Appendix C.9).

The table reports total predicted (PtP_t) and actual (AtA_t) losses and Pearson correlations between predicted and actual losses across all weight matrices of the fine-tuned models at each round. At ρt=0.9\rho_t=0.9, actual losses exceed predictions by 4.46% on ViT-B/32 and 31.64% on Qwen3-0.6B.

Table 16. Joint replacement by round: predicted (PtP_t) and actual (AtA_t) loss sums, and Pearson correlations (rtr_t) between predicted and actual losses across all weight matrices of the fine-tuned models at each round.

ρt\rho_tPtP_t (ViT-B/32)AtA_t (ViT-B/32)rtr_t (ViT-B/32)PtP_t (Qwen3-0.6B)AtA_t (Qwen3-0.6B)rtr_t (Qwen3-0.6B)
0.10.200.201.000.010.011.00
0.21.721.711.000.110.121.00
0.36.306.331.000.640.671.00
0.416.6016.721.002.462.581.00
0.537.2037.701.008.118.671.00
0.676.8979.511.0023.1925.580.99
0.7153.14163.971.0059.7168.460.99
0.8298.41327.220.99140.56170.800.98
0.9568.79594.140.99331.78436.750.96

W6: Scaling to 8B.

We evaluate OBMerge and the baselines on Qwen3-8B across the seven CSR tasks. OBMerge achieves the highest average accuracy among the compared methods on this larger model (Table 17; Appendix C.13). For example, it reaches 91.12%, compared with 90.05% for RegMean and 90.36% for ACTMat.

Table 17. Qwen3-8B: average full-test accuracy (%) on seven CSR tasks.

MethodQwen3-8B
Iso-C88.87
Iso-CTS87.91
ACTMat90.36
TSV-M89.62
OBIM80.66
RegMean90.05
w/ OBMerge (LS2)91.12
Reviewer_JbtGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Optimal Brain Merge (OBMerge), an iterative framework for refining model merging through parameter replacement. Inspired by Optimal Brain Damage, the proposed method estimates the loss change associated with replacing individual task-specific parameters with their corresponding values in a merged model. Building upon RegMean and RegMean++, OBMerge iteratively replaces low-scoring parameters and re-merges the modified models. The paper investigates global task loss and local reconstruction loss criteria using first- and second-order Taylor approximations under different task scopes. It also introduces a synthetic calibration pipeline that uses an LLM to generate task-specific examples from dataset descriptions and instruction templates. Experiments on two vision and three language models show improvements over RegMean and RegMean++ with real calibration data, while selected synthetic data configurations obtain competitive results.

Critical strengths

[S1] A practical refinement mechanism. OBMerge offers a straightforward method for refining existing RegMean and RegMean++ solutions without jointly retraining the underlying task-specific models. With real calibration data, the method improves both initial merging operators across all five evaluated models. The improvements are particularly notable for Llama-3.2-1B on commonsense reasoning and Llama-3.2-3B on MergeBench. The method is also compatible with different loss-based replacement criteria, making it a potentially useful extension of reconstruction-oriented merging.

[S2] Broad empirical analysis. The paper evaluates multiple model families and task settings, compares first- and second-order criteria, studies iterative replacement schedules, and analyzes parameter selection agreement. The synthetic calibration experiments use a structured comparison of description sources and instruction templates, providing useful evidence about which forms of task information are necessary for effective data-dependent merging. The inclusion of complete generation prompts and per-task results further supports reproducibility.

Critical weaknesses

[W1] Novelty. Closely related approaches have already investigated parameter importance and iterative model merging. In particular, OBIM [Want et al., arXiv’25] introduces loss-change-based saliency estimation and an iterative parameter mask-based merging procedure. SA-Merging [Park et al., ICML’26] also proposes an iterative saliency-guided merging strategy, although using a different data-free saliency formulation. OBMerge differs by scoring replacement toward the current merged parameters and repeatedly applying reconstruction-based merging operators. However, the paper does not sufficiently demonstrate the benefit of this particular replacement score in the effects of iterative merging and mask selection. The relatively poor performance of the implemented OBIM baseline does not by itself establish which component of OBMerge produces the improvement.

[W2] Missing ablations. The main methodological claim of the paper is that Taylor-based replacement scores identify parameters that can be safely substituted with merged values. However, the experimental evaluation does not adequately distinguish the contribution of these scores from the effects of iterative replacement and re-merging. In particular, the paper does not compare against simple alternatives such as random replacement, parameter distance-based selection, or curvature-only selection under identical merging operators and replacement schedules. Figure 2 demonstrates that multiple iteration schedules can improve performance over RegMean, including a single replacement iteration, but does not establish that the proposed importance criterion is essential for these gains. Moreover, the use of RegMean and RegMean++ may itself contribute substantially to the improvements.

Questions for authors

[Q1] Could the authors compare OBMerge against random replacement, magnitude-based replacement using the distance between fine-tuned and merged parameters, and curvature-based replacement under identical merging operators and replacement schedules? In particular, how much of the improvement is attributable to the proposed loss-change criterion rather than the iterative re-merging procedure itself? These comparisons would help establish whether the proposed score provides a distinct methodological benefit.

[Q2] The replacement scores estimate individual parameter effects, whereas the proposed algorithm simultaneously replaces up to 90% of parameters. Could the authors analyze the correlation between predicted replacement scores and actual loss changes after jointly replacing selected parameters? Additionally, how does the complete first-plus-second-order Taylor score compare with the separate first-order and second-order scores used in the experiments? Such analyses would clarify whether the theoretical motivation adequately explains the behavior of the implemented procedure.

[Q3] Could the authors provide comparisons in which the major baseline methods are allowed to select their merging coefficients and other relevant hyperparameters using the same calibration budget? Appendix D.6 shows that coefficient selection can substantially improve several existing methods, making the fixed-default comparison potentially insufficient.

[Q4] Appendix C.4 reports substantial accuracy degradation when OBMerge is applied to Iso-C and Iso-CTS, despite initial improvements in some iterations. Could the authors clarify what causes these failures and whether they arise from the replacement scores, the iterative update, or incompatibility with the underlying merging operators? Is there a measurable criterion that can determine when iterations should stop before performance deteriorates? These explanations would help characterize the applicability and limitations of the proposed framework.

5 comments
AuthorsOpenReview ↗

Author response to Reviewer JbtG

Thank you for your comments.

W1: Replacement target and comparison with related methods.

To our knowledge, OBMerge is the first to use estimated objective changes to guide replacement with merged values. We also introduce an LLM-based pipeline to generate calibration data for scoring, merging, and coefficient selection. The fundamental difference is the replacement target: OBMerge replaces fine-tuned parameters with values from the previous merge, while OBIM and SA-Merging zero task-vector entries, restoring pre-trained parameter values.

OBIM and OBMerge both use Taylor expansions, but their scoring objectives differ. OBIM uses a layer-wise MSE between pre-trained and fine-tuned outputs, with both models fixed. OBMerge instead estimates the objective change caused by replacing a fine-tuned parameter with its corresponding value from the previous merge. Building on this formulation, we develop a multi-iteration framework and extend scoring to global task loss and local reconstruction loss, each under total and separate task scopes, yielding four scoring objectives (Table 1 in the manuscript).

SA-Merging multiplies connectivity-score gradients by the aggregate task vector and prunes low-scoring entries at a fixed per-round ratio. However, each round's merged model is obtained from the sum of the cumulatively pruned task vectors, whereas OBMerge obtains its merged model by re-merging fine-tuned models after replacing selected parameters with values from the previous merge. At each round, OBMerge starts from the original fine-tuned models and uses Taylor-based replacement scores to select an increasing fraction of low-scoring parameters for replacement.

We implemented SA-Merging following Algorithm 1 in the SA-Merging paper, as official code was not publicly available to us. We use SynFlow's reference code to compute its connectivity score (SynFlow reference implementation).

To compute each fine-tuned model's connectivity score, we run a full-model forward pass with continuous embeddings filled with ones and absolute weights, then sum the outputs (CLIP pooled features or LM logits). We use 50 tokens for ViT-B/32 and 257 for ViT-L/14, including the CLS token, and 256 tokens for CSR. We prune at p=0.2p=0.2 per round for T=10T=10 rounds, then add the final task-vector sum to the pre-trained parameters.

Table 3 and Appendix C.5 show that OBMerge outperforms both OBIM and SA-Merging on all four models. For example, on ViT-B/32, OBMerge reaches 84.00%, compared with 72.68% for OBIM and 39.44% for SA-Merging. On Llama-3.2-1B, OBMerge reaches 84.70%, compared with 61.41% for OBIM and 71.50% for SA-Merging.

Table 3. Merging results: average accuracy (%) on vision and CSR. SA-Merging uses our SynFlow-based implementation. Bold marks the best result per model.

MethodViT-B/32 (Vision)ViT-L/14 (Vision)Qwen3-0.6B (CSR)Llama-3.2-1B (CSR)
OBIM72.6883.8472.0161.41
SA-Merging39.4451.9677.8071.50
RegMean w/ OBMerge (LS2)84.0091.1683.9484.70

We evaluate OBIM's replacement with pre-trained values on ViT-B/32, holding RegMean, the fine-tuned models, calibration data, and layer-wise selection fixed. Note that OBIM's objective does not involve the evolving merged model, so it cannot directly guide multi-round refinement of the merge. Given the pre-trained parameter θp\theta_p and its original fine-tuned value θi,p0\theta_{i,p}^0, we rank parameters p=(r,c)p=(r,c) by Si,p=(θp−θi,p0)2[Xi0(Xi0)⊤]ccS_{i,p}=(\theta_p-\theta_{i,p}^0)^2[\boldsymbol{X}_i^0(\boldsymbol{X}_i^0)^\top]_{cc}. For each ρ\rho, we independently reset the lowest-scoring fraction of parameters in the original fine-tuned models and merge once. Accuracy drops from 82.74% to 72.07% at ρ=0.9\rho=0.9, versus 84.00% for OBMerge (Table 4).

Table 4. Replacement with pre-trained values on ViT-B/32: average test accuracy (%).

Replacement ratio ρ\rhoViT-B/32
0 (RegMean)82.74
0.182.73
0.282.69
0.382.61
0.482.43
0.582.07
0.681.43
0.780.18
0.877.47
0.972.07
AuthorsOpenReview ↗

Response continued

W2 and Q1: Contribution of the replacement score.

We compare random, distance, curvature, and LS2 scores under the separate local reconstruction objective, with identical merging operators and replacement schedules. LS2 performs best on four of five models (Table 5; Appendix C.6). On Llama-3.2-1B, LS2 improves the initial merge by 2.92 percentage points, compared with 0.72 for random replacement. On ViT-B/32, LS2 reaches 84.00%, compared with 82.64% for random replacement.

Specifically, let δ:=δi,pt=θm,pt−1−θi,p0\delta:=\delta_{i,p}^t=\theta_{\mathrm{m},p}^{t-1}-\theta_{i,p}^0, where parameter pp occupies position (r,c)(r,c) in its weight matrix. We define the four scores as

S_{\mathrm{random}} & \sim U(0,1), & S_{\mathrm{distance}} & = \delta^2, \\\\[3pt] S_{\mathrm{curvature}} & = 2[\boldsymbol{X}_i^0(\boldsymbol{X}_i^0)^\top]_{cc}, & S_{\mathrm{LS2}} & = \delta^2[\boldsymbol{X}_i^0(\boldsymbol{X}_i^0)^\top]_{cc}. \end{array}$$ Here $2[\boldsymbol{X}_i^0(\boldsymbol{X}_i^0)^\top]_{cc}$ is the Hessian diagonal entry for parameter $p$ under the separate local reconstruction objective (Table 1 in the manuscript). **Table 5.** Replacement-score ablation: average accuracy (%) on vision and CSR, and average MergeBench score. Bold marks the best result per model. | Method | ViT-B/32 (Vision) | ViT-L/14 (Vision) | Qwen3-0.6B (CSR) | Llama-3.2-1B (CSR) | Llama-3.2-3B (MergeBench) | | :--- | :---: | :---: | :---: | :---: | :---: | | RegMean | 82.74 | 90.55 | 82.88 | 81.78 | 41.31 | | w/ OBMerge (Random) | 82.64 | 90.48 | 82.60 | 82.50 | 41.88 | | w/ OBMerge (Distance) | 75.45 | 90.04 | **84.29** | 82.13 | 41.39 | | w/ OBMerge (Curvature) | 32.22 | 4.95 | 64.06 | 58.49 | 22.11 | | w/ OBMerge (LS2) | **84.00** | **91.16** | 83.94 | **84.70** | **43.34** | ## Q2: Joint replacement and complete Taylor scores. We analyze the correlation between predicted replacement scores and actual loss changes after jointly replacing selected parameters and find generally high correlations (Table 6; Appendix C.9). The table reports total predicted ($P_t$) and actual ($A_t$) losses and Pearson correlations between predicted and actual losses across all weight matrices of the fine-tuned models at each round. At $\rho_t=0.9$, actual losses exceed predictions by 4.46% on ViT-B/32 and 31.64% on Qwen3-0.6B. **Table 6.** Joint replacement by round: predicted ($P_t$) and actual ($A_t$) loss sums, and Pearson correlations ($r_t$) between predicted and actual losses across all weight matrices of the fine-tuned models at each round. | $\rho_t$ | $P_t$ (ViT-B/32) | $A_t$ (ViT-B/32) | $r_t$ (ViT-B/32) | $P_t$ (Qwen3-0.6B) | $A_t$ (Qwen3-0.6B) | $r_t$ (Qwen3-0.6B) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | 0.1 | 0.20 | 0.20 | 1.00 | 0.01 | 0.01 | 1.00 | | 0.2 | 1.72 | 1.71 | 1.00 | 0.11 | 0.12 | 1.00 | | 0.3 | 6.30 | 6.33 | 1.00 | 0.64 | 0.67 | 1.00 | | 0.4 | 16.60 | 16.72 | 1.00 | 2.46 | 2.58 | 1.00 | | 0.5 | 37.20 | 37.70 | 1.00 | 8.11 | 8.67 | 1.00 | | 0.6 | 76.89 | 79.51 | 1.00 | 23.19 | 25.58 | 0.99 | | 0.7 | 153.14 | 163.97 | 1.00 | 59.71 | 68.46 | 0.99 | | 0.8 | 298.41 | 327.22 | 0.99 | 140.56 | 170.80 | 0.98 | | 0.9 | 568.79 | 594.14 | 0.99 | 331.78 | 436.75 | 0.96 | We also investigate combining first- and second-order Taylor terms and find that the benefit depends on the model and scoring objective (Table 7; Appendix C.8). On Qwen3-0.6B, GT1 + GT2 reaches 84.90%, exceeding GT1 (84.44%) and GT2 (84.53%). On ViT-B/32, the same combination reaches 82.67%, below GT2 (83.90%) and LS2 (84.00%). LT1 + LT2 also underperforms LT2 on both models. The scores are $$\begin{aligned} S_{\mathrm{first}}&=|\delta{g}_{i,p}|, & S_{\mathrm{second}}&=\left|\tfrac12\delta^2{H}_{i,pp}\right|,\\\\ S_{\mathrm{combined}}&=\left|\delta{g}_{i,p}+\tfrac12\delta^2{H}_{i,pp}\right|. \end{aligned}$$ For the separate local reconstruction objective, the first-order term vanishes at the original fine-tuned weights (${g}_{i,p}=0$), so LS2 already equals the combined first- and second-order score. **Table 7.** Replacement-score comparison: average test accuracy (%) on vision and CSR. Combined scores take one absolute value after addition. Bold marks the best result per model. | Method | ViT-B/32 (Vision) | Qwen3-0.6B (CSR) | | :--- | :---: | :---: | | RegMean | 82.74 | 82.88 | | w/ OBMerge (LS2) | **84.00** | 83.94 | | w/ OBMerge (LT1) | 7.98 | 76.95 | | w/ OBMerge (LT2) | 80.29 | 84.24 | | w/ OBMerge (LT1 + LT2) | 19.47 | 80.32 | | w/ OBMerge (GT1) | 82.66 | 84.44 | | w/ OBMerge (GT2, Fisher) | 83.90 | 84.53 | | w/ OBMerge (GT1 + GT2, Fisher) | 82.67 | **84.90** | | w/ OBMerge (GS1) | 83.68 | 83.60 | | w/ OBMerge (GS2, Fisher) | 83.73 | 83.79 | | w/ OBMerge (GS1 + GS2, Fisher) | 83.70 | 83.90 |
AuthorsOpenReview ↗

Response continued

Q3: Coefficient tuning and complementarity.

We understand the reviewer's tuning request as selecting the best non-diagonal scaling factor α\alpha for RegMean++. Following RegMean and RegMean++, we report sweeps over α∈0.1,0.2,…,0.9,0.95\alpha\in\\{0.1,0.2,\ldots,0.9,0.95\\}, excluding α=1\alpha=1 because these papers report a sharp accuracy drop at this value. Each coefficient is evaluated directly on the full test sets (Table 8; Appendix C.11). For RegMean++, the coefficients used in our manuscript, α=0.95\alpha=0.95 for vision and 0.9 for CSR, give the highest accuracy within this grid. For RegMean on Qwen3-0.6B, α=0.9\alpha=0.9 and 0.95 perform similarly (83.00% versus 83.03%). This is a test-set sensitivity comparison, rather than coefficient selection under a shared calibration budget.

OBMerge is complementary to coefficient tuning. In the manuscript's results, OBMerge (LS2) improves RegMean from 82.74% to 83.95% on ViT-B/32 at α=0.95\alpha=0.95, and RegMean++ from 83.30% to 84.15% on Qwen3-0.6B at α=0.9\alpha=0.9 (Table 2).

Table 8. RegMean and RegMean++ coefficient sensitivity: average test accuracy (%) on vision and CSR. Bold marks the best coefficient for each method and model.

MethodViT-B/32 (Vision)Qwen3-0.6B (CSR)
RegMean (α=0.1\alpha=0.1)74.2675.00
RegMean (α=0.2\alpha=0.2)76.4477.76
RegMean (α=0.3\alpha=0.3)77.7479.65
RegMean (α=0.4\alpha=0.4)78.8080.66
RegMean (α=0.5\alpha=0.5)79.5781.49
RegMean (α=0.6\alpha=0.6)80.3182.08
RegMean (α=0.7\alpha=0.7)81.0782.23
RegMean (α=0.8\alpha=0.8)81.7382.41
RegMean (α=0.9\alpha=0.9)82.3983.00
RegMean (α=0.95\alpha=0.95)82.7483.03
RegMean++ (α=0.1\alpha=0.1)75.6275.14
RegMean++ (α=0.2\alpha=0.2)77.6478.27
RegMean++ (α=0.3\alpha=0.3)78.9780.24
RegMean++ (α=0.4\alpha=0.4)80.0281.29
RegMean++ (α=0.5\alpha=0.5)80.9381.95
RegMean++ (α=0.6\alpha=0.6)81.7982.26
RegMean++ (α=0.7\alpha=0.7)82.5982.55
RegMean++ (α=0.8\alpha=0.8)83.4383.10
RegMean++ (α=0.9\alpha=0.9)84.2183.32
RegMean++ (α=0.95\alpha=0.95)84.5583.23

Q4: Iteration selection and operator limitations.

The accuracy drop in later iterations with Iso-C/Iso-CTS might occur because re-merging adds the same task-vector updates multiple times. As more parameters are replaced, each modified fine-tuned model contains more values from the previous merge. Iso-C sums the resulting task vectors, while Iso-CTS constructs its common subspace from their sum, potentially amplifying these updates at each iteration. RegMean minimizes a closely related reconstruction objective, which may explain its more stable behavior.

To avoid performance deterioration, we select a model from 10 iterations using 100 COCO val2017 (Lin et al., 2014) image–caption pairs for vision and 100 fixed self-generated validation examples for each CSR task. We show the results in Table 9 (Appendix C.4). For example, on Qwen3-0.6B, we select iteration 6 for Iso-CTS with OBMerge and obtain 83.05% accuracy. Extending our proposed self-generated calibration method to generate task-specific synthetic images may improve model selection; we leave this to future work.

Table 9. Average test accuracy (%) on vision and CSR; parentheses indicate selected iterations.

MethodViT-B/32 (Vision)ViT-L/14 (Vision)Qwen3-0.6B (CSR)Llama-3.2-1B (CSR)
Iso-C80.4790.7681.1979.07
w/ OBMerge (LS2)81.69 (2)91.14 (2)83.24 (6)79.30 (3)
Iso-CTS78.9090.4379.1776.54
w/ OBMerge (LS2)81.82 (3)91.47 (3)83.05 (6)78.36 (4)

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their feedback. To our knowledge, OBMerge is the first to use estimated objective changes to guide replacement with merged values. The fundamental difference is the replacement target: OBMerge replaces fine-tuned parameters with values from the previous merge, while OBIM and SA-Merging zero task-vector entries, restoring pre-trained parameter values. Our replacement scores depend on the evolving merged model and can therefore directly guide multi-round refinement of the merge. We also introduce an LLM-based pipeline that generates calibration data for scoring, merging, and coefficient selection. We add comparisons with OBIM and our SynFlow-based SA-Merging implementation and find that OBMerge outperforms both on all four models (Table 1). For example, on ViT-B/32, OBMerge reaches 84.00%, compared with 72.68% for OBIM and 39.44% for SA-Merging.

Table 1. Merging results: average accuracy (%) on vision and CSR. SA-Merging uses our SynFlow-based implementation. Bold marks the best result per model.

MethodViT-B/32 (Vision)ViT-L/14 (Vision)Qwen3-0.6B (CSR)Llama-3.2-1B (CSR)
OBIM72.6883.8472.0161.41
SA-Merging39.4451.9677.8071.50
RegMean w/ OBMerge (LS2)84.0091.1683.9484.70

We evaluate OBIM's replacement with pre-trained values on ViT-B/32, holding RegMean, the fine-tuned models, calibration data, and layer-wise selection fixed. Note that OBIM's objective does not involve the evolving merged model, so it cannot directly guide multi-round refinement of the merge. Given the pre-trained parameter θp\theta_p and its original fine-tuned value θi,p0\theta_{i,p}^0, we rank parameters p=(r,c)p=(r,c) by Si,p=(θp−θi,p0)2[Xi0(Xi0)⊤]ccS_{i,p}=(\theta_p-\theta_{i,p}^0)^2[\boldsymbol{X}_i^0(\boldsymbol{X}_i^0)^\top]_{cc}. For each ρ\rho, we independently reset the lowest-scoring fraction of parameters in the original fine-tuned models and merge once. Accuracy drops from 82.74% to 72.07% at ρ=0.9\rho=0.9, versus 84.00% for OBMerge (Table 2).

Table 2. Replacement with pre-trained values on ViT-B/32: average test accuracy (%).

Replacement ratio ρ\rhoViT-B/32
0 (RegMean)82.74
0.182.73
0.282.69
0.382.61
0.482.43
0.582.07
0.681.43
0.780.18
0.877.47
0.972.07

We add an ablation of the replacement score by comparing random, distance, curvature, and LS2 under the separate local reconstruction objective. LS2 outperforms random replacement on all five models and performs best on four.

We also examine coefficient sensitivity and computational cost. Full-test coefficient sweeps on ViT-B/32 and Qwen3-0.6B confirm that the RegMean++ coefficients used in the manuscript are best within the reported grid. End-to-end profiling on ViT-B/32 shows that ten rounds take 1.44× and 3.73× the single-merge time for RegMean and RegMean++, respectively.

To assess robustness and scalability, we run experiments with three calibration seeds on four models and evaluate Qwen3-8B. OBMerge improves on RegMean for all four models at every seed. On Qwen3-8B, it achieves the highest average accuracy among all compared methods at 91.12%, surpassing RegMean (90.05%) and the strongest other baseline, ACTMat (90.36%).

Further details and point-by-point responses are provided in our replies to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.