- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OBMerge, an iterative model merging framework that adapts Optimal Brain Damage (OBD) from pruning to model merging, using Taylor expansion to estimate loss changes and decide which fine-tuned parameters should be replaced with corresponding merged values. The method follows a score-and-replace then re-merge loop, built primarily on top of RegMean and RegMean++ as base merging operators. The paper also derives and compares multiple replacement score variants under different loss functions and task scopes, and proposes an LLM-based pipeline to generate synthetic calibration data as a substitute for real data. Experiments across vision and language models show consistent accuracy improvements over baseline merging methods.
Critical strengths
The synthetic calibration data pipeline is well-motivated and addresses a meaningful practical bottleneck. Data-dependent merging methods typically require real calibration examples and incur storage costs for precomputed Gram matrices; this work shows that LLM-generated synthetic data from task descriptions and instruction templates can deliver comparable performance, which extends the applicability of merging methods to low-data or data-restricted scenarios.
The paper is clearly written and well-structured. The framework derivation, scoring variants, experimental setup, and ablation results are presented in a logical, easy-to-follow manner, with sufficient technical details and supplementary materials to support reproducibility.
Critical weaknesses
The technical contribution is largely incremental rather than conceptually novel. Adapting the OBD criterion from "setting parameters to zero" to "replacing parameters with merged values" is a natural and straightforward variant of the original pruning idea, and the overall framework is essentially a combination of existing components (OBD-style importance scoring + RegMean merging + iterative refinement) without fundamentally new insights into the mechanism of model merging. This limits the conceptual novelty and broader impact of the work.
The method is tightly coupled to reconstruction-based merging operators and lacks generalizability. As shown in the appendix, OBMerge delivers marginal gains with ACTMat/TSV-M and causes severe performance collapse with Iso-C/Iso-CTS in later iterations. This narrow compatibility undermines the framing of OBMerge as a general-purpose merging framework, and restricts its practical utility to a small subset of merging methods.
Questions for authors
The iterative replacement schedule and ratio choices currently lack principled guidance. The paper evaluates only a small set of linear schedules and selects the best one empirically, but provides no theory or heuristic for how to set replacement ratios or determine stopping criteria for unseen tasks and model architectures. How should practitioners configure these hyperparameters without exhaustive tuning?
It is unclear whether the computational overhead of multiple re-merging rounds is justified compared to a well-tuned single-pass baseline. Could you provide a direct comparison of computational cost (wall-clock time, memory usage) between OBMerge and tuned RegMean++, and quantify the performance gain per additional iteration round?
Have you investigated potential modifications to improve OBMerge’s compatibility with non-reconstruction-based merging operators? For example, would adjusting the scoring function, replacement schedule, or re-merging logic prevent the performance collapse observed with Iso-C and Iso-CTS?
3 comments
Response continued
W2, Q1, and Q3: Schedules and operator compatibility.
In Figure 2 of the manuscript, we evaluate six replacement schedules with different starting ratios and numbers of iterations. All six schedules, including the single-iteration schedule, improve on RegMean across the three language models. These results demonstrate that OBMerge works well across multiple schedules, so a simple schedule can yield good results. We leave optimal scheduling to future work.
The accuracy drop in later iterations with Iso-C/Iso-CTS might occur because re-merging adds the same task-vector updates multiple times. As more parameters are replaced, each modified fine-tuned model contains more values from the previous merge. Iso-C sums the resulting task vectors, while Iso-CTS constructs its common subspace from their sum, potentially amplifying these updates at each iteration. RegMean minimizes a closely related reconstruction objective, which may explain its more stable behavior.
To avoid performance deterioration, we select a model from 10 iterations using 100 COCO val2017 (Lin et al., 2014) image–caption pairs for vision and 100 fixed self-generated validation examples for each CSR task. We show the results in Table 20 (Appendix C.4). For example, on Qwen3-0.6B, we select iteration 6 for Iso-CTS with OBMerge and obtain 83.05% accuracy. Extending our proposed self-generated calibration method to generate task-specific synthetic images may improve model selection; we leave this to future work.
Table 20. Average test accuracy (%) on vision and CSR; parentheses indicate selected iterations.
| Method | ViT-B/32 (Vision) | ViT-L/14 (Vision) | Qwen3-0.6B (CSR) | Llama-3.2-1B (CSR) |
|---|---|---|---|---|
| Iso-C | 80.47 | 90.76 | 81.19 | 79.07 |
| w/ OBMerge (LS2) | 81.69 (2) | 91.14 (2) | 83.24 (6) | 79.30 (3) |
| Iso-CTS | 78.90 | 90.43 | 79.17 | 76.54 |
| w/ OBMerge (LS2) | 81.82 (3) | 91.47 (3) | 83.05 (6) | 78.36 (4) |
Q2: Merging cost.
We report end-to-end merging time and peak memory usage on ViT-B/32 in Table 21. End-to-end time includes initialization, Gram collection from 512 images/task, all merging rounds, and checkpoint saving. Ten rounds of OBMerge take 1.44× and 3.73× the time of a single merge for RegMean and RegMean++, respectively.
Figure 2 and Table 12 in the manuscript report per-iteration performance on language and vision models, respectively. For example, RegMean with OBMerge (LS2) on ViT-B/32 reaches 83.20%, 83.68%, 83.83%, and 83.95% at iterations 6–9, corresponding to gains of 0.48, 0.15, and 0.12 percentage points per additional iteration.
Table 21. ViT-B/32 merging cost: end-to-end time (s), peak memory (GiB), and average vision test accuracy (%). Times are means over two A40 runs; memory is the maximum across runs.
| Operator | Total rounds | End-to-end (s) | Peak memory (GiB) | Acc. (%) |
|---|---|---|---|---|
| RegMean | 1 | 83.52 | 14.84 | 82.74 |
| RegMean | 10 | 119.90 | 17.56 | 84.01 |
| RegMean++ | 1 | 106.90 | 21.39 | 84.55 |
| RegMean++ | 10 | 398.48 | 24.47 | 85.31 |
Kind reminder
Thanks again for your helpful feedback. We have added experiments and clarifications addressing your concerns, including replacement-target ablations and comparisons with OBIM and SA-Merging, validation-based iteration selection for Iso-C/Iso-CTS, and merging-cost measurements.
We would appreciate it if you could take another look at our rebuttal and updated manuscript and reconsider your assessment in light of these results. We would be happy to clarify any remaining questions.
Thank you again for your time!
Author response to Reviewer 7rZu
Thank you for your comments.
W1: Contribution and replacement target.
To our knowledge, OBMerge is the first to use estimated objective changes to guide replacement with merged values. We also introduce an LLM-based pipeline to generate calibration data for scoring, merging, and coefficient selection. The fundamental difference is the replacement target: OBMerge replaces fine-tuned parameters with values from the previous merge, while OBIM and SA-Merging zero task-vector entries, restoring pre-trained parameter values.
OBIM uses a layer-wise MSE between pre-trained and fine-tuned outputs, with both models fixed. OBMerge instead scores replacement with values from the previous merge, using global task loss or local reconstruction loss under total or separate task scopes (Table 1 in the manuscript).
We evaluate OBIM's replacement with pre-trained values on ViT-B/32, holding RegMean, the fine-tuned models, calibration data, and layer-wise selection fixed. Note that OBIM's objective does not involve the evolving merged model, so it cannot directly guide multi-round refinement of the merge. Given the pre-trained parameter θp and its original fine-tuned value θi,p0, we rank parameters p=(r,c) by Si,p=(θp−θi,p0)2[Xi0(Xi0)⊤]cc. For each ρ, we independently reset the lowest-scoring fraction of parameters in the original fine-tuned models and merge once. Accuracy drops from 82.74% to 72.07% at ρ=0.9, versus 84.00% for OBMerge (Table 18).
Table 18. Replacement with pre-trained values on ViT-B/32: average test accuracy (%).
SA-Merging multiplies connectivity-score gradients by the aggregate task vector and prunes low-scoring entries at a fixed per-round ratio. However, each round's merged model is obtained from the sum of the cumulatively pruned task vectors, whereas OBMerge obtains its merged model by re-merging fine-tuned models after replacing selected parameters with values from the previous merge. At each round, OBMerge starts from the original fine-tuned models and uses Taylor-based replacement scores to select an increasing fraction of low-scoring parameters for replacement.
We implemented SA-Merging following Algorithm 1 in the SA-Merging paper, as official code was not publicly available to us. We use SynFlow's reference code to compute its connectivity score (SynFlow reference implementation).
To compute each fine-tuned model's connectivity score, we run a full-model forward pass with continuous embeddings filled with ones and absolute weights, then sum the outputs (CLIP pooled features or LM logits). We use 50 tokens for ViT-B/32 and 257 for ViT-L/14, including the CLS token, and 256 tokens for CSR. We prune at p=0.2 per round for T=10 rounds, then add the final task-vector sum to the pre-trained parameters.
Table 19 and Appendix C.5 show that OBMerge outperforms both OBIM and SA-Merging on all four models. For example, on ViT-B/32, OBMerge reaches 84.00%, compared with 72.68% for OBIM and 39.44% for SA-Merging. On Llama-3.2-1B, OBMerge reaches 84.70%, compared with 61.41% for OBIM and 71.50% for SA-Merging.
Table 19. Merging results: average accuracy (%) on vision and CSR. SA-Merging uses our SynFlow-based implementation. Bold marks the best result per model.