OBMerge: Optimal Brain Merge via Iterative Parameter Replacement

Model merging combines task-specific fine-tuned models without joint training. Replacing some fine-tuned parameters with merged values can reduce conflicts between tasks, but deciding which parameters to replace requires estimating the loss change caused by each replacement. We introduce Optimal Brain Merge (OBMerge), an iterative model-merging framework based on parameter replacement. OBMerge adapts Optimal Brain Damage to model merging, using a Taylor expansion to estimate the loss change from replacing a fine-tuned parameter with its corresponding merged value, rather than setting it to zero as in pruning. Starting from an initial merge, OBMerge follows a score & replace $\to$ re-merge loop. At each iteration, we replace low-scoring parameters with their corresponding merged values while retaining high-scoring task-specific parameters, then re-merge the modified models. We also compare replacement scores across objectives and task scopes. We further introduce a pipeline in which an LLM generates calibration data using task and dataset descriptions together with instruction templates. We use the resulting synthetic data for replacement scoring, merging, and selecting merging coefficients. Experiments on two vision and three language models show that OBMerge achieves the highest average accuracy among merging methods on four of the five models, and that synthetic calibration data can give results comparable to real calibration data. For example, on seven commonsense reasoning tasks with Llama-3.2-1B, OBMerge achieves $84.30\%$, compared with $81.13\%$ for RegMean and $81.40\%$ for RegMean with synthetic calibration data.