acceptodds
Withdrawn from ICLR 2027

What Is a Merge Worth? Data and Training Budgets in Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging can produce a checkpoint for immediate deployment or an initialization for further adaptation. These uses raise different questions: a merged model's deployment advantage may change when the pretrained base is also allowed to train, and single-budget evaluations cannot separate the value of initialization from access to more data or optimization. We compare fixed TSV and pretrained CLIP initializations across eight classification tasks, using matched adaptation and paired seeds to measure changes in their performance gap along data and update axes that share a common endpoint. On ViT-B/32, increasing the fitting pool reduces the gap more than extending training on the full pool; on ViT-B/16, the data contrast points in the same direction but remains uncertain. TSV retains an advantage at the shared endpoint, while supervised and magnitude controls show that the result is not limited to distillation or explained by update norm alone. The findings suggest that practitioners compare merged and pretrained starts across adaptation budgets and interpret merge value within the evaluated access regime, rather than as a universal trade-off or end-to-end saving.

Primary area:
interpretability and explainable AI
Keywords:
Model Merging, Vision Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sXowOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the performance advantage of a TSV-merged initialization over a pretrained CLIP model changes with additional training data and optimizer updates. Experiments on eight classification tasks show that increasing data reduces this gap more than increasing updates on ViT-B/32, while the merged initialization retains an advantage within the evaluated budgets.

Critical strengths
  1. The matched-adaptation design, paired seeds, and shared endpoint provide a controlled empirical comparison of initialization advantages under different resource budgets.

  2. The supervised and magnitude-matched controls help establish that the observed gains are not solely attributable to the distillation objective or update magnitude.

Critical weaknesses
  1. The main findings overlap with existing research on model merging, multi-task learning, and initialization advantages, making the distinct contribution unclear.

  2. The central data-versus-training observation may largely reflect general learning behavior rather than a property specific to model merging.

Questions for authors
  1. What is the fundamental contribution beyond existing research? Prior work has already investigated the relationship between model merging and multi-task learning, including the possibility that jointly trained models outperform merged models. For example, EMR-Merging (NeurIPS 2024, Table 2) reports 88.9% accuracy for traditional MTL on the eight-task ViT-B/32 benchmark, exceeding the 87.32% achieved by the adapted TSV model in this paper, although the training protocols differ. It seems that the relationship between model merging and continued training from the base model has already been explored. What fundamentally new insights does this paper provide?

  2. Is the central observation genuinely specific to model merging? The finding that additional training data can provide greater benefits than repeated optimization on a fixed dataset is broadly consistent with established empirical understanding. Although this paper measures the reduction in the initialization gap rather than absolute accuracy improvements, it remains unclear what merging-specific insight follows from this observation.

  3. The primary experiments focus on TSV merging, two CLIP backbones, and eight classification tasks. Do the conclusions generalize to other model architectures, such as CNN or LLM?

Reviewer_Kj7uOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper experimentally investigates whether the advantage that the merged weights obtained from model merging (e.g, TSV) provide over base weights in a setting where the initialized model learns a task. Through controlled experiments combining TSV and ViT, the paper finds that the gains obtained from the merged weights are strongly dependent on the amount of data.

Critical strengths
  • Experiments are being conducted in which the merge weights and pre-training weights generated by TSV are controlled based on two factors: data volume and the number of optimization steps.
Critical weaknesses
  • The paper fails to clearly define its motivation and scope. The introduction does not cite any existing literature, resulting in a presentation that relies heavily on the reader’s prior knowledge. Furthermore, the writing is extremely vague and of poor quality. For example, it is likely that readers will find it difficult to understand the motivation for using model merging techniques, which were designed to integrate knowledge from multiple models without training, for retraining within the same domain.
  • The problem setup is unrealistic. The dataset used for analysis is not a new holdout dataset, but rather a known dataset that has already been used to train an expert model. In other words, it is possible to train the base model using methods such as multi-task learning, which deviates from the standard setup for model merging, which assumes scenarios where additional training or access to the model is not possible.
  • The paper ignores the training cost of specialized models. This stems from the peculiarity of the problem formulation discussed above. Considering the training cost of task-specific models, it would also be possible to train a base model for multi-task learning at the same cost. However, the paper does not conduct an evaluation based on such a precise alignment of computational budgets.
  • The validity of the computational budget constraint based on the number of optimization steps is weak. Although the paper controls the number of optimizer updates as a computational budget, this may cause overfitting when using a small data pool, thereby weakening the validity of the overall evaluation. In fact, a clear tendency toward overfitting can be observed in Table 4. This may undermine the validity of the paper’s conclusion that the benefits of model merging are driven more by the amount of data than by the number of optimization steps.
  • The conclusion that the benefit of model merging stems from the amount of data may also depend on the model size. While the experiments with ViT-B/32 do show a tendency for an increase in data to reduce the advantage of model merging, the variance between seeds is large in the ViT-B/16 experiments, and there is no statistical significance.
Questions for authors
  • What is the motivation behind evaluating a setting where the merged weights serve as an initialization for further adaptation on the exact same dataset used to train the specialist models?
  • Why is the number of optimization updates included as an independent evaluation axis despite the observed risk of overfitting?
  • Did you compare the merged initialization against a multi-task learning baseline under an equivalent total computational budget that accounts for the training costs of the specialist models?
Reviewer_85dDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The work titled “WHAT IS A MERGE WORTH? DATA AND TRAINING BUDGETS IN MODEL MERGING”, explores the advantage that a merged checkpoint offers for continued training compared to starting from scratch from the pre-trained checkpoint. Specifically the authors look at a budgeted setting, limited amount of data/compute for the training process. The work evaluates across the popular 8 task vision setting and observes that the head start obtained from merging continues to be advantageous even after continued training with a limited budget.

Critical strengths

[1] Studying if the advantage of starting from a merged checkpoint rather than starting from a pre-trained model is interesting.

Critical weaknesses

[1] The work focuses on a limited data/compute budget setting for continued learning from a merged model versus the pre-trained model; this is not very interesting, as it is expected that the head start gained from merging would continue over a pre-trained model, especially when evaluated on a very small compute budget.

[2] Except for the related works section, the authors fail to cite any references to prior work throughout the paper.

[3] The evaluation is performed across a single 8-task vision setting with very small model sizes, which makes the analysis less interesting.

Questions for authors

[1] Can the authors analyze without a tight limitation on the compute budget?

[2] Can the authors analyze across models of different sizes and across different modalities?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.