- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how the performance advantage of a TSV-merged initialization over a pretrained CLIP model changes with additional training data and optimizer updates. Experiments on eight classification tasks show that increasing data reduces this gap more than increasing updates on ViT-B/32, while the merged initialization retains an advantage within the evaluated budgets.
Critical strengths
-
The matched-adaptation design, paired seeds, and shared endpoint provide a controlled empirical comparison of initialization advantages under different resource budgets.
-
The supervised and magnitude-matched controls help establish that the observed gains are not solely attributable to the distillation objective or update magnitude.
Critical weaknesses
-
The main findings overlap with existing research on model merging, multi-task learning, and initialization advantages, making the distinct contribution unclear.
-
The central data-versus-training observation may largely reflect general learning behavior rather than a property specific to model merging.
Questions for authors
-
What is the fundamental contribution beyond existing research? Prior work has already investigated the relationship between model merging and multi-task learning, including the possibility that jointly trained models outperform merged models. For example, EMR-Merging (NeurIPS 2024, Table 2) reports 88.9% accuracy for traditional MTL on the eight-task ViT-B/32 benchmark, exceeding the 87.32% achieved by the adapted TSV model in this paper, although the training protocols differ. It seems that the relationship between model merging and continued training from the base model has already been explored. What fundamentally new insights does this paper provide?
-
Is the central observation genuinely specific to model merging? The finding that additional training data can provide greater benefits than repeated optimization on a fixed dataset is broadly consistent with established empirical understanding. Although this paper measures the reduction in the initialization gap rather than absolute accuracy improvements, it remains unclear what merging-specific insight follows from this observation.
-
The primary experiments focus on TSV merging, two CLIP backbones, and eight classification tasks. Do the conclusions generalize to other model architectures, such as CNN or LLM?