- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes to refine a merged model using a few labeled examples per task. It computes few-shot replay gradients via a backward pass and, rather than applying these gradients directly, uses them solely to select and scale updates along the corresponding task vector (the expert checkpoint difference). The key insight is the task-specific gradient identifies favorable coordinates for update, while the task vector constrains the update so that the model cannot move away from or beyond the expert. Experiments on GLUE-8, CLIP-8, and CLIP-20 show improved results over both standard merging baselines and AdamW fine-tuning.
Critical strengths
- The paper provides empirical experiments and releases source code, facilitating reproducibility.
- The key insight is the task-specific gradient identifies favorable coordinates for update, while the task vector constrains the update so that the model cannot move away from or beyond the expert.
Critical weaknesses
Major Concern
-
The writing is poor and falls well below the standard expected of a venue submission. Several sentences are difficult to read, e.g., "Replay gradients select locally favorable movements" and "AdamW improve every evaluated merge in mean task performance". These phrase read as stilted noun strings rather than well-formed academic prose.
-
A major concern is that the proposed method requires repeated task-wise forward and backward computations to compute replay gradients. In contrast, gradient-free merging algorithms such as TIES-Merging and Task Arithmetic operate purely on parameter-space arithmetic and incur no such cost. The authors should report end-to-end wall-clock time, peak memory usage, and the total selection-plus-refit cost against similar training-required methods like Twin Merging, CAT Merging and LOT Merging.
-
XGD depends on a small set of labeled exemplars to compute replay gradients. If these samples contain label noise or exhibit distributional shift, the resulting gradient can incorrectly admit or reject coordinates, or distort their step sizes, even though the expert constraint may reject some erroneous directions. The authors should report selection stability and provide controlled label-noise, input-corruption, or replay-distribution-shift experiments.
-
The related work section is notably incomplete, covering only a narrow subset of publications from 2023–2025. Rather than conducting a thorough, first-hand review of the broader model merging landscape, the authors appear to rely heavily on citing a single survey to represent prior work.
-
The evaluation is restricted to relatively small-scale models: RoBERTa-base on GLUE-8 for NLP and CLIP ViT-B/32 for image classification. The paper omits testing on modern large language models (e.g., Qwen, LLaMA) and more complex reasoning or knowledge-intensive benchmarks (e.g., GSM8K, MMLU).
Other Concern
The following concerns pertain to experimental design, implementation details, and reproducibility, as identified through AI-assisted analysis.
1. The reported common four-fold selection protocol does not match the implementation
Section 4 and Appendix B.1 state that hyperparameters, including those of the initial merges, are selected by averaging held-out task metrics across four fold rotations. However, scripts/cv_protocol.py selects checkpoint-based merging candidates using scores computed on the full label budget. Fisher and TATR also use full-budget selection, while aTLAS is fitted on the full labeled budget with fixed settings. Furthermore, ls_select_and_refit() selects Localize-and-Stitch settings using only fold 0 as validation, without rotating the folds, before refitting on the complete budget. Its subsequent refinement uses four folds, but its initialization selection does not.These procedures should not be described as a common four-fold protocol. Full-budget scoring and averaging fold metrics are not generally equivalent for GLUE’s nonlinear metrics, including MCC, F1, and correlations. The authors should accurately disclose the method-specific procedures or rerun the affected comparisons under the stated protocol, and assess whether the rankings change.
2. Appendix E is inconsistent with the main experiments
The main text reports five buffer draws, seeds 103–107, and four-fold selection at both label budgets. Appendix E instead reports three draws, seeds 103–105, and single-split selection at 16-shot, while claiming that its per-task tables expand the main results. For example, the CLIP-8 pretrained-start XGD task scores in Table 8 average to approximately 0.6646, whereas Table 1 reports 0.643. At 16-shot, Table 9 averages to approximately 0.6809, whereas Table 1 reports 0.703. The released files explain the 8-shot discrepancy: the appendix uses the first three draws rather than all five. The older 16-shot appendix cannot be reconstructed from the shipped four-fold results. The authors should regenerate the appendix from the exact states used for the main tables and identify the provenance of the superseded results.
3. Merge-start refinement validation is conditional on an initialization already adapted using all labels
The initialization is selected or fitted using the full label budget and then reused by every refinement fold. Although each fold starts from a fresh parameter clone and fresh optimizer moments, its held-out labels may already have influenced that initialization. Consequently, the refinement folds are held out from the additional refinement gradients, but not independently held out from the complete adaptation pipeline. This is particularly clear for aTLAS, whose initialization is supervised-fitted on all budget examples.
Questions for authors
See Weakness