acceptodds
Withdrawn from ICLR 2027

Never Overshoot: Expert-Anchored Gradient Descent for Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging combines task-specific experts fine-tuned from a common pretrained model into a single model, which often underperforms the experts on their respective tasks. We study refinement using a few labeled examples per task, already available for selecting or fitting the merge. We introduce expert-anchored gradient descent (XGD), which uses the original expert checkpoints to guide updates from these small buffers. XGD scales each coordinate's learning rate by its distance to the task expert and permits only movements toward that expert, without passing it. Replay gradients select locally favorable movements, and expert offsets are recomputed as refinement proceeds. On GLUE-8, CLIP-8 and CLIP-20 with 8 or 16 labels per task, both XGD and AdamW improve every evaluated merge in mean task performance, reusing the examples already used to select or fit it. Thus, few-shot refinement offers gains without additional labels. Across five buffer draws, XGD has the best mean among the compared refinements in five of six benchmark–budget settings. Starting directly from the pretrained model, XGD also outperforms many merge baselines, combining expert information without a separate merging algorithm. From the same merges at eight labels per task, XGD retains higher mean ImageNet accuracy than AdamW in every comparison and lower mean WikiText-2 masked-LM loss in all but one.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Model Merging, Gradient Descent

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pV3tOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to refine a merged model using a few labeled examples per task. It computes few-shot replay gradients via a backward pass and, rather than applying these gradients directly, uses them solely to select and scale updates along the corresponding task vector (the expert checkpoint difference). The key insight is the task-specific gradient identifies favorable coordinates for update, while the task vector constrains the update so that the model cannot move away from or beyond the expert. Experiments on GLUE-8, CLIP-8, and CLIP-20 show improved results over both standard merging baselines and AdamW fine-tuning.

Critical strengths
  • The paper provides empirical experiments and releases source code, facilitating reproducibility.
  • The key insight is the task-specific gradient identifies favorable coordinates for update, while the task vector constrains the update so that the model cannot move away from or beyond the expert.
Critical weaknesses

Major Concern

  • The writing is poor and falls well below the standard expected of a venue submission. Several sentences are difficult to read, e.g., "Replay gradients select locally favorable movements" and "AdamW improve every evaluated merge in mean task performance". These phrase read as stilted noun strings rather than well-formed academic prose.

  • A major concern is that the proposed method requires repeated task-wise forward and backward computations to compute replay gradients. In contrast, gradient-free merging algorithms such as TIES-Merging and Task Arithmetic operate purely on parameter-space arithmetic and incur no such cost. The authors should report end-to-end wall-clock time, peak memory usage, and the total selection-plus-refit cost against similar training-required methods like Twin Merging, CAT Merging and LOT Merging.

  • XGD depends on a small set of labeled exemplars to compute replay gradients. If these samples contain label noise or exhibit distributional shift, the resulting gradient can incorrectly admit or reject coordinates, or distort their step sizes, even though the expert constraint may reject some erroneous directions. The authors should report selection stability and provide controlled label-noise, input-corruption, or replay-distribution-shift experiments.

  • The related work section is notably incomplete, covering only a narrow subset of publications from 2023–2025. Rather than conducting a thorough, first-hand review of the broader model merging landscape, the authors appear to rely heavily on citing a single survey to represent prior work.

  • The evaluation is restricted to relatively small-scale models: RoBERTa-base on GLUE-8 for NLP and CLIP ViT-B/32 for image classification. The paper omits testing on modern large language models (e.g., Qwen, LLaMA) and more complex reasoning or knowledge-intensive benchmarks (e.g., GSM8K, MMLU).


Other Concern

The following concerns pertain to experimental design, implementation details, and reproducibility, as identified through AI-assisted analysis.

1. The reported common four-fold selection protocol does not match the implementation

Section 4 and Appendix B.1 state that hyperparameters, including those of the initial merges, are selected by averaging held-out task metrics across four fold rotations. However, scripts/cv_protocol.py selects checkpoint-based merging candidates using scores computed on the full label budget. Fisher and TATR also use full-budget selection, while aTLAS is fitted on the full labeled budget with fixed settings. Furthermore, ls_select_and_refit() selects Localize-and-Stitch settings using only fold 0 as validation, without rotating the folds, before refitting on the complete budget. Its subsequent refinement uses four folds, but its initialization selection does not.These procedures should not be described as a common four-fold protocol. Full-budget scoring and averaging fold metrics are not generally equivalent for GLUE’s nonlinear metrics, including MCC, F1, and correlations. The authors should accurately disclose the method-specific procedures or rerun the affected comparisons under the stated protocol, and assess whether the rankings change.

2. Appendix E is inconsistent with the main experiments

The main text reports five buffer draws, seeds 103–107, and four-fold selection at both label budgets. Appendix E instead reports three draws, seeds 103–105, and single-split selection at 16-shot, while claiming that its per-task tables expand the main results. For example, the CLIP-8 pretrained-start XGD task scores in Table 8 average to approximately 0.6646, whereas Table 1 reports 0.643. At 16-shot, Table 9 averages to approximately 0.6809, whereas Table 1 reports 0.703. The released files explain the 8-shot discrepancy: the appendix uses the first three draws rather than all five. The older 16-shot appendix cannot be reconstructed from the shipped four-fold results. The authors should regenerate the appendix from the exact states used for the main tables and identify the provenance of the superseded results.

3. Merge-start refinement validation is conditional on an initialization already adapted using all labels

The initialization is selected or fitted using the full label budget and then reused by every refinement fold. Although each fold starts from a fresh parameter clone and fresh optimizer moments, its held-out labels may already have influenced that initialization. Consequently, the refinement folds are held out from the additional refinement gradients, but not independently held out from the complete adaptation pipeline. This is particularly clear for aTLAS, whose initialization is supervised-fitted on all budget examples.

Questions for authors

See Weakness

Reviewer_XXQUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes XGD, which fine-tunes merged models using a small number of labels. It adjusts coordinate-wise learning rates based on the distance to experts, allows updates only toward experts, and constrains step sizes to avoid overshooting expert values. Instead of learning separate merging coefficients, it directly updates the weights of the shared encoder and can also start from a pretrained model. It evaluates performance and preservation of pretraining capability using 8 or 16 labels per task on GLUE-8, CLIP-8, and CLIP-20.

Critical strengths
  1. The comparison with AdamW under the same label budget and number of training steps is appropriate.

  2. XGD achieves additional gains across various merged initializations.

Critical weaknesses
  1. Unlike ProDistill [1] and Expert Merging [2], which learn task-vector merging coefficients using expert information, XGD directly updates the shared encoder weights via gradient-based optimization under expert-derived constraints. While this is a valid post-optimization strategy, the paper does not sufficiently justify why this formulation offers a distinctive contribution to model merging, rather than simply serving as another constrained fine-tuning procedure

  2. Much of the reported performance recovery appears attributable to additional optimization steps rather than XGD's specific design. For instance, on CLIP-8 with 16 labels, TIES improves from 73.3% to 77.3% with XGD, whereas standard AdamW already achieves 77.2%. Similarly, for aTLAS, XGD and AdamW yield nearly identical performance at 74.4% and 74.3%, respectively (Tables 1 and 2). These marginal differences raise questions about the practical utility of the proposed optimizer, especially on stronger initializations

  3. Strong merged models and supporting experimental evidence are lacking. It is not applied to TSV-M [3] and Iso-C/Iso-CTS [4], which report strong performance on 8 and 20 tasks using CLIP ViT-B/32, it cannot be verified whether additional gains from XGD remain on stronger initializations.

  4. The proposed method adds post-merging learning rate selection and a 50-sweep optimization. Since runtime and memory consumption are not reported, it is difficult to judge whether minor accuracy gains justify these added costs (§4, Appendix B).

[1] Scalable Model Merging with Progressive Layer-wise Distillation, ICML 2025.

[2] Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking, ICLR 2026.

[3] Task Singular Vectors: Reducing Task Interference in Model Merging, CVPR 2025.

[4] No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces, ICML 2025.

Questions for authors
  1. How do results change if the step-size upper bound is removed while maintaining direction restriction and distance scaling?

  2. Do additional gains remain after constructing TSV-Merge or Iso-C/Iso-CTS using identical checkpoints and label budgets?

Reviewer_1XFfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents XGD (Expert-Anchored Gradient Descent) to improve the performance of merged models using small labelled datasets, available for configuring the merge. Since the few labelled samples can generate noisy updates, XGD anchors the updates w.r.t to fine-tuned models - (1) by permitting movements only towards the expert (gating) and (2) by not exceeding it (capping). Evaluations on GLUE-8, CLIP-8 and CLIP-20 benchmarks show that XGD improves the performance of the merged model across various choices of merging algorithms, compared to AdamW, in majority of the cases. Experiments also demonstrate that XGD is also able to better retain pre-trained capabilities compared to AdamW.

Critical strengths

The formulation of XGD using task-experts to guide noisy gradients is simple and sound. The insight that the data used for configuring the merge can be reused for improving the model through gradient updates is noteworthy. The focus of the paper on the few-shot setting also addresses a real-world use case where such availability of such samples could be scarce.

Critical weaknesses

My main critic of the paper is that it evaluates only very small models, CLIP ViT-B/32 (151M parameters) and RoBERTa-base (125M parameters). Thus the empirical evidence for XGD's benefits is fairly weak, and not demonstrated at scale.

My second main critic of the paper is that it does not compare against other data-based approaches, such as AdaMerge [1] or Surgery [2], which do not even require labels. Since the budget of 8 or 16 examples can be used in any possible way to the improve the merged model's performance, the paper should compare against a larger set of these contemporary data-based approaches.

References

[1] Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X. and Tao, D., 2024, May. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations (Vol. 2024, pp. 22743-22763).

[2] Yang, E., Shen, L., Wang, Z., Guo, G., Chen, X., Wang, X. and Tao, D. (2024) Representation surgery for multi-task model merging, Proceedings of the 41st International Conference on Machine Learning, 235, pp. 56332–56356.

Questions for authors
  1. Why are the small labelled datasets referred to as buffers?
  2. Could the authors include average expert performance as Oracle in Tables 1 and 2 for better effective comparison? How much of the gap w.r.t Oracle still remains after XGD?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.