acceptodds
Withdrawn from ICLR 2027

A Controlled Benchmark for Weight-Space Model Merging Across Medical Imaging Domains

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Consolidating medical-imaging specialists into a shared model requires preserving performance across distinct imaging domains. Weight-space model merging offers a way to combine independently fine-tuned encoders without retraining, but comparisons between merging methods are sensitive to validation protocols. Official evaluation scripts for TSV-Merge and isotropic merging use wider scaling-coefficient ranges for selected spectral methods than for other tunable baselines. We introduce a controlled benchmark spanning dermoscopy, chest radiography, and histopathology, with three binary tasks and training images per specialist. We compare 11 methods across seven vision-transformer backbones and three seeds using a shared scaling-coefficient grid. The best observed merger reaches mean AUROC against for task-aware specialist routing, with tuned Task Arithmetic close behind at . We also introduce Gram-LS, a closed-form method based on task-vector projection constraints that performs comparably to tuned summation. Within-method validation tuning improves Gram-LS and Task Arithmetic by approximately – AUROC, exceeding the spread among five leading tuned methods, while search bounds can change method rankings. Iso-C and Iso-CTS fall – AUROC below tuned summation. These findings establish tuned summation as a strong baseline in this setting and show that specialist references and validation protocols are essential to assessing the performance cost of consolidation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, task arithmetic, task vectors, medical imaging, vision transformers, evaluation protocols, model consolidation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Nw6YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates eleven weight-space merging methods across three binary medical-imaging tasks, seven pretrained backbones, and three seeds. It quantifies the performance cost of consolidating specialists, studies validation-search sensitivity, and introduces a projection-based Gram-LS rule. Its principal contribution is empirical: tuned summation is competitive and validation choices can matter more than differences among leading merging methods.

Critical strengths
  1. A useful empirical question and informative negative results. The study measures storage consolidation against retaining task-specific specialists, with a common task formulation and multi-backbone evaluation. A strong tuned-summation baseline and the acknowledged lack of a Gram-LS advantage are useful findings; a benchmark need not introduce a winning algorithm to contribute.

  2. The within-method tuning evidence and detailed result records add practical value. Independent aggregation of the supplied main-method records agrees with the reported means. These findings should be recognized separately from the method-ranking claims affected by the concerns below.

Critical weaknesses
  1. Fidelity of central baseline comparisons remains unresolved. The supplied iso_c.py and iso_cts.py paths sum non-matrix updates rather than averaging them as specified in the reference Iso method. Iso-CTS also discards task-residual singular values and derives its isotropic scale from the summed-update spectrum, differing from the reference algorithm. The LiNeS path defaults to a plural layers regex while the ViT backend defines singular layer, without forwarding the backend pattern in the inspected constructors. These differences could affect the large Iso deficit and the interpretation of LiNeS ranking changes. Their actual experimental impact is not established because the executed source version is not bound to the records.

  2. Actual grouped-split provenance needs clarification. The inspected CheXpert/ISIC loaders do not provide patient/lesion group keys to uncached split generation, which falls back to image-level splitting. Cached splits or preprocessing could resolve the discrepancy with the paper. This is not evidence of observed leakage, but it matters to what the benchmark measures.

These concerns affect central empirical conclusions rather than peripheral implementation details. They do not automatically invalidate the separate finding that tuning improves simple summation.

Questions for authors
  1. Which exact implementation generated the reported cells? Can reference-output checks for Iso and effective layer scales for LiNeS establish fidelity, or can the affected key comparisons be updated if the submitted paths were executed? I do not assume that correction will reverse the ranking.
  2. Can the actual split records confirm the patient/lesion separation described in the paper? Existing split evidence is sufficient if the submitted loader path was not used.

I currently recommend weak rejection, with substantial room to revise this assessment if these two issues are resolved. The concern is reliability of the benchmark conclusions, not the absence of a new state-of-the-art algorithm or a complete geometric explanation.

Reviewer_M9biOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper benchmarks existing weight-space model merging methods across medical imaging domains and introduces a Gram-matrix-based merging approach. While the comparisons are reasonably systematic, the methodological contribution is limited, and the experimental findings offer little new insight beyond the importance of hyperparameter tuning.

Critical strengths

The paper provides a relatively systematic comparison of existing merging methods across multiple backbones. The analysis of hyperparameter sensitivity is useful for understanding how evaluation protocols affect method rankings.

Critical weaknesses

The novelty and significance of this work are limited. The main finding that hyperparameter tuning can substantially affect model merging performance is not particularly surprising. The proposed Gram-LS method is also a relatively straightforward modification of task-vector merging and achieves essentially identical performance to the much simpler Task Arithmetic baseline (both 0.808 AUROC).

The benchmark itself is also rather limited. Despite evaluating seven backbones, the main experiments involve only three binary classification tasks, each restricted to 2,000 training images. This highly simplified setting does not adequately reflect realistic model merging scenarios.

Questions for authors

None

Reviewer_vDuZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper benchmarks medical model merging methods and introduces a Gram-matrix-based merging method (Gram-LS), showing that hyperparameter tuning matters more than method choice.

Critical strengths
  • The paper tries to address the important problem of consolidating specialized medical imaging models while preserving task performance.

  • The paper provides a systematic comparison of 11 merging methods across seven backbones under a controlled evaluation protocol.

Critical weaknesses
  1. Gram-LS is presented as a methodological contribution, yet it reduces exactly to Task Arithmetic when task vectors are orthogonal and achieves virtually identical performance (0.808 AUROC) in the main benchmark. Moreover, the authors report low pairwise task-vector similarities (≤0.14) and coefficients close to unity (0.90-0.98) for six of seven backbones.

Can the authors demonstrate a setting in which Gram-LS provides a statistically significant and practically meaningful improvement over properly tuned Task Arithmetic? If not, what justifies presenting Gram-LS as a novel methodological contribution rather than a geometric reformulation of an existing baseline?

  1. The authors claim that standardizing the number of training images to 2,000 per specialist ensures a controlled comparison across medical imaging domains. However, the underlying datasets differ substantially in size, class distribution, and intrinsic task difficulty. Furthermore, early stopping and backbone-specific learning rates mean that equal image counts do not imply equal optimization budgets or comparable task-vector geometries. How can the authors establish that the observed method rankings reflect intrinsic merging capabilities rather than artifacts of the arbitrary 2,000-image cap and heterogeneous optimization dynamics? Would the central conclusions remain unchanged under different training budgets and matched optimization steps?

  2. The paper argues that hyperparameter tuning effects exceed the performance differences among leading merging methods, based on paired comparisons across 21 backbone-seed cells. However, these observations are not independent: three seeds are nested within each of only seven backbone architectures. The authors also acknowledge that their statistical tests are unadjusted for multiple comparisons. Would the claimed dominance of hyperparameter tuning effects, as well as the apparent statistical indistinguishability of the leading methods, survive a backbone-level hierarchical analysis with appropriate multiple-comparison correction? How robust are the paper's central empirical conclusions?

Questions for authors

The authors should address the points listed in the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.