- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper evaluates eleven weight-space merging methods across three binary medical-imaging tasks, seven pretrained backbones, and three seeds. It quantifies the performance cost of consolidating specialists, studies validation-search sensitivity, and introduces a projection-based Gram-LS rule. Its principal contribution is empirical: tuned summation is competitive and validation choices can matter more than differences among leading merging methods.
Critical strengths
-
A useful empirical question and informative negative results. The study measures storage consolidation against retaining task-specific specialists, with a common task formulation and multi-backbone evaluation. A strong tuned-summation baseline and the acknowledged lack of a Gram-LS advantage are useful findings; a benchmark need not introduce a winning algorithm to contribute.
-
The within-method tuning evidence and detailed result records add practical value. Independent aggregation of the supplied main-method records agrees with the reported means. These findings should be recognized separately from the method-ranking claims affected by the concerns below.
Critical weaknesses
-
Fidelity of central baseline comparisons remains unresolved. The supplied iso_c.py and iso_cts.py paths sum non-matrix updates rather than averaging them as specified in the reference Iso method. Iso-CTS also discards task-residual singular values and derives its isotropic scale from the summed-update spectrum, differing from the reference algorithm. The LiNeS path defaults to a plural layers regex while the ViT backend defines singular layer, without forwarding the backend pattern in the inspected constructors. These differences could affect the large Iso deficit and the interpretation of LiNeS ranking changes. Their actual experimental impact is not established because the executed source version is not bound to the records.
-
Actual grouped-split provenance needs clarification. The inspected CheXpert/ISIC loaders do not provide patient/lesion group keys to uncached split generation, which falls back to image-level splitting. Cached splits or preprocessing could resolve the discrepancy with the paper. This is not evidence of observed leakage, but it matters to what the benchmark measures.
These concerns affect central empirical conclusions rather than peripheral implementation details. They do not automatically invalidate the separate finding that tuning improves simple summation.
Questions for authors
- Which exact implementation generated the reported cells? Can reference-output checks for Iso and effective layer scales for LiNeS establish fidelity, or can the affected key comparisons be updated if the submitted paths were executed? I do not assume that correction will reverse the ranking.
- Can the actual split records confirm the patient/lesion separation described in the paper? Existing split evidence is sufficient if the submitted loader path was not used.
I currently recommend weak rejection, with substantial room to revise this assessment if these two issues are resolved. The concern is reliability of the benchmark conclusions, not the absence of a new state-of-the-art algorithm or a complete geometric explanation.