- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper evaluates MergeKit methods, and hierarchical or sequential compositions of them, on four LoRA experts. It then refines the merged model with anchored weighting and layer-adaptive scaling. It also proposes capability gates and compares screening rankings with confirmation rankings.
I recommend rejection. The submission reads more like an unfinished experiment log than a mature research paper. The technical contribution is not well established, key methodological details are missing, and inconsistencies in the figures and reported numbers undermine confidence in the conclusions.
Critical strengths
- Evaluating both target performance and retention of general capabilities is practically relevant.
- Checking screening results against a later confirmation stage is good evaluation practice.
Critical weaknesses
-
The contribution is mostly a sweep over an existing toolkit followed by configuration search. Running many existing merging methods is not, by itself, technically novel. The paper does not show that hierarchical chains generally outperform well-tuned single-method merges or joint fine-tuning. The winning recipe is tied to one experimental setting, and the proposed refinements are justified mainly by leaderboard scores.
-
The method is underspecified. The paper introduces capability-gated training scores, anchored weighting, layer-adaptive scaling, and several recipe laws, but gives neither complete mathematical definitions nor an explicit algorithm. The merge-stage capability thresholds are described only as a "small delta." Yet the reproducibility statement says the exact thresholds appear in Sections 3 and 4. The inputs and weighting rules at each chain stage are also unclear. Without these details, the central method cannot be independently assessed or reproduced.
-
The evaluation is too weak to support the claimed reliability guarantees. General capabilities are measured on only 48 examples per suite. Without uncertainty analysis, this cannot show that capabilities are preserved within small tolerances. The screening audit computes correlations over just five selected finalists, which restricts the score range and introduces selection effects. Low correlations in this setting therefore do not show that screening rankings are generally unreliable. Repeated data splits, uncertainty estimates, and multiple training seeds are needed, especially since the experiments otherwise use a single fixed seed.
-
The figures contain substantive inconsistencies, not just cosmetic ones:
- Figure 2 claims an MMLU gain of about 21 points, while Section 5.3 reports 16.7 points for the stated winning model.
- Figure 2 also shows a GSM8K drop, whereas the text reports no change.
- Figure 5 has a clipped title and mixes TMMLU+ terminology with GSS axis labels.
- Figure 6 uses GSS in its title but labels its axes as TMMLU+ screening means.
- The configuration labels are too small to read easily.
Every figure should map consistently to its metric, model, and experimental run.
-
The empirical study is incomplete, and several interpretations go beyond the evidence:
- The experiments use one base model and four experts trained on small subsets of a single benchmark.
- Transfer is assessed on only 256 ARC examples, and the planned CMMLU evaluation was dropped because of a loading issue.
- The ARC improvement from 0.945 to 0.969 corresponds to about six more correct answers and needs a paired uncertainty analysis.
- Multi-seed runs and confirmation on a second base model are deferred.
- Checkpoints that stay unchanged under larger training budgets do not prove that model capacity was never a bottleneck.
- Passing capability gates does not establish a general mechanism by which merging repairs lost capabilities.
-
The discussion and references contain verifiable factual errors:
- Sections 5.2 and 6 attribute Model Stock's poor performance to its needing far more than four models. However, the original paper explicitly emphasizes using only two fine-tuned models.
- The MergeKit reference is listed as COLM 2025, but the official publication appeared in the EMNLP 2024 Industry Track.
- Its cited identifier, arXiv:2503.13241, belongs to an unrelated compressive sensing paper.
These errors raise concerns about how the baselines are interpreted and how carefully the submission was prepared.
Questions for authors
- Beyond running existing MergeKit methods and picking the best configuration, what generalizable contribution does the paper make?
- Can the authors fully define the chain operations, weighting schemes, training score, and all capability thresholds?
- Can the winning chain be compared with the strongest single-method baselines under identical data splits, expert checkpoints, and evaluation budgets?
- Can larger evaluations, repeated splits, and multiple seeds support the claims about capability retention and screening reliability?
- Can the authors reconcile Figure 2 with Section 5.3, and clarify whether Figures 5 and 6 report GSS or TMMLU+ accuracy?
- Can the authors correct the interpretation of Model Stock and audit the bibliography?