acceptodds
Withdrawn from ICLR 2027

Beyond Single-Method Merging: Hierarchical Chains and Capability-Guarded Evaluation of LoRA Experts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging offers a nearly training-free alternative to costly joint fine-tuning, yet practical guidance on which merging recipe to use remains anecdotal. Furthermore, merged models can quietly trade away general capabilities for target-domain gains. We conduct an exhaustive, guard-railed registry sweep: all twenty merge methods exposed by the mergekit toolkit are natively exercised against four real LoRA experts (Chinese, math, social, science) fine-tuned on real TMMLU+ data from gemma-4-26B-A4B-it. Across 105 structural configurations (singles, hierarchies, sequential chains, and a passthrough control), each recipe is scored on TMMLU+ under a strict three-metric capability contract (MMLU, GSM8K, IFEval). A subsequent refinement stage optimizes the best recipes via anchored weighting and layer-adaptive scaling (44 additional configurations). Empirically, our results strongly validate the efficacy of merging: the best confirmed recipe (a performance-anchored DELLA to RAM chain) elevates macro TMMLU+ from 0.629 (base) to 0.720, passes all capability gates, and improves unseen-dataset transfer (ARC: 0.969 vs. 0.945). Crucially, our rigorous audit surfaces a systemic cautionary finding: across three independent runs, small-sample screening rankings fail to predict confirmation rankings (Spearman ρ ≤ 0.30), underscoring that leaderboard evaluations of merge recipes without explicit large-sample confirmation are inherently unreliable. We release all YAML recipes, scores, and audit artifacts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Merging, LoRA, Hierarchical Merging, Merge Chains, Capability Preservation, MergeKit, Gemma

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2s8YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates MergeKit methods, and hierarchical or sequential compositions of them, on four LoRA experts. It then refines the merged model with anchored weighting and layer-adaptive scaling. It also proposes capability gates and compares screening rankings with confirmation rankings.

I recommend rejection. The submission reads more like an unfinished experiment log than a mature research paper. The technical contribution is not well established, key methodological details are missing, and inconsistencies in the figures and reported numbers undermine confidence in the conclusions.

Critical strengths
  • Evaluating both target performance and retention of general capabilities is practically relevant.
  • Checking screening results against a later confirmation stage is good evaluation practice.
Critical weaknesses
  • The contribution is mostly a sweep over an existing toolkit followed by configuration search. Running many existing merging methods is not, by itself, technically novel. The paper does not show that hierarchical chains generally outperform well-tuned single-method merges or joint fine-tuning. The winning recipe is tied to one experimental setting, and the proposed refinements are justified mainly by leaderboard scores.

  • The method is underspecified. The paper introduces capability-gated training scores, anchored weighting, layer-adaptive scaling, and several recipe laws, but gives neither complete mathematical definitions nor an explicit algorithm. The merge-stage capability thresholds are described only as a "small delta." Yet the reproducibility statement says the exact thresholds appear in Sections 3 and 4. The inputs and weighting rules at each chain stage are also unclear. Without these details, the central method cannot be independently assessed or reproduced.

  • The evaluation is too weak to support the claimed reliability guarantees. General capabilities are measured on only 48 examples per suite. Without uncertainty analysis, this cannot show that capabilities are preserved within small tolerances. The screening audit computes correlations over just five selected finalists, which restricts the score range and introduces selection effects. Low correlations in this setting therefore do not show that screening rankings are generally unreliable. Repeated data splits, uncertainty estimates, and multiple training seeds are needed, especially since the experiments otherwise use a single fixed seed.

  • The figures contain substantive inconsistencies, not just cosmetic ones:

    • Figure 2 claims an MMLU gain of about 21 points, while Section 5.3 reports 16.7 points for the stated winning model.
    • Figure 2 also shows a GSM8K drop, whereas the text reports no change.
    • Figure 5 has a clipped title and mixes TMMLU+ terminology with GSS axis labels.
    • Figure 6 uses GSS in its title but labels its axes as TMMLU+ screening means.
    • The configuration labels are too small to read easily.

    Every figure should map consistently to its metric, model, and experimental run.

  • The empirical study is incomplete, and several interpretations go beyond the evidence:

    • The experiments use one base model and four experts trained on small subsets of a single benchmark.
    • Transfer is assessed on only 256 ARC examples, and the planned CMMLU evaluation was dropped because of a loading issue.
    • The ARC improvement from 0.945 to 0.969 corresponds to about six more correct answers and needs a paired uncertainty analysis.
    • Multi-seed runs and confirmation on a second base model are deferred.
    • Checkpoints that stay unchanged under larger training budgets do not prove that model capacity was never a bottleneck.
    • Passing capability gates does not establish a general mechanism by which merging repairs lost capabilities.
  • The discussion and references contain verifiable factual errors:

    • Sections 5.2 and 6 attribute Model Stock's poor performance to its needing far more than four models. However, the original paper explicitly emphasizes using only two fine-tuned models.
    • The MergeKit reference is listed as COLM 2025, but the official publication appeared in the EMNLP 2024 Industry Track.
    • Its cited identifier, arXiv:2503.13241, belongs to an unrelated compressive sensing paper.

    These errors raise concerns about how the baselines are interpreted and how carefully the submission was prepared.

Questions for authors
  • Beyond running existing MergeKit methods and picking the best configuration, what generalizable contribution does the paper make?
  • Can the authors fully define the chain operations, weighting schemes, training score, and all capability thresholds?
  • Can the winning chain be compared with the strongest single-method baselines under identical data splits, expert checkpoints, and evaluation budgets?
  • Can larger evaluations, repeated splits, and multiple seeds support the claims about capability retention and screening reliability?
  • Can the authors reconcile Figure 2 with Section 5.3, and clarify whether Figures 5 and 6 report GSS or TMMLU+ accuracy?
  • Can the authors correct the interpretation of Model Stock and audit the bibliography?
Reviewer_PjVVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper benchmarks existing merging methods and compositional recipes for four LoRA experts, with capability constraints and screening-versus-confirmation evaluation.

Critical strengths

Broad experimental coverage and disclosure of evaluation budgets, training settings, and failed reliability audits.

Critical weaknesses
  1. Limited scientific contribution. The paper reads primarily as an experimental report combining existing methods. It provides no convincing mechanistic or generalizable insight: explanations such as “regularization” and “compounding interference” are asserted without controlled evidence.
  2. Insufficient statistical support. General-capability evaluations use only 48 examples per suite, and screening–confirmation correlations are computed over just five selected finalists., These results cannot substantiate strong claims about capability preservation or recipe superiority.
  3. Unclear comparisons and narrow validation. The promoted chain scores 0.720, whereas single-method DELLA reaches 0.723 in another run with different reference scores. Without matched comparisons, multiple seeds, or a second backbone, the central claim favoring compositional merging remains unsupported.
Questions for authors

I rewrite the weakness above into the questions below:

  1. What new scientific insight does this study establish beyond a configuration sweep, and which controlled experiments validate the proposed mechanisms?
  2. Do capability preservation, ranking instability, and recipe superiority persist under larger model params evaluations
  3. Under identical experts, evaluation samples, and tuning budgets, does the proposed chain outperform single-method DELLA, and does this advantage transfer to another backbone?

And, could the authors explain the text in Line 151-152, "Results are reproduced as a snapshot of the live tree-search run as of 2026-09-25 18:00 (GMT+8)."?

Reviewer_qTvLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares 20 MergeKit methods for combining four LoRA experts, including single-method, hierarchical, and sequential recipes. It evaluates target-task performance alongside general-capability checks and uses screening–confirmation comparisons to assess ranking reliability.

Critical strengths

The sweep covers a broad set of practical merging recipes. Checking general capabilities alongside target accuracy, and openly reporting unstable screening rankings, are useful aspects of the study.

Critical weaknesses
  • The capability-preservation claim is not adequately supported. The main capability checks use only 48 examples per benchmark; on GSM8K, one answer changes accuracy by 2.08 percentage points. Passing point-estimate gates on these small samples, without paired uncertainty estimates, does not establish that capabilities have been preserved.

  • The ranking audit does not justify the broader conclusions. Correlations are computed over only five finalists already selected by screening, which cannot establish that rankings across the full registry are inherently unreliable. The selection criterion is also unclear: Eq. (1) presents GSS as the headline score, but Table 4 does not consistently follow GSS order.

Questions for authors

See Weaknesses, Please.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.