acceptodds
Withdrawn from ICLR 2027

MinimaxMerge: Balancing Task Retention in Training-Free Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training-free model merging combines task-specialized LLMs without additional data or training, yet existing methods largely focus on reducing interference among task vectors while optimizing aggregate performance, leaving the weakest task unprotected. We introduce MinimaxMerge, a training-free and data-free approach that instead optimizes a worst-task reconstruction objective when combining multiple specialists. Across mathematics, code, and science experts, MinimaxMerge achieves the highest worst-task retention on dense Gemma-4-12B (1.0067 vs. 1.0048 for the strongest baseline) while maintaining mean performance. On Gemma-4-26B-A4B, under attention-only adaptation, it achieves the highest worst-task accuracy, while TIES narrowly outperforms it in worst-task retention (1.0196 vs. 1.0187). Replacing the minimax objective with a mean objective decreases worst-task accuracy by 1.66 points, demonstrating that objective choice materially affects retention beyond the merging procedure itself. Cross-family experiments further show that these gains are not universal: TIES outperforms MinimaxMerge on OLMo-2-13B (0.9672 vs. 0.9542 worst-task retention), while the mean objective performs best on SmolLM3-3B. Together, these results establish the merge objective as an important but model-dependent design dimension for preserving capabilities in multi-task model merging.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model merging, large language models, training-free adaptation, multi-task learning, model composition

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_6W6BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes MinimaxMerge, a training- and data-free method for merging multiple task-specialized LLM adapters by directly optimizing a worst-case (minimax) objective over tasks. Specifically, it minimizes the maximum per-task reconstruction loss of reduced task vectors per merge unit (typically per layer), where each task’s surrogate loss is an exact quadratic defined by Gram inner products; the resulting convex min–max problem is solved via a tangent-cut (cutting-plane) LP with a certified primal–dual gap.

Critical strengths

1.Technical novelty and innovation. Introduces an explicit worst-case objective for coefficient selection in training-free merging, moving beyond aggregate/mean-focused objectives that can mask buy-out of the weakest task.

2.The experimental design is rigorous,following strict procedural standards and conducting detailed ablation studies.

Critical weaknesses

Technical limitations or concerns. The surrogate-to-accuracy link is unvalidated on the binding worst task (code): measured correlations are weak/wrong-signed, undermining the core premise that minimizing the surrogate protects real worst-task performance.

Experimental gaps or methodological issues. 1.Very limited task suite (T=3: GSM8K, ARC-C, MBPP/HumanEval) constrains generality; no larger multi-task rosters where minimax conservatism might matter most. 2.Many results are “not resolved” under the authors’ gate; adopting a decisive margin from unrelated work instead of a power analysis limits interpretability and masks effect sizes that might be practically relevant.

Clarity or presentation issues. 1.The narrative is verbose and leans heavily on preregistration language, which sometimes obscures central takeaways; key quantitative results could be surfaced more directly. 2.The current version of the paper significantly exceeds the page limit set by the conference.

Questions for authors

1.Can you provide results with a soft-robust objective (e.g., CVaR or log-sum-exp/temperatured max) to trade off conservatism and average performance, and does that alleviate the code-surrogate mismatch?

2.How sensitive are the results to allowing small negative coefficients (relaxing the simplex to an affine constraint with L2 or L1 regularization)? Could controlled extrapolation help the weakest task?

3.Can you report compute/time overhead relative to baselines (per-layer LP rounds, average number of cuts), and whether early stopping on the gap meaningfully changes accuracy?

4.Have you tried feature- or output-level surrogates (e.g., minimizing layer feature drift à la LOT, or logit-space reconstruction on a small unlabeled corpus) while keeping the rest of your pipeline identical?

Reviewer_Y4YvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the potential neglect of the weakest task in training-free model merging, where existing methods primarily emphasize aggregate performance. It proposes MinimaxMerge, which minimizes the maximum task-vector reconstruction loss through per-layer convex optimization with certified solutions. Experiments show modest improvements on dense Gemma models, but the gains are statistically inconclusive and do not consistently generalize across model families. Otherwise, another issue is that the manuscript exceeds the conference page limit and should be shortened accordingly.

Critical strengths
  1. The paper identifies a meaningful limitation of aggregate-oriented model merging and motivates worst-task protection as an alternative optimization objective.
  2. The proposed framework is mathematically well-defined, with exact quadratic reconstruction objectives and a tractable optimization procedure with explicit optimality-gap certificates.
  3. The objective-swap ablation and cross-family evaluations are valuable, and the transparent reporting of negative results strengthens the empirical analysis.
Critical weaknesses
  1. Insufficient justification of the central motivation. The paper assumes that minimizing the maximum task-vector reconstruction loss protects the weakest downstream task. However, parameter-space reconstruction error does not necessarily reflect task-specific performance degradation. More importantly, Section 6.3 reports no meaningful correlation between reconstruction loss and code performance (Pearson r = +0.17), although code is the weakest task in terms of raw accuracy. This substantially weakens the central motivation and the interpretation of the empirical improvements.
  2. Potential mathematical inconsistency in the objective ablation. Under Eqs. (3)–(5), minimizing the mean squared reconstruction loss admits uniform averaging as a global optimum. Therefore, the mean-objective control should produce the same merged parameters as Task Arithmetic when using identical task vectors, scaling, and merging configurations. However, Tables 1 and 2 report different results (worst-task accuracy: 0.5286 vs. 0.5331). This discrepancy raises concerns about the implementation consistency and the validity of the objective-swap analysis.
  3. Ambiguity in the definition of the weakest task. The paper alternates between worst-task accuracy and worst-task retention, which measure different properties. On dense Gemma, code determines the lowest raw accuracy, whereas ARC-Challenge determines the minimum normalized retention. However, the optimization objective does not explicitly account for either task accuracy or normalization by individual expert performance. Consequently, the connection between the stated motivation, optimization objective, and evaluation metrics remains unclear.
  4. Mismatch between per-layer optimization and global task protection. MinimaxMerge independently minimizes the worst reconstruction loss within each layer. However, minimizing the maximum loss separately across layers is not equivalent to minimizing the maximum accumulated reconstruction loss across tasks for the entire model. The proposed per-layer objective therefore does not directly establish global worst-task protection.
  5. Limited methodological novelty. The proposed objective is closely related to classical minimax-center optimization, while the certified cutting-plane procedure relies on established convex optimization techniques. Although applying this formulation to model merging is meaningful, the paper does not sufficiently establish why this particular geometric objective is preferable to alternative aggregation or risk-sensitive objectives. Furthermore, solver optimality certificates apply only to the reconstruction surrogate, not downstream task performance.
  6. Insufficient empirical evidence and limited generalizability. The improvements on dense Gemma are statistically unresolved, and the 1.66-point advantage over the mean-objective control is also inconclusive. Moreover, TIES or the mean objective performs better on other model families. The high-conflict experiment in Section 6.3 also fails to support the proposed relationship between task-vector conflict and minimax benefits (Spearman ρ = −0.40). Together, these results leave the conditions under which minimax merging is beneficial largely unexplained.
  7. Limited evidence for compatibility with interference-reduction methods. Combining TIES reduction with MinimaxMerge decreases worst-task performance relative to plain TIES in the dense experiment, with no demonstrated improvement in the OLMo replication. This challenges the proposed separation between interference reduction and objective optimization and suggests that the effectiveness of the minimax objective depends strongly on the underlying task-vector geometry.
Questions for authors
  1. Given the lack of correlation between reconstruction loss and code accuracy, what evidence supports using worst-case parameter reconstruction as a surrogate for worst-task performance? Under what conditions should this relationship hold?
  2. Could the authors explain why the mean-objective control differs from Task Arithmetic despite their mathematical equivalence under the stated formulation? Are there differences in scaling, constraints, or implementation?
  3. Which notion of worst-task protection is the intended target: raw accuracy, normalized retention, or reconstruction loss? How does the proposed objective account for the potentially different weakest tasks identified by these metrics?
  4. Why is independent per-layer minimax optimization expected to improve global worst-task retention? Could the authors clarify the relationship between the local objectives and the overall task-level objective?
  5. What is the substantive methodological advance beyond applying classical minimax-center optimization to task vectors? How does the approach compare conceptually with other risk-sensitive aggregation objectives?
  6. Given the statistically inconclusive results and cross-family performance reversals, what properties of task-vector geometry determine when minimax optimization is beneficial? How do the authors explain the negative results in the high-conflict analysis and the TIES-reduction experiments?
Reviewer_3UM1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

N/A

Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.