- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The SLIM uses individual-expert and equal-weight pair evaluations to construct a quadratic surrogate for merging-coefficient optimization on the simplex, demonstrating accurate aggregate-performance prediction and competitive merge quality with 36 evaluations on two eight-expert LLM setups.
Critical strengths
- The SLIM solves a very painful problem of reducing the number of evaluations when merging models, and this motivation is very good.
- The method is efficient and can be extended to multi model merging through a few pairwise merged evaluations. This method itself is very clever.
Critical weaknesses
There are limited additional benefit and scope. Surrogate optimization improves ID scores over the best measured lattice point by only 0.78 and 0.05 points, with mixed OOD effects.
Questions for authors
- See the Critical Weaknesses. 2.Were expert counts other than eight evaluated? If so, how does prediction accuracy change with the number of experts?
4 comments
Response (2/3): Weakness and Q1, additional benefit of surrogate optimization
"There are limited additional benefit and scope. Surrogate optimization improves ID scores over the best measured lattice point by only 0.78 and 0.05 points, with mixed OOD effects."
Short answer. We agree on ID AVG: optimizing the surrogate is not reliably better than taking the best lattice point, and after the correction in comment 1/3 the Llama ID difference on the same GPU node is −1.04 [−2.35, 0.33] (Table 18). Optimization still adds three benefits, detailed below: higher OOD AVG in four of five comparisons (three with intervals excluding zero), a more stable selection on Gemma, and predictions for unevaluated mixtures. Against other ways of spending the 36 evaluations, most of SLIM's advantage comes from the lattice design; GP-BO splits the four settings 2–2 with SLIM (Sec. 5.3). We answer the scope part together with Q2 in comment 3/3.
What optimization adds over the best lattice point.
- Higher OOD AVG in four of five comparisons. Across five distinct comparisons (two with all eight experts, Table 18; three with 5-expert subsets whose selections differ from those with all eight, Table 14), SLIM's OOD AVG is higher in four (+4.05, +1.70, and +2.12 with intervals excluding zero; +0.44 [−0.35, 1.22]) and lower in one (Gemma, −1.58 [−2.93, −0.14]); ID differences range from −1.04 to +1.08. Because the ID benchmarks are used for selection, the held-out OOD comparisons carry the main evidence (App. C.5). These differences reflect domain trade-offs; on Llama OOD, for example, the +4.05 comes mainly from AlpacaEval LC (+24.8; the best lattice point scores 10.0) and safety (+8.3), offset by multilingual (−17.6; Tables 19–20).
- A more stable selection on Gemma. When benchmark items are resampled 1,000 times, the best lattice point switches between Instruction A + B and Instruction B + Safety (ID AVG 65.34 vs. 65.19) in 40.9% of resamples, whereas SLIM keeps its three experts and moves by an ℓ1 distance above 0.1 in only 0.1% (Table 4; App. E.3). On Llama the mean distance is comparable (0.065 vs. 0.078); SLIM's distance exceeds 0.1 more often (17.5% vs. 7.8%) but 0.3 less often (4.0% vs. 7.8%; Table 4).
- Predictions where solutions lie. The surrogate predicts unevaluated mixtures at no extra evaluation cost. Near the selected solutions its errors are similar to the held-out MAE, although at the selected coefficients it overestimates AVG by 1.4–2.2 points (App. D.2); on sparse mixtures of two or three experts, where SLIM's solutions lie, its MAE is 1.3–2.0 versus 4.7–11.0 for ridge fitted to 36 random points (seed means; Table 17). The optimization itself takes 0.25 s (Gemma) and 1.62 s (Llama) on a CPU (App. C.3).
Most of SLIM's advantage comes from the lattice design (Sec. 5.3, Table 5). With the same experts, simplex, and 36 ID evaluations, SLIM and the best lattice point both score above the seed means of evolutionary merging (CMA-ES) on the simplex, MAP on the simplex, random best-of-36, and a ridge-fitted quadratic on 36 random points in all four settings. SLIM's paired intervals against these four baselines exclude zero in 15 of 16 cases (the exception is ridge on Llama ID, +0.37 [−0.85, 1.65]), and SLIM is higher in 46 of the 48 per-seed comparisons (Table 18). GP-BO (Gaussian-process Bayesian optimization), suggested by Reviewer LPDs, is on par: SLIM is higher on Gemma ID and Llama OOD and lower on Gemma OOD and Llama ID, with all four intervals excluding zero (Table 18), and SLIM's 36 evaluations can run in parallel, whereas GP-BO needs 28 sequential queries (App. C.3).
Response (3/3): Q2 and scope, expert counts other than eight
"Were expert counts other than eight evaluated? If so, how does prediction accuracy change with the number of experts?"
Not in the submission; the revision adds T = 5–7. We evaluate four subsets (S5A/S5B: the A/B experts for instruction, math, and code plus multilingual and safety; S6 = S5A + Instruction B; S7 = S6 + Math B) and re-measure T = 8, each on 50 held-out Dirichlet(1) mixtures (App. D.1, Table 13):
| Subset (T) | Gemma ρ / MAE | Llama ρ / MAE |
|---|---|---|
| S5A (5) | 0.974 / 2.20 | 0.881 / 0.79 |
| S5B (5) | 0.964 / 1.61 | 0.969 / 1.39 |
| S6 (6) | 0.945 / 2.07 | 0.978 / 0.90 |
| S7 (7) | 0.954 / 1.82 | 0.956 / 1.02 |
| All (8) | 0.926 / 1.79 | 0.951 / 1.01 |
MAE shows no systematic change with T, and ρ for T = 5–7 stays within 0.88–0.98, at the level of T = 8 (last row). With intervals from resampling the held-out mixtures, two differences from T = 8 exclude zero, one better and one worse: a higher ρ on Gemma S5A (+0.048 [0.003, 0.113]) and a higher MAE on Llama S5B (+0.38 [0.06, 0.70]; App. D.1). Each subset changes both T and which experts are included, and three Llama subsets were measured on a different GPU type than T = 8 (Table 13), so we do not read the differences as an effect of T. Selection within the 5-expert subsets (Table 14) is part of comment 2/3.
Each architecture's pool has eight experts, so we could not test T > 8, and we list this as a limitation (Sec. 6); the lattice would need T(T+1)/2 evaluations, e.g., 45 at T = 9. For context, Wang et al. (ICML 2026) report that, with cross-entropy loss and equal-weight merging, most of the improvement from nine experts comes from the first five or six and that gains flatten from about ten in a 16-expert pool (App. D.1); we have not tested whether this holds for SLIM. On scope beyond the expert count, the global scale in comment 1/3 is a first step outside the simplex (App. E.4).
We would be grateful if you would consider these additions in your assessment.
Thank you for the author's detailed response. All my concerns have been resolved, and I will increase my rating to 3.
Response (1/3): Revised PDF and a correction to Table 3
Thank you for the clear and constructive review. Here is where the revised PDF (changes in red) addresses it; comments 2/3 and 3/3 give our point-by-point reply. Unless noted, brackets are 95% CIs from an item-level paired bootstrap, with seeds held fixed for three-seed baselines.
Correction to Table 3, which your Weakness cites. Re-measuring revealed an inconsistency in the submitted Table 3 that favored SLIM: its best-lattice ID values came from search-time lattice evaluations whose MBPP scores, unlike SLIM's, were never rescored after CPU contention had made correct programs time out. The best lattice point's Llama coding score is 50.0 in the submission and 53.8 re-measured (Table 19; +0.76 AVG), and this change, together with smaller re-measurement differences, makes the Llama ID gap negative. The revised Table 3 reports the best lattice point measured on the same GPU node as Table 5 and keeps SLIM's submitted entries; on that node, SLIM, MAP (n = 45), and evolutionary merging (gen = 20) differ from Table 9 by at most 0.34 before rounding (App. C.5). SLIM's surrogate is fitted to the same search-time lattice evaluations; refitting SLIM on the re-measured lattice changes each selected coefficient by at most 0.02 (App. E.3). SLIM − best lattice point (AVG):
The best-vertex OOD entries of Table 3 change by +0.02 (Gemma) and +0.09 (Llama) to match Tables 11–12; the submitted entries came from a separate evaluation run of the same expert.