acceptodds
Under review as a conference paper at ICLR 2027

SLIM: Simplex-Lattice Interpolation Merging

Abstract

Optimizing merging coefficients for large language models can require many costly benchmark evaluations. We propose Simplex-Lattice Interpolation Merging (SLIM), which constructs a quadratic surrogate of aggregate performance on the coefficient simplex using a classical mixture design. Evaluations of individual experts and equal-weight pairs determine the surrogate with the minimum number of measurements needed to identify a general quadratic on this domain. SLIM then optimizes the surrogate without further target-metric evaluations. Experiments on two model architectures demonstrate accurate prediction of unseen multi-expert mixtures and competitive merge performance under limited evaluation budgets. Matched-budget comparisons show that structured evaluation points improve prediction fidelity over random designs, including those using regularized fitting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model merging, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_if7nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The SLIM uses individual-expert and equal-weight pair evaluations to construct a quadratic surrogate for merging-coefficient optimization on the simplex, demonstrating accurate aggregate-performance prediction and competitive merge quality with 36 evaluations on two eight-expert LLM setups.

Critical strengths
  1. The SLIM solves a very painful problem of reducing the number of evaluations when merging models, and this motivation is very good.
  2. The method is efficient and can be extended to multi model merging through a few pairwise merged evaluations. This method itself is very clever.
Critical weaknesses

There are limited additional benefit and scope. Surrogate optimization improves ID scores over the best measured lattice point by only 0.78 and 0.05 points, with mixed OOD effects.

Questions for authors
  1. See the Critical Weaknesses. 2.Were expert counts other than eight evaluated? If so, how does prediction accuracy change with the number of experts?
4 comments
AuthorsOpenReview ↗

Response (1/3): Revised PDF and a correction to Table 3

Thank you for the clear and constructive review. Here is where the revised PDF (changes in red) addresses it; comments 2/3 and 3/3 give our point-by-point reply. Unless noted, brackets are 95% CIs from an item-level paired bootstrap, with seeds held fixed for three-seed baselines.

  • Weakness and Q1 (additional benefit of surrogate optimization): Sec. 5.2 compares SLIM with the best lattice point using paired intervals and the stability of the selected coefficients (Tables 3–4, App. E.3); Sec. 5.3 (Table 5) adds matched-budget baselines on the simplex; Sec. 6 ("The Net Value of Optimization") states that the gain in AVG over the best lattice point is setting-dependent.
  • Q2 (expert counts): App. D.1 (Tables 13–14) evaluates prediction and selection on subsets with T = 5–7; the Limitations state that more than eight experts are untested.
  • Results from other reviewers' requests that bear on your Weakness: a global scale s on SLIM's coefficients, suggested by Reviewer LPDs, takes SLIM beyond the simplex with five evaluations beyond SLIM's own model and raises Gemma OOD AVG from 50.56 to 53.65, on an ID margin of only 0.43 (point estimates; on Llama, s = 1 remains best; App. E.4); with the over-refusal penalty requested by Reviewer pDqt, or with safety removed, SLIM stays ahead of the best lattice point in OOD AVG on Llama and behind it on Gemma (Table 25, App. F.1).

Correction to Table 3, which your Weakness cites. Re-measuring revealed an inconsistency in the submitted Table 3 that favored SLIM: its best-lattice ID values came from search-time lattice evaluations whose MBPP scores, unlike SLIM's, were never rescored after CPU contention had made correct programs time out. The best lattice point's Llama coding score is 50.0 in the submission and 53.8 re-measured (Table 19; +0.76 AVG), and this change, together with smaller re-measurement differences, makes the Llama ID gap negative. The revised Table 3 reports the best lattice point measured on the same GPU node as Table 5 and keeps SLIM's submitted entries; on that node, SLIM, MAP (n = 45), and evolutionary merging (gen = 20) differ from Table 9 by at most 0.34 before rounding (App. C.5). SLIM's surrogate is fitted to the same search-time lattice evaluations; refitting SLIM on the re-measured lattice changes each selected coefficient by at most 0.02 (App. E.3). SLIM − best lattice point (AVG):

Gemma IDGemma OODLlama IDLlama OOD
Submitted Table 3+0.78−1.55+0.05+4.34
Revised Table 3 (SLIM as submitted)+0.8−1.8−0.8+4.0
Both on the same GPU node (Table 18)+1.08 [−0.13, 2.35]−1.58 [−2.93, −0.14]−1.04 [−2.35, 0.33]+4.05 [2.53, 5.56]

The best-vertex OOD entries of Table 3 change by +0.02 (Gemma) and +0.09 (Llama) to match Tables 11–12; the submitted entries came from a separate evaluation run of the same expert.

AuthorsOpenReview ↗

Response (2/3): Weakness and Q1, additional benefit of surrogate optimization

"There are limited additional benefit and scope. Surrogate optimization improves ID scores over the best measured lattice point by only 0.78 and 0.05 points, with mixed OOD effects."

Short answer. We agree on ID AVG: optimizing the surrogate is not reliably better than taking the best lattice point, and after the correction in comment 1/3 the Llama ID difference on the same GPU node is −1.04 [−2.35, 0.33] (Table 18). Optimization still adds three benefits, detailed below: higher OOD AVG in four of five comparisons (three with intervals excluding zero), a more stable selection on Gemma, and predictions for unevaluated mixtures. Against other ways of spending the 36 evaluations, most of SLIM's advantage comes from the lattice design; GP-BO splits the four settings 2–2 with SLIM (Sec. 5.3). We answer the scope part together with Q2 in comment 3/3.

What optimization adds over the best lattice point.

  • Higher OOD AVG in four of five comparisons. Across five distinct comparisons (two with all eight experts, Table 18; three with 5-expert subsets whose selections differ from those with all eight, Table 14), SLIM's OOD AVG is higher in four (+4.05, +1.70, and +2.12 with intervals excluding zero; +0.44 [−0.35, 1.22]) and lower in one (Gemma, −1.58 [−2.93, −0.14]); ID differences range from −1.04 to +1.08. Because the ID benchmarks are used for selection, the held-out OOD comparisons carry the main evidence (App. C.5). These differences reflect domain trade-offs; on Llama OOD, for example, the +4.05 comes mainly from AlpacaEval LC (+24.8; the best lattice point scores 10.0) and safety (+8.3), offset by multilingual (−17.6; Tables 19–20).
  • A more stable selection on Gemma. When benchmark items are resampled 1,000 times, the best lattice point switches between Instruction A + B and Instruction B + Safety (ID AVG 65.34 vs. 65.19) in 40.9% of resamples, whereas SLIM keeps its three experts and moves by an ℓ1 distance above 0.1 in only 0.1% (Table 4; App. E.3). On Llama the mean distance is comparable (0.065 vs. 0.078); SLIM's distance exceeds 0.1 more often (17.5% vs. 7.8%) but 0.3 less often (4.0% vs. 7.8%; Table 4).
  • Predictions where solutions lie. The surrogate predicts unevaluated mixtures at no extra evaluation cost. Near the selected solutions its errors are similar to the held-out MAE, although at the selected coefficients it overestimates AVG by 1.4–2.2 points (App. D.2); on sparse mixtures of two or three experts, where SLIM's solutions lie, its MAE is 1.3–2.0 versus 4.7–11.0 for ridge fitted to 36 random points (seed means; Table 17). The optimization itself takes 0.25 s (Gemma) and 1.62 s (Llama) on a CPU (App. C.3).

Most of SLIM's advantage comes from the lattice design (Sec. 5.3, Table 5). With the same experts, simplex, and 36 ID evaluations, SLIM and the best lattice point both score above the seed means of evolutionary merging (CMA-ES) on the simplex, MAP on the simplex, random best-of-36, and a ridge-fitted quadratic on 36 random points in all four settings. SLIM's paired intervals against these four baselines exclude zero in 15 of 16 cases (the exception is ridge on Llama ID, +0.37 [−0.85, 1.65]), and SLIM is higher in 46 of the 48 per-seed comparisons (Table 18). GP-BO (Gaussian-process Bayesian optimization), suggested by Reviewer LPDs, is on par: SLIM is higher on Gemma ID and Llama OOD and lower on Gemma OOD and Llama ID, with all four intervals excluding zero (Table 18), and SLIM's 36 evaluations can run in parallel, whereas GP-BO needs 28 sequential queries (App. C.3).

AuthorsOpenReview ↗

Response (3/3): Q2 and scope, expert counts other than eight

"Were expert counts other than eight evaluated? If so, how does prediction accuracy change with the number of experts?"

Not in the submission; the revision adds T = 5–7. We evaluate four subsets (S5A/S5B: the A/B experts for instruction, math, and code plus multilingual and safety; S6 = S5A + Instruction B; S7 = S6 + Math B) and re-measure T = 8, each on 50 held-out Dirichlet(1) mixtures (App. D.1, Table 13):

Subset (T)Gemma ρ / MAELlama ρ / MAE
S5A (5)0.974 / 2.200.881 / 0.79
S5B (5)0.964 / 1.610.969 / 1.39
S6 (6)0.945 / 2.070.978 / 0.90
S7 (7)0.954 / 1.820.956 / 1.02
All (8)0.926 / 1.790.951 / 1.01

MAE shows no systematic change with T, and ρ for T = 5–7 stays within 0.88–0.98, at the level of T = 8 (last row). With intervals from resampling the held-out mixtures, two differences from T = 8 exclude zero, one better and one worse: a higher ρ on Gemma S5A (+0.048 [0.003, 0.113]) and a higher MAE on Llama S5B (+0.38 [0.06, 0.70]; App. D.1). Each subset changes both T and which experts are included, and three Llama subsets were measured on a different GPU type than T = 8 (Table 13), so we do not read the differences as an effect of T. Selection within the 5-expert subsets (Table 14) is part of comment 2/3.

Each architecture's pool has eight experts, so we could not test T > 8, and we list this as a limitation (Sec. 6); the lattice would need T(T+1)/2 evaluations, e.g., 45 at T = 9. For context, Wang et al. (ICML 2026) report that, with cross-entropy loss and equal-weight merging, most of the improvement from nine experts comes from the first five or six and that gains flatten from about ten in a 16-expert pool (App. D.1); we have not tested whether this holds for SLIM. On scope beyond the expert count, the global scale in comment 1/3 is a first step outside the simplex (App. E.4).

We would be grateful if you would consider these additions in your assessment.

Thank you for the author's detailed response. All my concerns have been resolved, and I will increase my rating to 3.

Reviewer_LPDsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reframes linear task-vector merging under a sum-to-one constraint as a classical mixture experiment, which is an elegant bridge between model merging and the design-of-experiments literature. SLIM evaluates the T experts and all equal-weight pairs (Scheffe's {T,2} simplex-lattice, 36 evaluations for T = 8), fits the unique quadratic interpolant of the average benchmark score over the simplex, and optimizes it without further evaluations. On Gemma-2-9B and Llama-3.1-8B with eight experts each, the surrogate accurately predicts held-out eight-expert mixtures. With only 36 evaluations, SLIM outperforms MAP at 45 evaluations and is competitive with evolutionary merging at 150–200 evaluations in three of four settings.

The practical impact is a cheap, tuning-free and deterministic recipe for choosing merge coefficients, together with the useful insight that pairwise merges carry most of the information needed to predict multi-expert mixtures.

Critical strengths
  1. The method is simple, principled and hyperparameter-free. It has a closed-form interpolant (Eq 7), costs only about 4–5 GPU-hours, and is easy to reproduce, tying to authors' commitment to release all measured scores.
  2. The evaluation is transparent. Fidelity is tested on 150 interior held-out mixtures that use all eight experts (construction uses at most two), plus edge holdouts, and informative ablations cover design placement, surrogate optimization and degree. The paper also reports candidly where SLIM does not win, which makes the positive results more credible.
Critical weaknesses

It is hard to tell how much of the gain over MAP and evolutionary merging comes from the lattice design and how much from the coefficient domain. SLIM searches the simplex, while those baselines search [0,1]^8 and select coefficient sums of 2.5–3.75 (Fig. 4).

The authors acknowledge this in Section 6. Running the baselines under the same simplex constraint, and adding a simple GP-based Bayesian optimization baseline at 36 evaluations, would make the efficiency claim much more convincing.

Questions for authors

Could SLIM be extended to also select the overall task-vector scale, for example by parameterizing λ = s·w with w on the simplex and evaluating a few values of s ?

4 comments
AuthorsOpenReview ↗

Response (1/3): Revised PDF

Thank you for the thoughtful review and the concrete suggestions. We ran both experiments you suggested and tested the scale extension you proposed. In short, at 36 evaluations on the simplex, SLIM is above MAP, CMA-ES, and random selection in all four settings and above ridge in three (tying the fourth), and it splits 2–2 with GP-BO (Gaussian-process Bayesian optimization) while needing no sequential rounds. Here is where the revised PDF (changes in red) addresses your review; comments 2/3 and 3/3 give our point-by-point reply.

  • Weakness, lattice design vs. coefficient domain: Sec. 5.3 and Table 5 (matched-budget comparison on the simplex, including random design + ridge, which shares SLIM's quadratic class), App. E.2 and Table 22 (CMA-ES in [0,1]^8 versus the simplex), and Sec. 6 ("Evaluation Design as the Main Source of Gains"; "The Simplex: Predictability versus Scale").
  • Weakness, GP-BO at 36 evaluations: Sec. 5.3, Table 5, and Figure 7 (selected coefficients of every matched-budget method); settings in App. C.3.
  • Question, overall scale: App. E.4 and Table 24.
  • Presentation: to make the paper easier to follow, Sec. 5 now reports results only and Sec. 6 collects the discussion, including the former Secs. 4.4 (selected coefficients) and 5.3 (polynomial degree).
  • Results from other reviewers' requests that bear on your Weakness: the best lattice point (raised by Reviewers if7n and pDqt), which uses the same 36 lattice evaluations without the surrogate, is also above the seed means of the four simplex baselines other than GP-BO in all four settings (point estimates; Table 5), so with the domain fixed, most of SLIM's advantage over these baselines comes from where the 36 evaluations are placed rather than from optimizing the surrogate. On mixtures of two or three experts, where the selected solutions lie, the lattice surrogate also predicts AVG with MAE 1.3–2.0, versus 4.7–11.0 (seed means) for ridge fitted to 36 random points (requested by Reviewer pDqt; App. D.3, Table 17), a design effect with the function class and domain fixed.
AuthorsOpenReview ↗

Response (2/3): Weakness, lattice design vs. coefficient domain and GP-BO

Brackets are 95% CIs from an item-level paired bootstrap (App. C.5); for three-seed baselines the seeds are held fixed.

Weakness: lattice design vs. coefficient domain

"It is hard to tell how much of the gain over MAP and evolutionary merging comes from the lattice design and how much from the coefficient domain. … Running the baselines under the same simplex constraint …"

All methods below use the same eight experts, the same simplex, and 36 ID evaluations (Table 5; three seeds where applicable; mean ± std; all models of an architecture on the same GPU node). "Evolutionary merging" in our paper is the CMA-ES method of Akiba et al. (2025), so we use CMA-ES with the original search settings and map candidates to the simplex by λ = x / Σx (App. C.3). Bold marks column maxima.

AVGGemma IDGemma OODLlama IDLlama OOD
SLIM66.4350.5656.3747.28
Best lattice point65.3452.1357.4043.23
GP-BO65.25 ± 0.5852.69 ± 0.7258.41 ± 1.2442.81 ± 1.89
Random design + ridge64.73 ± 1.7647.11 ± 2.9156.00 ± 1.5041.48 ± 1.01
CMA-ES (simplex)60.88 ± 0.9246.41 ± 2.1050.66 ± 1.1138.82 ± 1.28
MAP (simplex)58.85 ± 5.8944.00 ± 5.4548.97 ± 7.2436.61 ± 4.65
Random best-of-3659.25 ± 0.8144.23 ± 2.2950.44 ± 0.1838.88 ± 1.31
  • Design effect (domain fixed). The cleanest comparison is with ridge, which fits the same quadratic function class to 36 random points, so evaluation placement is the main difference: SLIM is higher on Gemma ID/OOD (+1.70 [0.55, 2.85] / +3.44 [2.03, 4.87]) and Llama OOD (+5.80 [4.43, 7.26]) and tied on Llama ID (+0.37 [−0.85, 1.65]; Table 18). SLIM also exceeds the seed means of simplex-constrained MAP (+6.6 to +10.7; MAP varies strongly across seeds) and CMA-ES (+4.1 to +8.5) in all four settings, all CIs excluding zero (Table 18). SLIM's lead over MAP in the paper's setting ([0,1]^8, 45 evaluations, re-measured) is +9.1 to +10.3 (Table 19), so most of it remains on the simplex.
  • Domain effect (method and budget fixed). We compared CMA-ES after 36 evaluations in the cube [0,1]^8 (first 36 evaluations of the paper's runs, re-measured) and on the simplex, seed by seed (App. E.2, Table 22). On Gemma the cube is lower on OOD in all three seeds (−4.3 to −10.8; ID −2.8 to +0.5), whereas on Llama it is higher on ID (+7.3 to +8.4) and OOD (+1.0 to +2.9). The cube solutions have coefficient sums of 2.8–4.1, so this effect includes overall scale.

In short, with the domain fixed, the lattice design outperforms or ties every other tested way of spending the 36 evaluations except GP-BO (discussed next), while the domain effect is model-dependent: the larger domain helps on Llama, consistent with Llama ID being where SLIM trails evolutionary merging in [0,1]^8 (Secs. 4.3 and 6), whose solutions have the large coefficient sums you noted (Figure 4).

Weakness: GP-based Bayesian optimization at 36 evaluations

"Running the baselines under the same simplex constraint, and adding a simple GP-based Bayesian optimization baseline at 36 evaluations, …"

GP-BO (8 vertices, then 28 expected-improvement queries with a Matérn-5/2 ARD kernel; App. C.3) is the strongest baseline; SLIM, GP-BO, and the best lattice point are the top three in every column. Neither SLIM nor GP-BO dominates (SLIM − GP-BO): Gemma ID +1.17 [0.28, 2.10] and OOD −2.14 [−3.19, −1.00]; Llama ID −2.05 [−3.23, −0.81] and OOD +4.47 [3.27, 5.74], with the same sign for every GP-BO seed (Table 18). SLIM's Llama OOD lead rests on safety (+27.15; Table 20): without safety, GP-BO's mean is higher, while with an over-refusal penalty SLIM stays ahead (48.10 vs. 47.46; Table 25). In all six runs GP-BO also selects sparse mixtures of two or three experts (largest coefficient 0.50–0.64; Figure 7), like SLIM, suggesting that the simplex contains a high-performing, predictable region that a surrogate can locate with few evaluations. The practical difference is that SLIM's 36 evaluations are fixed in advance and can run in parallel, with deterministic selection and no surrogate hyperparameters, whereas GP-BO needs 28 sequential rounds.

AuthorsOpenReview ↗

Response (3/3): Question, selecting the overall scale

"Could SLIM be extended to also select the overall task-vector scale, …?"

Yes. Keeping SLIM's w and evaluating λ = s·w for s ∈ {1.0, 1.5, …, 3.5} (five evaluations beyond SLIM's own s = 1 model), we select s by ID AVG (App. E.4). On Gemma, s = 1.5 is selected on an ID margin of only 0.43 (66.86 vs. 66.43) and raises OOD AVG from 50.56 to 53.65 (point estimates; Table 24). On Llama, s = 1 remains best and s ≥ 2 collapses coding to near zero (Table 24). SLIM's Llama solution uses only Instruction B and Safety (Figure 4), whereas the CMA-ES solutions after 36 evaluations in [0,1]^8, which score higher than SLIM on Llama ID (Table 19), put no more than 0.21 of their total weight on any one expert (Table 22), suggesting that s and w need to be chosen together. That would require a surrogate q_s(w) for each of K scales (36K evaluations) or a mixture–amount design (App. E.4); we leave it to future work.

AuthorsOpenReview ↗

Friendly reminder before the discussion deadline

Dear Reviewer LPDs,

Thank you again for your review. As the discussion period closes on November 18 (AoE), we wanted to briefly check in. Following your suggestions, we ran the baselines under the same simplex constraint, compared against GP-BO at 36 evaluations, and tested rescaling the overall coefficient scale (comments 2/3 and 3/3; Sec. 5.3, Table 5, and App. E.2 and E.4 of the revised PDF). We would be grateful to know whether these address your concerns, and we are happy to clarify anything further before the deadline. If they do, we would appreciate it if you could consider reflecting this in your assessment.

Reviewer_pDqtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates linear merging of experts sharing a common base as a mixture experiment on the coefficient simplex. SLIM adopts the classical quadratic Scheffe simplex-lattice design: evaluating the T individual experts and T(T−1)/2 equal-weight expert pairs uniquely identifies a general quadratic surrogate on the simplex. It then optimizes the surrogate using multistart SLSQP without further target-metric evaluations. Experiments use eight experts spanning five domains for each of Gemma-2-9B and Llama-3.1-8B, requiring 36 construction evaluations. On 150 held-out interior mixtures, the surrogate achieves AVG Spearman correlations of 0.936/0.959. The paper reports competitive merging under limited evaluation budgets and uses random-design, best-design-point, and cubic-interpolation ablations to examine evaluation placement and continuous optimization.

Critical strengths
  1. A simple, reusable method is supported by credible evidence of low-budget prediction fidelity. Table 2 compares lattice and random LS/ridge fits within the same quadratic function class and simplex, rather than relying only on unstable unregularized random interpolation. At 36 evaluations, random ridge MAE is 2.90 versus approximately 1.93 for the lattice on Gemma, and 2.24 versus approximately 1.07 on Llama. The 150 interior mixtures and 56 edge points are excluded from quadratic construction, providing substantive support for predicting multi-expert AVG from pairwise measurements.

  2. The paper evaluates cost and generalization and includes informative ablations. It reports GPU-hours as well as evaluation counts and reserves OOD benchmarks from coefficient selection. Table 3 compares the best vertex, best lattice point, quadratic SLIM, and cubic SLIM. This shows that much of the ID improvement comes from evaluating expert pairs and that increasing polynomial degree does not consistently improve OOD performance. These controls make the method's practical role and limitations measurable.

Critical weaknesses
  1. The benefit of surrogate optimization over direct candidate selection is not stable, and the main performance comparisons do not fully isolate its source. In Table 3, moving from the best lattice point to quadratic SLIM improves Gemma ID from 65.37 to 66.15 and Llama ID from 56.56 to 56.61. Gemma OOD decreases from 51.85 to 50.30, whereas Llama OOD improves from 42.93 to 47.27. Continuous optimization therefore does not consistently outperform direct selection using the same 36 evaluations. Table 2 controls the coefficient domain but evaluates prediction, not final model selection. The main MAP/CMA-ES comparisons still change evaluation placement, surrogate fitting, and feasible domain together. Since the mathematical construction is largely established, the contribution depends especially on whether its additional empirical selection benefit is robust.

  2. Generalization is evaluated on two fixed expert sets, with limited coverage of the optimization region and measurement uncertainty. The Dirichlet(1) interior points and two held-out points per edge are useful, but Section 4.4 shows that selected solutions concentrate on a few experts. Overall correlation does not directly validate these sparse optimum regions. Each architecture has one eight-expert set, and SLIM is reported for one run; deterministic optimizer initialization does not eliminate uncertainty from finite benchmarks, sample selection, or evaluator models. Moreover, Table 8 reports a Gemma math correlation of −0.24 and a safety-score IQR of 47.2, so aggregate fidelity does not imply reliable prediction in every domain. The use of refusal rate as the safety score also requires clarification about whether over-refusal can improve AVG. These limitations narrow the evidence for general low-cost merging across capabilities.

Questions for authors
  1. Net value of optimization: Under the same simplex, 36-evaluation budget, and expert set, how do SLIM, best-lattice-point selection, random candidate selection, and simplex-constrained search or surrogate baselines compare in per-domain ID/OOD scores and paired uncertainty? Does the prediction advantage in Table 2 translate into better final model selection?
  2. Reliability near selected solutions: Please report the surrogate error at the selected coefficients and a small local validation set excluded from selection. Do prediction and selection advantages persist for sparser mixtures or alternative expert subsets within the same domains? These checks would test whether global fidelity applies to the region that determines the final result.
  3. Evaluation uncertainty and AVG: Could benchmark-example resampling provide intervals for the key score differences, together with final-model scores for all five domains? Which WildGuardTest examples enter the refusal-rate score, and does it distinguish refusal of harmful requests from over-refusal of benign requests? Would penalizing the latter change the selected coefficients or relative method ranking?
5 comments
AuthorsOpenReview ↗

Response (1/4): Revised PDF and a correction to Table 3

Thank you for the careful and detailed review. Your questions led to most of the new experiments. In short, the revision narrows our claims: the gain over the best lattice point is setting-dependent; at the same budget, SLIM beats the same quadratic fitted to random points in three settings (tying the fourth) and random selection, MAP, and CMA-ES in all four, and splits 2–2 with GP-BO; and its Llama OOD lead depends on safety. Here is where the revised PDF (changes in red) addresses your review; comments 2/4 to 4/4 give our point-by-point reply. Table numbers refer to the revision, where the submission's Tables 7–8 are now Tables 9–10; the former Sec. 4.4 is now part of Sec. 6.

  • Q1/W1 (net value of optimization): Sec. 5.2 (Table 3 corrected; coefficient stability in Table 4 and App. E.3), Sec. 5.3 (Table 5 and Figure 7; intervals and domain scores in Tables 18–20, App. E.1), App. E.2 (coefficient domain), and Sec. 6 ("The Net Value of Optimization").
  • Q2/W2 (reliability near the selected solutions): App. D.1–D.3 (Tables 13–17) and App. E.3 (resampling the lattice measurements; SLIM_R).
  • Q3/W2 (intervals, five-domain scores, safety): paired intervals in Tables 18 and 20, with the method in App. C.5; five-domain scores in Tables 11–12 and 19; the safety benchmarks contain only harmful prompts (App. C.2); over-refusal and the safety weight in App. F (Tables 25–26); the Limitations state that SLIM's Llama OOD lead relies on safety.
  • A result from another reviewer's request that bears on W1: rescaling SLIM's coefficients as s·λ̂ (suggested by Reviewer LPDs; App. E.4) raises Gemma OOD AVG from 50.56 to 53.65 with five evaluations beyond SLIM's own model, on an ID margin of only 0.43 (point estimates); on Llama, s = 1 remains best. With App. E.2, this suggests that the overall scale, which the simplex fixes, contributes to the feasible-domain effect in W1.

Correction to Table 3, which W1 cites. Re-measuring revealed an inconsistency in the submitted Table 3 that favored SLIM: its best-lattice ID values came from search-time lattice evaluations whose MBPP scores, unlike SLIM's, were never rescored after we found that CPU contention had made correct programs time out. The best lattice point's Llama coding score is 50.0 in the submission and 53.8 re-measured (Table 19; +0.76 AVG). With this and smaller re-measurement differences, SLIM − best lattice point (Gemma ID / OOD, Llama ID / OOD) changes from +0.78 / −1.55 / +0.05 / +4.34 in the submission to +0.8 / −1.8 / −0.8 / +4.0 in the revised Table 3, which keeps SLIM's submitted entries, and to +1.08 / −1.58 / −1.04 / +4.05 when both are measured on the same GPU node (Table 18; first row of the table in comment 2/4). The revised Table 3 reports the best lattice point from the GPU node of Table 5; on that node, SLIM, MAP (n = 45), and evolutionary merging (gen = 20) differ from Table 9 by at most 0.34 before rounding (App. C.5). SLIM's surrogate is fitted to the same search-time lattice evaluations; refitting SLIM on the re-measured lattice changes each selected coefficient by at most 0.02 (App. E.3; comment 3/4). Best-vertex OOD now matches Tables 11–12 (+0.02 / +0.09), because the expert had been evaluated in two runs that differ only in the AlpacaEval annotation of identical outputs and, on Gemma, in 2 of 500 MATH-500 answers.

AuthorsOpenReview ↗

Response (2/4): Q1 and W1, net value of optimization

Brackets are 95% CIs from an item-level paired bootstrap (10,000 resamples, per comparison, no multiplicity correction; App. C.5); for three-seed baselines the seeds are held fixed, and per-seed differences are in Table 18.

Q1: "Under the same simplex, 36-evaluation budget, and expert set, how do SLIM, best-lattice-point selection, random candidate selection, and simplex-constrained search or surrogate baselines compare in per-domain ID/OOD scores and paired uncertainty?"

SLIM's AVG is 66.43 / 50.56 / 56.37 / 47.28 (Gemma ID / OOD, Llama ID / OOD; Table 5). SLIM minus each alternative (AVG, Table 18; positive values favor SLIM; both models of every comparison measured on the same GPU node; three seeds for every method except SLIM and the best lattice point):

SLIM − methodGemma IDGemma OODLlama IDLlama OOD
Best lattice point+1.08 [−0.13, 2.35]−1.58 [−2.93, −0.14]−1.04 [−2.35, 0.33]+4.05 [2.53, 5.56]
Random best-of-36+7.18 [6.02, 8.38]+6.33 [4.88, 7.79]+5.93 [4.48, 7.43]+8.41 [6.84, 9.98]
Random design + ridge+1.70 [0.55, 2.85]+3.44 [2.03, 4.87]+0.37 [−0.85, 1.65]+5.80 [4.43, 7.26]
MAP (simplex)+7.58 [6.48, 8.68]+6.56 [5.21, 7.94]+7.39 [6.15, 8.71]+10.67 [9.26, 12.12]
CMA-ES (simplex; "Evol. Merge" in Table 5)+5.55 [4.35, 6.75]+4.15 [2.69, 5.60]+5.70 [4.21, 7.24]+8.46 [6.84, 10.11]
GP-BO (simplex)+1.17 [0.28, 2.10]−2.14 [−3.19, −1.00]−2.05 [−3.23, −0.81]+4.47 [3.27, 5.74]

GP-BO (Gaussian-process Bayesian optimization suggested by Reviewer LPDs), the strongest baseline, splits 2–2 with SLIM; both select mixtures of two or three experts (Figure 7), and SLIM's 36 evaluations can run in parallel, whereas GP-BO needs 28 sequential queries (App. C.3). Per-domain scores (Table 19) and per-domain paired intervals (Table 20) are in App. E.1. Against the best lattice point, SLIM trades instruction following for safety on Gemma (ID IF −11.3, safety +14.4) and, on Llama, gains math and coding but loses multilingual (ID math +5.3, coding +3.4, multilingual −17.9); its Llama OOD lead (+4.05) comes from AlpacaEval LC (+24.8; the best lattice point scores 10.0) and safety (+8.3), offset by multilingual (−17.6; Tables 19–20).

  • (W1) "The benefit of surrogate optimization over direct candidate selection is not stable, …" We agree that the sign of the AVG gain varies across architectures and splits, and the ID differences are within uncertainty on both models (first row above). The selected coefficients, by contrast, are more stable than the best lattice point on Gemma: when benchmark items are resampled 1,000 times, the best lattice point switches between two different two-expert models (Instruction A + B vs. Instruction B + Safety; ID AVG 65.34 vs. 65.19) in 40.9% of replicates, whereas SLIM's coefficients move by a median ℓ1 distance of 0.03 and keep the same three experts (Table 4; App. E.3). On Llama the average change is comparable (mean ℓ1 0.065 vs. 0.078); SLIM changes more often by small amounts, but changes with ℓ1 > 0.3 are rarer (4.0% vs. 7.8%; Table 4).
  • (Q1) "Does the prediction advantage in Table 2 translate into better final model selection?" Partly. Ridge fitted to 36 random points uses the same function class and predicts worse (Table 2); it also selects worse models in three of four settings, including both OOD columns, and is within uncertainty on Llama ID (Table 18).
  • (W1) "The main MAP/CMA-ES comparisons still change evaluation placement, surrogate fitting, and feasible domain together." All rows above share the simplex and the budget. The best lattice point, which uses the same 36 evaluations without the surrogate, is also above the seed means of MAP and CMA-ES on the simplex in all four settings (point estimates; Table 5), so with the domain fixed, most of SLIM's advantage over them comes from where the evaluations are placed rather than from optimizing the surrogate (Sec. 6). Holding CMA-ES and 36 evaluations fixed, moving from [0,1]^8 to the simplex raises Gemma OOD AVG by 4.3–10.8 points but lowers Llama ID AVG by 7.3–8.4 (Table 22), so the domain effect depends on the model.
  • (W1) "Since the mathematical construction is largely established, …" We agree and do not claim the simplex-lattice construction as new: SLIM takes Scheffé's {T,2} design and its quadratic interpolant from classical mixture experiments (Secs. 2 and 3). Our aim is to bring this design to LLM merging and to characterize empirically when it suffices for coefficient selection, which the matched-budget comparisons above test directly (Sec. 5.3).
AuthorsOpenReview ↗

Response (3/4): Q2 and W2, reliability near the selected solutions

  • (Q2) "Please report the surrogate error at the selected coefficients …" The surrogate overestimates AVG there by 2.17 (Gemma) and 1.41 (Llama) points (App. D.2), slightly more than its held-out MAE (1.93 and 1.07; Sec. 4.2), as expected when selecting a surrogate's maximizer, and by 2.99 and 1.76 when refit on the re-measured lattice; the largest domain errors are Gemma safety (+11.6) and Llama multilingual (+10.1; Table 15).
  • (Q2) "… and a small local validation set excluded from selection." 16 mixtures moving the selected coefficients toward each expert by ε ∈ {0.05, 0.10}: MAE 1.79 (Gemma; mostly optimistic, mean +1.55) and 0.75 (Llama); maximum error 2.45 and 2.05 (Table 16).
  • (Q2) "Do prediction and selection advantages persist for sparser mixtures …" and (W2) "Overall correlation does not directly validate these sparse optimum regions." The prediction advantage over random designs persists where selected solutions concentrate (MAE, SLIM vs. ridge on 36 random points, mean of three seeds; App. D.3, Table 17):
Held-out setGemmaLlama
Two-expert mixtures on edges (56)1.74 vs. 11.041.38 vs. 10.88
Three-expert equal mixtures (56)1.96 vs. 5.841.34 vs. 4.70
Local set above (16)1.79 vs. 2.360.75 vs. 9.36
Interior mixtures of all eight (37–42)1.81 vs. 3.060.99 vs. 1.98

The first three sets lie outside the interior region that random designs sample, so ridge extrapolates there; on Gemma's local set, one ridge seed is more accurate (1.19; App. D.3).

  • (Q2) "… or alternative expert subsets within the same domains?" and (W2) "Each architecture has one eight-expert set, …" S5A and S5B each keep one expert per domain (A or B variants); S6 and S7 extend S5A (App. D.1). AVG prediction on 50 held-out mixtures each reaches ρ 0.88–0.98 (MAE 0.79–2.20), comparable to T = 8 (App. D.1, Table 13; Llama S5B has a higher MAE, +0.38 [0.06, 0.70], with the interval from resampling held-out mixtures). In the three 5-expert cases whose selections differ from those with all eight, SLIM exceeds the best subset lattice point on OOD with intervals excluding zero in two (+1.70 and +2.12; third +0.44 [−0.35, 1.22]), with ID differences of −0.91 to +1.03 (Table 14). These subsets come from the same pool, so one expert set per architecture remains a limitation (Sec. 6).
  • (W2) "… SLIM is reported for one run; …" SLIM is deterministic given its 36 evaluations, so its uncertainty comes from the evaluations. Resampling benchmark items 1,000 times (the uncertainty from finite benchmarks and from which items are sampled) and refitting SLIM each time moves its coefficients by a median ℓ1 distance of 0.03 on both models (Table 23); on Llama a small Instruction A weight (median 0.07) enters in 25.8% of replicates (App. E.3). Refitting SLIM on an independent re-measurement of all 36 lattice points changes each coefficient by at most 0.02 and OOD AVG by +0.02 and −0.24, but ID AVG by −0.88 [−1.40, −0.37] and +0.61 (App. E.3), so ID differences of about one point between nearby models should be read with care. The AlpacaEval judge (gpt-4o-2024-08-06, one annotation per reported score; App. C.5) and the WildGuard classifier are held fixed, so our intervals exclude evaluator variance.
  • (W2) "Moreover, Table 8 reports a Gemma math correlation of −0.24 and a safety-score IQR of 47.2, …" (now Table 10). We agree; Sec. 4.2 limits our claim to AVG, and the domain errors at the selected point above (Table 15) show the same pattern.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the PDF revision

We thank all reviewers for their comments. We have uploaded a revised PDF in which all changes are marked in red; the main text still ends on page 10. Reviewer IDs in brackets show whose comments each change addresses, and our detailed replies remain under each review.

Added

  • Matched-budget comparison on the simplex (same eight experts, simplex, and 36 ID evaluations): GP-BO (Gaussian-process Bayesian optimization), random design + ridge, MAP and evolutionary merging (CMA-ES) on the simplex, and random best-of-36, with item-level paired bootstrap intervals — Sec. 5.3, Table 5, Figure 7, App. E.1 [if7n, LPDs, pDqt]
  • SLIM versus the best lattice point: paired intervals and the stability of the selected coefficients under item resampling — Sec. 5.2, Table 4, App. E.3 [if7n, pDqt]
  • Expert subsets with T = 5–7, for prediction and selection — App. D.1 [if7n, pDqt]
  • Prediction error at and near the selected coefficients and on sparse mixtures — App. D.2–D.3 [pDqt]
  • Coefficient domain ([0,1]^8 versus the simplex at a fixed budget) and a global scale on SLIM's coefficients — App. E.2, E.4 [LPDs]
  • Over-refusal on benign prompts and sensitivity to the safety weight — App. F [pDqt]
  • Domain scores of every method, the base model, and the experts — Tables 11–12 [pDqt]; hardware, intervals, and deviations from the pre-specified protocol — App. C.5; symbols of the new analyses — Table 6

Corrected

  • Table 3 now reports the best lattice point measured on the same GPU node as Table 5, and its best-vertex OOD values match Tables 11–12. The submitted best-lattice ID values contained an MBPP scoring inconsistency that favored SLIM: on Llama ID, the paired comparison on the same GPU node gives SLIM − best lattice point = −1.04 [−2.35, 0.33] instead of the submitted +0.05. Details are in our replies to if7n and pDqt.

Merged and reorganized

  • Former Sec. 4.4 (selected coefficients) and Sec. 5.3 (polynomial degree) are merged into Sec. 6, which now has five paragraphs: sufficiency of a quadratic surrogate; evaluation design as the main source of gains; the net value of optimization; the simplex: predictability versus scale; limitations. Sec. 5.3 is now the matched-budget comparison.
  • The appendix adds C.5 and D–F; former C.5–C.7 are now C.6–C.8. The two new main-text tables shift later table numbers by two (e.g., submission Tables 7–8 are now Tables 9–10).

Removed

  • No submitted result was removed; we only cut text that Sec. 5 and Sec. 6 repeated.

Claims narrowed

  • The gain of surrogate optimization over the best lattice point is setting-dependent (Sec. 5.2, Sec. 6).
  • SLIM's Llama OOD lead relies on safety, although with an over-refusal penalty its rank among the 36-evaluation methods is unchanged (App. F.1).
  • At the selected coefficients, the surrogate overestimates AVG by 1.4–2.2 points (App. D.2), and each architecture uses one expert pool, so more than eight experts are untested (Limitations).
AuthorsOpenReview ↗

Summary of the rebuttal and discussion

Dear Area Chair and Reviewers,

We thank the reviewers for their constructive feedback. Below is a brief summary of the rebuttal; details are in our replies under each review and in the revised PDF (changes in red).

SLIM in brief. SLIM selects merging coefficients for T expert LLMs on the coefficient simplex. It evaluates the T experts and their equal-weight pairs (36 evaluations for T = 8), the minimum needed to identify a general quadratic on the simplex (Scheffé's simplex-lattice design), interpolates a quadratic surrogate of aggregate performance from them, and optimizes it without further benchmark evaluations. We do not claim the design itself as new; our contribution is its use for cheap and predictable coefficient search.

Discussion status. We responded point by point to all concerns raised by the three reviewers. Reviewer if7n replied and raised their rating from 2 to 3. Reviewers LPDs and pDqt have not yet responded during the discussion period, so we summarize below how we addressed their concerns.

Main concerns and what we added

  1. Net value of surrogate optimization (if7n, LPDs, pDqt). We added matched-budget baselines on the same simplex with the same 36 evaluations: random best-of-36, a ridge-fitted quadratic on random points, MAP, CMA-ES, and GP-BO (suggested by LPDs). SLIM and the best lattice point both score above random selection, MAP, CMA-ES, and the ridge surrogate in all four architecture × ID/OOD settings (SLIM's intervals exclude zero in 15 of 16 cases), and SLIM splits the four settings 2–2 with GP-BO (Sec. 5.3, Table 5). Most of SLIM's value therefore lies in the lattice design; against the best lattice point, the sign of the gain depends on the setting, which the revision now states. What optimization adds is a more stable selection on Gemma (under item resampling, the best lattice point switches between two different models in 40.9% of resamples, while SLIM's coefficients move by a median ℓ1 distance of 0.03) and predictions for unevaluated mixtures (Sec. 5.2, Sec. 6).
  2. Lattice design vs. coefficient domain (LPDs). With the domain fixed, the lattice design leads the ridge surrogate, MAP, and CMA-ES. Running CMA-ES in the cube and on the simplex at the same budget shows that the domain effect is model-dependent: the cube helps on Llama, the simplex on Gemma OOD (App. E.2).
  3. Generalization and reliability (if7n, pDqt). Prediction accuracy remains comparable for expert subsets with T = 5–7 (ρ 0.88–0.98 on 50 held-out mixtures each), near the selected solutions (local MAE 0.75–1.79, although the prediction at the selected point itself is optimistic by 1.4–2.2), and on sparse mixtures, where SLIM's error is lower than that of random designs (e.g., 1.4–1.7 vs. about 11 on two-expert mixtures) (App. D.1–D.3).
  4. Safety metric and over-refusal (pDqt). We clarified that the score uses harmful prompts only, and we newly measured over-refusal on 940 benign prompts for all 54 final models. With an over-refusal penalty, SLIM's rank among the 36-evaluation methods is unchanged. We also disclose that on Llama SLIM has the highest over-refusal rate (16.3%) and that part of its advantage comes from refusing harmful prompts (App. F).
  5. Overall scale (LPDs). Tuning a global scale on top of SLIM's coefficients improves Gemma OOD AVG from 50.6 to 53.7 with five extra evaluations, on an ID margin of only 0.43 (point estimates), and gives no gain on Llama (App. E.4).

Correction and limitations

While re-measuring, we found an inconsistency in Table 3 that favored SLIM: its best-lattice ID values came from search-time lattice evaluations whose MBPP scores, unlike SLIM's, were never rescored after a grading problem was fixed. For Llama, this, together with smaller re-measurement differences, changes the same-GPU-node ID gap from +0.05 to −1.04 [−2.35, 0.33] (Table 18). The revised Table 3 reports the best lattice point from that GPU node, while SLIM keeps its submitted entry (within 0.28 of Table 5), and our responses explain the cause. The revision also states the remaining limitations: one expert set per architecture, T ≤ 8, and a safety-dependent advantage on Llama.

In summary, the additions support the paper's central claim that a minimal structured design with 36 evaluations yields an accurate surrogate and competitive merges, and the revision states more precisely where continuous optimization adds value and where it does not.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.