acceptodds
Under review as a conference paper at ICLR 2027

BOTAS: Bayesian Optimization for Zero-Shot Source Task Selection in Merging of LLMs

Abstract

Model merging combines independently fine-tuned checkpoints into a single model without joint training. However, existing evaluations assess the merged model only on the same in-domain source tasks, leaving zero-shot transfer to an unseen target task underresearched. To this end, we conduct an exhaustive study of source task subset selection for model merging, evaluating all subsets of 12 source tasks against 12 disjoint held-out target tasks, each at 40 scaling coefficients. Our results show that transfer is strongly aligned with the task family and that optimal mixtures typically involve fewer than 5 source tasks. We also show that a naive training-free solution based on training-free geometric selection, i.e., a task similarity measured by alignment between task vectors, does not identify the best mixture. To address this, we introduce BOTAS (Bayesian Optimization for TAsk Selection), which models the discrete space of source subsets with a Gaussian process surrogate. Without access to any target training data, BOTAS recovers the true optimal subset in 83% of runs while exploring only 1.5% of the search space. In addition, we show that in four out of twelve target tasks, the best merged subset outperforms full fine-tuning on the target itself. Finally, we release the full evaluation as a reusable benchmark for future source-selection methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, task arithmetic, source task selection, Bayesian optimization, multi-task transfer learning, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PJTqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies selecting subsets of existing task models to merge for a different target task. It evaluates 4,083 subsets of 12 sources across 40 shared scaling coefficients and 12 targets. BOTAS uses binary subset indicators, a standard Gaussian process, and Expected Improvement with labeled target evaluations. Within 63 subset queries, it reaches the exhaustive optimum in 83% of runs, versus 7% for random search (pp. 4–6; p. 20, Table 12).

Critical strengths
  1. Studying transfer beyond the source tasks is useful for reusing existing models. The exhaustive table allows selected mixtures to be compared directly with the optimum and makes these expensive evaluations available for subsequent studies.
  2. GP/EI suits this expensive, discrete black-box problem, and the recovery-rate improvement over random search is substantial. Geometric scores show some correlation with performance yet fail to reliably select the best mixture, cautioning against using parameter similarity directly for selection.
Critical weaknesses
  1. The discussion on p. 3 already covers source selection and subset selection before merging, making transfer through a selected combination an important distinction. Yet single-source candidates are excluded (p. 4), and their target performance is not reported. The results therefore do not establish how much is gained by combining sources. This leaves the distinction from existing source selection unclear and weakens an interpretation based on useful source complementarity.
  2. Standard GP/EI is a reasonable choice, but Table 4 and Appendix omit forward-addition or backward-removal searches with matched feedback and budgets. Beating random search does not establish BO's additional benefit. Cost reporting also needs correction: each subset query in Eq. 5 evaluates 40 coefficients, so 63 queries require 2,520 target-set evaluations. The relative saving in subset queries remains valid, but it should not be described as only 63 model evaluations.
  3. The use of target labels is explicit and retrospective lookup validly evaluates search efficiency. However, selection and final reporting use the same examples. The four reported wins over target fine-tuning therefore do not establish generalization to independent examples. With only 56 examples for CB and 100 for COPA, selection bias is particularly relevant. Fixed-table optimization success and practical transfer gains should be reported separately.
Questions for authors
  1. Compared with the best single-source transfer, what evidence shows that combining sources itself provides a benefit? How does this distinguish the contribution from the source-selection work discussed on p. 3?

  2. How do forward-addition or backward-removal searches compare with BOTAS under the same 63-query budget? Could the comparison also report the model evaluations required by the inner coefficient sweeps?

  3. Do the four claimed wins over target fine-tuning on p. 7 persist on target examples excluded from subset and coefficient selection?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for an insightful review.

We address the weaknesses and questions in the following text and in the revised version of our paper.

Weakness 1 + Question 1: Single-source candidates are excluded

We have run the 12 single sources over the same 40 coefficients and the same 12 targets as the main grid, in every cell.

ModelSeedMixture betterMean gainMax gainTargets where a single source wins
Llama-3.2-1B4211/12+0.051+0.209CoLA
Llama-3.2-1B12310/12+0.049+0.195CoLA, COPA
Qwen2.5-1.5B4212/12+0.075+0.203none
Qwen2.5-1.5B12312/12+0.071+0.161none

Merging a selected subset beats the best single source on 10 to 12 of the 12 targets, depending on the seed and model, with mean gains ranging from 0.049 to 0.075, up to 0.209. The exceptions are confined to Llama, namely CoLA in both of its cells and COPA at seed 123, and on Qwen every target prefers a mixture. The gain is therefore not an artifact of one model or one seed, and the targets where a single source wins are few enough to name. CoLA is the one target that no source family covers, which is consistent with the reading we already give it in the paper.

We have added this comparison to Section 5.2 (Table 4), with the values for every cell in Appendix C.2 (Table 9), as it provides evidence that the problem we pose is not the source-selection problem in the prior work the reviewer cites.

Weakness 2 + Question 2: A discrete search baseline that is not random, and the reported cost

On the baseline. We have added greedy search over subset membership, scored retrospectively against the same grids and reported at the same budget of 63 subset queries. It appears in Table 1, in Sections 5.1 and 5.3, and for every cell in Appendix C.2.

Forward selection starts from the best single source and repeatedly adds the source that most improves target accuracy. Backward selection starts from all 12 sources and repeatedly drops one. Both stop when no single move improves accuracy. The results of the greedy forward and backward selection are shown in the following table:

MethodBudgetLlama 42Llama 123Qwen 42Qwen 123
Optimum Found
Greedy search (forward selection)6358%42%50%42%
Greedy search (backward selection)638%25%8%17%
BOTAS (ours)6383%74%68%65%
Diff to Optimum
Greedy search (forward selection)630.0070.0080.0150.017
Greedy search (backward selection)630.0500.0300.0180.014
BOTAS (ours)630.0030.0060.0020.002

We can see from the results that the greedy search doesn’t find the optimum within the budget of 63 as often as BOTAS does, and that the found solution differs more from the optimum. In addition, the greedy search has to evaluate every remaining source to add a single one, so its budget limits how large a mixture it can reach. Within 63 evaluations it reaches at most 6 sources, which already excludes the 8-source optimum of RTE, and the limit tightens as the pool of sources grows. BOTAS has no such restriction, and it is not tied to the vanilla task vector method.

When we consider the number of evaluations required to reach within 1% of the optimum, the two methods are comparable on average. Across the four cells, BOTAS needs fewer evaluations in three and the greedy search in one. The difference lies in reliability. The greedy search fails to reach that band within the budget on 2 to 4 targets per cell, including RTE in every cell at gaps of 0.04 to 0.08. BOTAS misses it on a single target in a single cell, CB on Llama-3.2-1B at seed 123, and only by one to two of its 56 examples, since on CB a single example is worth more than the 1% band. The averages for every cell are in Table 7 in Appendix C.2, and the per-target values for Llama-3.2-1B at seed 42 are in the following table.

Target (Llama 42)Greedy searchBOTAS
MRPC832.7
BoolQ2821.5
RTE6333.6
CoLA227.9
CB4012.2
SciTail6328.5
STS-B3336.6
CR2413.5
RT158.8
MultiRC1617.8
COPA2333.6
PIQA216.2
Mean26.423.6

On the cost. We thank the reviewer for this correction. We have rephrased it in Section 5.1. A budget of 63 subset queries is 1.5 percent of the 4,083 candidate subsets, and each query evaluates the merged model at 40 scaling coefficients on the target evaluation set. The budget is therefore 63 subset queries and 2,520 merged-model evaluations per target task. However, the selection of the coefficients is independent of the selection method. The savings we claim are from subset queries against the exhaustive set of 4,083 subsets, which amounts to 163,320 merged-model evaluations.

AuthorsOpenReview ↗

Rebuttal (2/2)

Weakness 3: Selection and reporting use the same examples

The reviewer is right, and acting on it changes one of our claims. We have run the test they requested, and we report the full results.

We split each target's evaluation set into two disjoint halves by row index. Both halves were then swept exhaustively, all 4,083 subsets at all 40 coefficients, for the four targets that carry the claim. The subset and the coefficient are selected on the first half, and the comparison against full fine-tuning is reported on the second, which the selection never saw. Full fine-tuning and zero-shot are scored on exactly the same held-out rows, by the same exact-match rule, from the per-example predictions the single-task runs already produced. As a check on that procedure, our recomputed full-set scores reproduce every published reference value for these eight model-task pairs to four decimal places.

TargetMerged on held-out halfFFT on held-out halfDiffBest possible on that half
CB0.8570.750+0.1070.929
RTE0.8330.797+0.0360.862
RT0.9080.925-0.0170.931
COPA0.7400.780-0.0400.780

Two of the four wins survive, and two do not. CB and RTE still exceed full fine-tuning on examples that played no part in selecting the mixture, CB by a wide margin. Rotten Tomatoes and COPA fall below it. In the revised paper, the claim is reduced from four targets to two in the abstract, the results in Section 5.2, and the conclusion. It is reported as a held-out result in Table 5 rather than an in-sample one, and the Limitations in Appendix A now describe this held-out evaluation and its scope, which covers these four targets for Llama-3.2-1B at seed 42.

I appreciate the author’s additional experiments and revisions, which have addressed some of my concerns. I remain concerned, however, about the practical cost of reusing existing models for new tasks. Although BOTAS significantly reduces search costs compared to exhaustive evaluation, it still requires repeated assessment of candidate combinations using labeled samples for each new task. This limits the method's utility in scenarios where feedback for the target task is limited; therefore, I will maintain my current rating.

Reviewer_CXFVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors argue that model merging is underexplored for OOD target tasks, and provides a new benchmark to study zero-shot task selection merging methods. They also propose BOTAS, a discrete task selection procedure which uses Bayesian optimization, which outperforms existing heuristics and random search on the optimization task.

Critical strengths
  • The paper identifies an important problem in model merging that is not commonly addressed, where we combine existing models on unseen target tasks. The authors provide demonstrations that existing methods do not easily work in this new setting, which motivates their proposed method. The new benchmark also provides a useful starting point for future research in this area, although the scope is fairly limited.
  • BOTAS is intuitive, uses an easy-to-understand GP and EI BO loop, and outperforms the training-free selection and random baselines. I also appreciate the ablations on the feature representation, and the analysis of the limitations of training-free selection methods were interesting.
Critical weaknesses
  • The evaluation setting, while thorough and involving a lot of combinations, only involves one base model, Llama-3.2-1B-Instruct, with one set of source and target tasks. This makes it difficult to see if many of the conclusions in the paper would generalize to other models, sources, setups, etc. For example, the authors claim that many of the optimal mixtures are small (fewer than 5), and that merging all of the sources is not optimal. However, this may be due to the specific sample of source and target tasks selected, or the specific model-merging technique selected (task arithmetic). Although I appreciate the authors' efforts in undertaking the significant evaluation procedure, unfortunately it's unclear if the setup and thus the benchmark contribution is sufficient for a general demonstration of task selection for LLM merging.
  • Is BOTAS the best method for this? I think there should be many other discrete space selection regimes that would be important to include which are not merely heuristics like the training-free scores or purely random search. For example, greedy search seems like a natural choice, especially since many of the optimal mixtures have less than 4 selections so it should take <65 evaluations.
Questions for authors
  • Could you demonstrate that the main findings hold for one additional setting with a different base model? A similar result would help convince me that the findings generalize and allow me to raise my score. I don't think this needs the full sweep that your current benchmark runs, but maybe just comparing BOTAS against random search, some other search methods like greedy, and comparing the final results against zero-shot vs full-merge and finetuning would be sufficient.
  • Why is task arithmetic model merging is a useful starting point for this task? I think existing literature identify that this merging method starts breaking down as we include more and more tasks, and so that may be a primary reason that BOTAS is able to successfully identify the small optimal subset.
3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for an insightful review.

We address the weaknesses and questions in the following text and in the revised version of our paper.

Weakness 1 + Question 1: The evaluation covers a single base model and a single set of tasks

We have extended the study to a second model family and to additional fine-tuning seeds. The exhaustive sweep has been repeated in full for Qwen2.5-1.5B-Instruct at seeds 42 and 123, and for Llama-3.2-1B-Instruct at seed 123. The reviewer suggested that a partial comparison would be sufficient. We report the complete sweep in every cell instead, so that recovery against the exhaustive optimum is measured rather than approximated.

The conclusions of the paper hold in all four model+seed columns.

MethodBudgetLlama 42Llama 123Qwen 42Qwen 123
Optimum Found
Brute force4,083100%100%100%100%
Target Similarity4,0838%0%8%0%
Cosine similarity4,0830%0%0%0%
iso-SAR4,0830%0%0%0%
Random search637%2%4%2%
Greedy search (backward selection)638%25%8%17%
Greedy search (forward selection)6358%42%50%42%
BOTAS (ours)6383%74%68%65%
Diff to Optimum
Brute force4,0830.0000.0000.0000.000
Target Similarity4,0830.1470.1640.1030.096
Cosine similarity4,0830.1630.1480.0910.088
iso-SAR4,0830.1550.1530.0830.087
Random search630.0300.0360.0180.019
Greedy search (backward selection)630.0500.0300.0180.014
Greedy search (forward selection)630.0070.0080.0150.017
BOTAS (ours)630.0030.0060.0020.002

The table repeats Table 1 of the submission for each cell, with the same methods, budgets, and two measures, including the greedy search baseline we added in response to Weakness 2. In the revised paper, this exact table is Table 6 in Appendix C.2, both backbones are reported in Tables 2 and 3, and the per-seed results are in Appendix C.1. BOTAS recovers the exhaustive optimum in 65 to 83 percent of runs and reaches the 1 percent band in 20 to 32 evaluations. BOTAS is first on both measures in every cell; iso-SAR never recovers an optimum in any cell; and every training-free signal scoring all 4,083 mixtures is beaten by BOTAS, which scores 63.

The reviewer also asks whether the preference for small mixtures follows from the particular sources and targets. It does not follow from a single sample, but it is weaker than the submission states. The subset that wins a target has a median size of 3-5 sources out of the 12 available per cell, and never exceeds 8.

ModelSeedMedian kk <= 4Min kMax k
Llama-3.2-1B42310/1228
Llama-3.2-1B12347/1228
Qwen2.5-1.5B424.56/1228
Qwen2.5-1.5B12354/1237

We have restated the claim in its weaker form, as supported by the data, in the abstract, in Section 5.2, and in the conclusion. Optimal mixtures are small relative to the pool, and merging all available sources is never optimal, rather than the stronger statement that they typically involve fewer than 5 tasks, which holds for Llama but not for Qwen at seed 123.

AuthorsOpenReview ↗

Rebuttal (2/2)

Weakness 2: Whether other discrete selection regimes should be included

We agree and have added it to Table 1, to the experimental setup in Section 5.1, and to the results in Section 5.3, with the results for every cell in Appendix C.2. Greedy search searches over subset membership rather than ranking mixtures by a precomputed score, and both of its directions are reported at the same budget of 63 subset queries as BOTAS.

We evaluate all 12 single sources, keep the best, then repeatedly add the remaining source that most improves target accuracy, and stop when no further addition improves it. In the following table, we present a comparison with BOTAS. We can see from the results that the greedy search doesn’t find the optimum within the budget of 63 as often as BOTAS does, and that the found solution differs more from the optimum. In addition, the greedy search must evaluate each remaining source before adding any, so its budget limits how large a mixture it can reach. Within 63 evaluations, it reaches at most 6 sources, which already excludes the 8-source optimum of RTE, and the limit tightens as the pool of sources grows. BOTAS has no such restriction, and it is not tied to the vanilla task vector method.

The reviewer also notes that most optimal mixtures contain at most 4 sources. Restricting an exhaustive search to such mixtures would still leave 781 subsets, more than 12 times the budget of 63 that BOTAS uses, and it would miss the optimum on the 2 to 8 targets per cell whose winning mixture is larger.

MethodBudgetLlama 42Llama 123Qwen 42Qwen 123
Optimum Found
Greedy search (forward selection)6358%42%50%42%
Greedy search (backward selection)638%25%8%17%
BOTAS (ours)6383%74%68%65%
Diff to Optimum
Greedy search (forward selection)630.0070.0080.0150.017
Greedy search (backward selection)630.0500.0300.0180.014
BOTAS (ours)630.0030.0060.0020.002

When we consider the number of evaluations required to reach within 1% of the optimum, the two methods are comparable on average. Across the four cells, BOTAS needs fewer evaluations in three and the greedy search in one. The difference lies in reliability. The greedy search fails to reach that band within the budget for 2 to 4 targets per cell, including RTE in every cell, with gaps of 0.04 to 0.08. BOTAS misses it on a single target in a single cell, CB on Llama-3.2-1B at seed 123, and only by one to two of its 56 examples, since on CB a single example is worth more than the 1% band. The averages for every cell are in Table 7 in Appendix C.2, and the per-target values for Llama-3.2-1B at seed 42 are in the following table.

Target (Llama 42)Greedy searchBOTAS
MRPC832.7
BoolQ2821.5
RTE6333.6
CoLA227.9
CB4012.2
SciTail6328.5
STS-B3336.6
CR2413.5
RT158.8
MultiRC1617.8
COPA2333.6
PIQA216.2
Mean26.423.6

Question 2: Why task arithmetic is a useful starting point

Task arithmetic is the formulation that the merging literature we compare against builds on, and it is the one that admits an exhaustive ground truth. A single scaling coefficient shared across the included sources makes the space enumerable at 4,083 subsets by 40 coefficients, which allows a selection method to be scored against the true optimum rather than another heuristic.

The reviewer is correct that task arithmetic degrades as more sources are included, and we agree that this contributes to the small optimal subsets. However, the merged model is scored at every subset size, so the degradation is visible in the grid, and a selection method is credited only for finding the subset that actually transfers best under the merging operator in use.

I would like to thank the authors for their significant efforts during the rebuttal, and for providing many new empirical results. My W2 has been addressed.

I still have some unresolved concerns about the paper regarding W1: I noted that I do not think the results are generalizable because of the limited evaluation setting, specifically saying the claims "may be due to the specific sample of source and target tasks selected, or the specific model-merging technique selected (task arithmetic)." I appreciate that the authors have included new base model results, but I don't think that they have fundamentally addressed my overall concerns. I specified that I would have liked to see "one additional setting with a different base model", meaning both a different eval setting and a different base model (sorry if that was unclear). The fact that some of the conclusions needed to be updated given the new base model further supports my concerns about generalization.

In order for this paper to represent a significant contribution, its main empirical findings should hold true outside of purely using task arithmetic on this particular dataset. I will therefore maintain my rating.

Reviewer_3yaTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Given a pretrained base LLM with parameters θpre∈Rd\theta_{pre}\in\mathbb{R^d} and multiple checkpoints finetuned on different datasets for different tasks θt∈Rd\theta_t\in\mathbb{R^d} where tt references a particular task, model merging is the field of methods for mixing and matching weights from the fintetuned models to make a new point in weight space that yields a model that performs well on all the finetuned tasks.

Task arithmetic is a popular form of model merging, taking the delta in model weights between pretrained and finetune models, called the “task vector”, and adding/subtracting delta from the pretrained weights leads to new model weights that have improved/degraded performance on the tasks the weights we finetuned for.

Given a pretrain model and set of task vectors Recent model merging approaches have made new models by adding a up the task vectors, multiplying by a scaler weight λ\lambda and adding to the pretrained model. Effectively, from the pretrained model weights, the final model is a λ\lambda step size in the direction of the sum of finetuned weight deltas. The design space of merged models consists of two parameters, the first is a binary mask S∈0,1TS\in {0, 1}^T over the menu of TT finetuned models marking inclusion/exclusion in the sum of task vectors, and λ∈[0,1]\lambda\in[0, 1] marking how much the sum of task vectors is to be added to the pretrained weights.

In the particular case study investigated in this paper, there is on pre-trained model and 12 specific finetuned models, the goal is to find the best model mixture for a new held out 13th task, by searching over the space of inclusion/exclusion masks SS and scalar weight λ\lambda. The proposed search algorithm is a bi-level optimization; Bayesian optimization determines which models to includes (the binary mask in SS) and the scalar weight λ\lambda is determined by exhaustive grid search over 40 values between 0 and 1. There are 12 held out tasks on which the merged model can be tested (12 independent options for the 13th task) and the BOTAS algorithm uses held out task performance to choose the subset of finetuned model checkpoints and weighting.

For the purpose of benchmarking and for the wider community the authors evaluated the full exhaustive range of model subsets and scalar weights on all 12 held out tasks to provide a dataset of results for the wider community.

Critical strengths
  • high impact problem, efficiently finding models reduces compute and speeds up model development.
  • BO is a good solution.
  • dataset of results is a valuable resource for the community (with caveats)
  • builds on the seminal work of [2] in model merging.

[2] Editing models with Task Arithmetic, Ilharco et.al. ICLR 2023 https://arxiv.org/pdf/2212.04089

Critical weaknesses

My main concern is with the design of the search space for the optimization algorithm to search over.

  • Problem Formulation Eq(2) shows that the way this paper forms the fundamental problem is bilevel optimization mimicking the oringal work on Task Arithmetic [2]. One could frame the problem as continuous mixtures of individual task vectors λ1,…,λ12∈[0,1]\lambda_1,…,\lambda_{12} \in [0, 1] and the merged weights are θmerged=θpre+∑tλtτt\theta_{merged} = \theta_{pre} + \sum_t \lambda_t\tau_t (which is a strict super set of the (S,λ)(S, \lambda) parameterization) which is closer to prior BO work on model merging [1]. This directly leads to mty next two comments.

  • BOTAS Algorithm is nice and simple but lack novelty or interesting extensions of BO to this use case. The BOTAS algorithm treats this as a bilevel formulation where

    • SS inclusion/exclusion is outer level of optimization performed by standard Bayesian optimisation applied directly to binary vectors
    • λ\lambda the single weight applied to the sum of models is the inner optimization uses a 1D grid search of 40 points (though any method may be used, even 1D BO).
    • applying vanilla BO to the space of continuous per task weights would be arguably much simpler and each evaluation would avoid an inner optimisation over λ\lambda and hence be 40x cheaper (but also less informative).
  • Dataset Usefulness given above

    • I do believe the dataset to be very useful, but it is specific to the formulation provided: all task weights are treated equally, it cannot be used to benchmark any other problem formulation.

I really like the paper, a good problem, a nice simple algorithm. However the BO is off-the-shelf and the inner grid search seems suboptimal, and the model merging dataset is valuable but has some limitations too, I am not sure this paper is sufficient for acceptance to ICLR.

[1] Dynamic Fisher-weighted Model Merging via {B}ayesian Optimization, Lee at.al. NAACL 2025, https://arxiv.org/pdf/2504.18992

[2] Editing models with Task Arithmetic, Ilharco et.al. ICLR 2023 https://arxiv.org/pdf/2212.04089

Questions for authors
  • Can this search space be defined as task specific continuous weights?
  • What is the advantage of the BOTAS designed search space?
  • Can BO over continuous weightings be added as a baseline? I assume not as it cannot be applied with this offline dataset.
4 comments

We thank the reviewer for an insightful review.

We address the weaknesses and questions in the following text and in the revised version of our paper.

Weakness 1 + Question 1: The problem formulation and the design of the search space

We agree with the reviewer that the search space can be defined over task-specific continuous weights, that these are a strict superset of the formulation we study, and that the bi-level form we adopt follows the original task arithmetic work. Our formulation is recovered from the continuous one by tying each included weight to a single shared value. We chose it deliberately, for a reason specific to the contribution we are making.

The contribution of the paper is an exhaustive ground truth and a selection method validated against it. A space of 4,083 subsets by 40 shared coefficients can be enumerated, and we enumerated it 4 times (2 models times 2 source training seeds). A space of continuous per-source weights cannot be enumerated, so a method searching it can be compared against other methods but not against the true optimum. Our central measurement, the share of runs that recover the exhaustive optimum within a budget, has no counterpart for continuous per-source weights.

Weakness 2 + Question 2: The Bayesian optimization is standard, and the advantage of the search space it is defined over

We agree that the BOTAS components are standard and we also do not claim a new acquisition function or a new kernel. The claim we make is that the discrete choice of which checkpoints to merge, for a target the checkpoints were never built for, has not been posed as a search problem validated against exhaustive ground truth, and that a standard Gaussian process over binary subset indicators solves it far better than the geometric criteria the merging literature currently relies on. The negative result on the subspace alignment ratio is part of that contribution and would not be visible without the exhaustive grid.

For BOTAS itself, the reviewer is right that the inner loop need not be a grid. Nothing in the method depends on it. A deployed selector could use any 1D optimizer, and the cost of one subset query would fall accordingly. We believe that this could be an interesting future upgrade to the BOTAS method, and we have added it as future work to the Limitations in Appendix A.

As for the advantage of the search space that BOTAS is defined over, three things follow from the shared coefficient. Only the first concerns search.

The space is enumerable. 4,083 subsets by 40 coefficients yield 163,320 merged models per cell, each scored on all 12 target tasks, which we ran, so a selection method can be evaluated against the true optimum rather than a competing heuristic. The share of runs that recover that optimum is the paper's central number, and it has no counterpart in a space that cannot be enumerated.

The sweep is affordable. One summed task vector is built per subset and rescaled in place across all 40 coefficients, so the inner sweep costs 40 evaluation passes and no further merges. Per-source continuous weights remove that reuse, since every evaluation requires a fresh weighted sum over the included task vectors. The reviewer is nevertheless right that one continuous evaluation is far cheaper than one subset query of BOTAS, which spans 40 coefficients, and our answer to Question 3 below shows that a continuous search can reach a strong mixture in fewer merged-model evaluations. The advantage of the shared coefficient therefore lies in the enumerable ground truth rather than in a lower search cost.

The grid is reusable. Because every coefficient is recorded rather than only the best one, the released table supports analyses we did not anticipate, including the distribution of the selected coefficient and the behavior of the training-free signals at a fixed coefficient.

Weakness 3: The released grid is specific to this formulation

We accept this and have stated the scope explicitly in the Limitations in Appendix A rather than leave it implied. The grid covers task arithmetic using a single shared scaling coefficient and cannot score methods that assign per-source weights or use a different merging operator. Within that scope, it serves its stated purpose: to allow any method that selects a subset to be compared against the true optimum without repeating 163,320 merged-model evaluations.

Question 3: Whether BO over continuous weightings can be added as a baseline

The reviewer anticipates the difficulty correctly. It cannot be scored against the released grid because the grid only contains the shared-coefficient parameterization. Adding it requires live merging and evaluation rather than lookup. We consider this a reasonable request for a comparison.

We are running this baseline on three targets and will report the results in a follow-up response during the discussion period, and we will add them to the paper.

AuthorsOpenReview ↗

BO over continuous weightings baseline

We have run this baseline. It uses the same Gaussian process and Expected Improvement as BOTAS, but searches a separate weight for each of the 12 sources in the box [0, 0.5]^12, and starts from 12 points drawn from the BOTAS space. Each candidate is merged and evaluated live. We ran it on Llama-3.2-1B-Instruct at seed 42 for three targets, with a single BO seed and a budget of 300 merged-model evaluations.

TargetBest within 63 evaluationsBest within 300 evaluationsOptimum of our grid
CB0.8750.8930.893
RTE0.8200.8410.841
COPA0.7200.7500.770

Within 300 evaluations, continuous BO reaches the optimum of our grid on CB and RTE and comes within 0.020 of it on COPA. Measured in merged-model evaluations, this is fewer than the 2,520 that BOTAS spends on its 63 subset queries. Searching per-source weights directly is therefore a competitive alternative, and its results can still be checked against our grid, although the continuous space itself cannot be enumerated.

Reviewer_3yaTOpenReview ↗

Thank you for the response.

I understand the points,

We need to discretize the search space somehow and the (S,λ)(S, \lambda) parametertization means S is finite set and λ\lambda can be discretized easily enough. I am not sure I agree that this requirement implies this parameterization though.

One could also use a latin hypercube design over the continuous space of mixtures [0,1]12[0, 1]^{12} and BO can be performed over the discrete set within the continuous space (we just replace the acquisition function search space with the discretization

  • xt+1=argmaxx∈Xα(x)x^{t+1} = argmax_{x\in\mathcal{X}}\alpha(x)

gets replaced with

  • xt+1=argmaxx∈Xdiscα(x)x^{t+1} = argmax_{x\in\mathcal{X}_{disc}}\alpha(x) )

as I believe has been common practice in the auto-ML community and other offline hyperparameter tuning benchmarks.

Thank you for running the GP baseline, do I understand the result correctly in that it reaches near optimal performance in a tiny fraction of the number of evaluations? It is hard to asses the significance of these differences in score, what is the lowest over the grid and the median?

I will keep my score as is.

AuthorsOpenReview ↗

Thank you.

We thank the reviewer for the answer and follow-up questions. Thank you for the question. Your reading is correct. With 300 merged-model evaluations, which is 0.18% of the 163,320 models in our grid, continuous BO achieves the best accuracy across the whole grid on CB and RTE, and comes within 0.020 of it on COPA. We will consider adding it as a separate variant of BOTAS and including it in a future iteration of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.