- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies selecting subsets of existing task models to merge for a different target task. It evaluates 4,083 subsets of 12 sources across 40 shared scaling coefficients and 12 targets. BOTAS uses binary subset indicators, a standard Gaussian process, and Expected Improvement with labeled target evaluations. Within 63 subset queries, it reaches the exhaustive optimum in 83% of runs, versus 7% for random search (pp. 4–6; p. 20, Table 12).
Critical strengths
- Studying transfer beyond the source tasks is useful for reusing existing models. The exhaustive table allows selected mixtures to be compared directly with the optimum and makes these expensive evaluations available for subsequent studies.
- GP/EI suits this expensive, discrete black-box problem, and the recovery-rate improvement over random search is substantial. Geometric scores show some correlation with performance yet fail to reliably select the best mixture, cautioning against using parameter similarity directly for selection.
Critical weaknesses
- The discussion on p. 3 already covers source selection and subset selection before merging, making transfer through a selected combination an important distinction. Yet single-source candidates are excluded (p. 4), and their target performance is not reported. The results therefore do not establish how much is gained by combining sources. This leaves the distinction from existing source selection unclear and weakens an interpretation based on useful source complementarity.
- Standard GP/EI is a reasonable choice, but Table 4 and Appendix omit forward-addition or backward-removal searches with matched feedback and budgets. Beating random search does not establish BO's additional benefit. Cost reporting also needs correction: each subset query in Eq. 5 evaluates 40 coefficients, so 63 queries require 2,520 target-set evaluations. The relative saving in subset queries remains valid, but it should not be described as only 63 model evaluations.
- The use of target labels is explicit and retrospective lookup validly evaluates search efficiency. However, selection and final reporting use the same examples. The four reported wins over target fine-tuning therefore do not establish generalization to independent examples. With only 56 examples for CB and 100 for COPA, selection bias is particularly relevant. Fixed-table optimization success and practical transfer gains should be reported separately.
Questions for authors
-
Compared with the best single-source transfer, what evidence shows that combining sources itself provides a benefit? How does this distinguish the contribution from the source-selection work discussed on p. 3?
-
How do forward-addition or backward-removal searches compare with BOTAS under the same 63-query budget? Could the comparison also report the model evaluations required by the inner coefficient sweeps?
-
Do the four claimed wins over target fine-tuning on p. 7 persist on target examples excluded from subset and coefficient selection?
3 comments
Rebuttal (2/2)
Weakness 3: Selection and reporting use the same examples
The reviewer is right, and acting on it changes one of our claims. We have run the test they requested, and we report the full results.
We split each target's evaluation set into two disjoint halves by row index. Both halves were then swept exhaustively, all 4,083 subsets at all 40 coefficients, for the four targets that carry the claim. The subset and the coefficient are selected on the first half, and the comparison against full fine-tuning is reported on the second, which the selection never saw. Full fine-tuning and zero-shot are scored on exactly the same held-out rows, by the same exact-match rule, from the per-example predictions the single-task runs already produced. As a check on that procedure, our recomputed full-set scores reproduce every published reference value for these eight model-task pairs to four decimal places.
| Target | Merged on held-out half | FFT on held-out half | Diff | Best possible on that half |
|---|---|---|---|---|
| CB | 0.857 | 0.750 | +0.107 | 0.929 |
| RTE | 0.833 | 0.797 | +0.036 | 0.862 |
| RT | 0.908 | 0.925 | -0.017 | 0.931 |
| COPA | 0.740 | 0.780 | -0.040 | 0.780 |
Two of the four wins survive, and two do not. CB and RTE still exceed full fine-tuning on examples that played no part in selecting the mixture, CB by a wide margin. Rotten Tomatoes and COPA fall below it. In the revised paper, the claim is reduced from four targets to two in the abstract, the results in Section 5.2, and the conclusion. It is reported as a held-out result in Table 5 rather than an in-sample one, and the Limitations in Appendix A now describe this held-out evaluation and its scope, which covers these four targets for Llama-3.2-1B at seed 42.
I appreciate the author’s additional experiments and revisions, which have addressed some of my concerns. I remain concerned, however, about the practical cost of reusing existing models for new tasks. Although BOTAS significantly reduces search costs compared to exhaustive evaluation, it still requires repeated assessment of candidate combinations using labeled samples for each new task. This limits the method's utility in scenarios where feedback for the target task is limited; therefore, I will maintain my current rating.
Rebuttal (1/2)
We thank the reviewer for an insightful review.
We address the weaknesses and questions in the following text and in the revised version of our paper.
Weakness 1 + Question 1: Single-source candidates are excluded
We have run the 12 single sources over the same 40 coefficients and the same 12 targets as the main grid, in every cell.
Merging a selected subset beats the best single source on 10 to 12 of the 12 targets, depending on the seed and model, with mean gains ranging from 0.049 to 0.075, up to 0.209. The exceptions are confined to Llama, namely CoLA in both of its cells and COPA at seed 123, and on Qwen every target prefers a mixture. The gain is therefore not an artifact of one model or one seed, and the targets where a single source wins are few enough to name. CoLA is the one target that no source family covers, which is consistent with the reading we already give it in the paper.
We have added this comparison to Section 5.2 (Table 4), with the values for every cell in Appendix C.2 (Table 9), as it provides evidence that the problem we pose is not the source-selection problem in the prior work the reviewer cites.
Weakness 2 + Question 2: A discrete search baseline that is not random, and the reported cost
On the baseline. We have added greedy search over subset membership, scored retrospectively against the same grids and reported at the same budget of 63 subset queries. It appears in Table 1, in Sections 5.1 and 5.3, and for every cell in Appendix C.2.
Forward selection starts from the best single source and repeatedly adds the source that most improves target accuracy. Backward selection starts from all 12 sources and repeatedly drops one. Both stop when no single move improves accuracy. The results of the greedy forward and backward selection are shown in the following table:
We can see from the results that the greedy search doesn’t find the optimum within the budget of 63 as often as BOTAS does, and that the found solution differs more from the optimum. In addition, the greedy search has to evaluate every remaining source to add a single one, so its budget limits how large a mixture it can reach. Within 63 evaluations it reaches at most 6 sources, which already excludes the 8-source optimum of RTE, and the limit tightens as the pool of sources grows. BOTAS has no such restriction, and it is not tied to the vanilla task vector method.
When we consider the number of evaluations required to reach within 1% of the optimum, the two methods are comparable on average. Across the four cells, BOTAS needs fewer evaluations in three and the greedy search in one. The difference lies in reliability. The greedy search fails to reach that band within the budget on 2 to 4 targets per cell, including RTE in every cell at gaps of 0.04 to 0.08. BOTAS misses it on a single target in a single cell, CB on Llama-3.2-1B at seed 123, and only by one to two of its 56 examples, since on CB a single example is worth more than the 1% band. The averages for every cell are in Table 7 in Appendix C.2, and the per-target values for Llama-3.2-1B at seed 42 are in the following table.
On the cost. We thank the reviewer for this correction. We have rephrased it in Section 5.1. A budget of 63 subset queries is 1.5 percent of the 4,083 candidate subsets, and each query evaluates the merged model at 40 scaling coefficients on the target evaluation set. The budget is therefore 63 subset queries and 2,520 merged-model evaluations per target task. However, the selection of the coefficients is independent of the selection method. The savings we claim are from subset queries against the exhaustive set of 4,083 subsets, which amounts to 163,320 merged-model evaluations.