acceptodds
Under review as a conference paper at ICLR 2027

You're Hired: Strategic Model Selection for LLM Collaboration

Abstract

While multi-agent and model collaboration algorithms gain traction to combine the strengths of diverse Large Language Models (LLMs), existing systems remain bottlenecked on pre-defined and hand-crafted model pools. In this work, we investigate the problem of model selection in multi-LLM systems. We propose and systematically evaluate a taxonomy of 9 selection algorithms ranging from diversity of model descriptions, capability-aware behavioral diversity, and LLM-based recruiters. We conduct extensive experiments across two candidate pools of 10 and 32 models, deployed in four model collaboration algorithms, and evaluated across tasks spanning math, coding, QA, and reasoning. Results demonstrate that successful selection algorithms greatly outperform random or heuristics-based teams such as merely selecting the models with top individual performance, by up to 36.1% across settings. Specifically, capability- and training-based selection strategies alleviate selection variance and achieve the best performance, which we recommend to employ before deploying real-world multi-LLM systems. Further analysis reveals that larger candidate pools pose greater challenges to shallow selection heuristics, while algorithms grounded in interacting with candidate models and understanding model capability robustly filter out misaligned, unsafe models, as well as generalizing to novel, out-of-distribution tasks. Together, we establish that principled and informed team selection is critical and present strong model selection algorithms for assembling effective multi-LLM systems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-llm systems, model selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ahNoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to select models for an LLM team. The authors compare methods based on model descriptions, measured capabilities, diversity, and LLM-based interviews. Experiments use two model pools and four collaboration methods, with further tests on larger pools, malicious models, and generalization to new tasks.

Critical strengths
  • Selecting models that work well together is a practical problem, and the paper offers a useful comparison of several approaches.

  • The experiments cover multiple collaboration methods. The tests involving malicious models and unseen tasks are also interesting.

Critical weaknesses
  • My main concern is how this work differs from existing ensemble-selection methods. Selecting complementary models from a candidate pool is already a well-studied problem, including greedy selection based on validation performance. What limitation of these approaches do the proposed strategies address?

  • The variance comparison in Section 5 does not seem fair. For random selection, the variation is across different teams. For the other methods, it comes from resampling test examples for a fixed selected team. These measure different sources of uncertainty, so smaller error bars do not establish that the selection method is more stable.

  • The diversity methods use different starting points. IRT and Idiosyncrasies begin with the two strongest models, while description diversity does not. It is therefore hard to tell how much of the gain comes from the diversity measure and how much comes from including strong models at the start.

Questions for authors
  • Could you add a greedy baseline that selects models based on the team’s performance on the development set?

  • Would the comparison change if all diversity methods started with the same two strongest models? Adding two random models to those two would also be a useful baseline.

  • Could you repeat selection with different development samples and training seeds to check how stable the selected teams and their performance are?

  • What is the cost of selection, including collecting responses and training the selectors? SFT scores all 35,960 teams in Pool 2. How practical would this be with a much larger candidate pool?

5 comments

Dear Reviewer ahNo,

Thank you for the detailed review. Revisions in the updated manuscript are highlighted in blue.

My main concern is how this work differs from existing ensemble-selection methods. Selecting complementary models from a candidate pool is already a well-studied problem, including greedy selection based on validation performance. What limitations of these approaches do the proposed strategies address?

Could you add a greedy baseline that selects models based on the team’s performance on the development set?

Response: We thank the reviewer for raising this important baseline. We implemented greedy forward selection on Pool 1, starting from the strongest solo model and sequentially adding candidates based on validation performance until reaching a team size of four. We evaluated this baseline across Prompt Routing and LLM-Blender on GSM8K and TruthfulQA.

Setting (Pool 1)Greedy Test ScoreRank among Table 2 SelectorsBest Proposed Selector
Prompt Routing / GSM8K55.511 / 1764.9 (Perf. Profile (least)) / 63.4 (Desc. (most))
LLM-Blender / GSM8K74.21 / 1773.9 (Top Solo) / 72.7 (Nested (least))
Prompt Routing / TQA57.75 / 1760.0 (Desc. (most) & Perf. Profile (least))
LLM-Blender / TQA58.56 / 1762.1 (Desc. (most)) / 58.7 (Perf. Profile (least))

Greedy selection underperforms our best selectors in 3 of 4 settings, by up to 14.5% relatively (9.4 points), while marginally achieving the best result on Blender/GSM8K. Moreover, greedy requires 24 and 90 collaborative evaluations per setting in Pools 1 and 2, respectively, whereas our representation-based selectors reuse candidate-level information across collaboration mechanisms, avoiding repeated team-level evaluations and inference costs. These results highlight the effectiveness and efficiency of our proposed suite of methods. We included this comparison in the revised Appendix I Greedy Ensemble Selection.

The variance comparison in Section 5 does not seem fair. For random selection, the variation is across different teams. For the other methods, it comes from resampling test examples for a fixed selected team. These measure different sources of uncertainty, so smaller error bars do not establish that the selection method is more stable.

Could you repeat selection with different development samples and training seeds to check how stable the selected teams and their performance are?

Response: To evaluate whether the stability advantage holds when repeating the full selection pipeline, we ran two additional experiments on Pool 1 across Prompt Routing and LLM-Blender on GSM8K and TruthfulQA (TQA), each with five replicates: (1) Dev Data Sensitivity, where we resampled selection data and refit or retrained the applicable selectors from scratch with a fixed seed; and (2) Training and Generation Seed Sensitivity, where we fixed selection data and varied initialization, data-ordering, and candidate sampling/judging seeds. We report the number of distinct teams selected and test accuracy (mean ± SD).

Exp 1: Resampled Selection Data

SettingsUnique TeamsPR / GSM8KPR / TQABlender / GSM8KBlender / TQA
Random Baseline5/550.5 ± 10.048.0 ± 5.759.8 ± 7.148.0 ± 5.4
Single Best Model (lima)1/565.6 ± 0.059.3 ± 0.065.6 ± 0.059.3 ± 0.0
Description (most)1/563.4 ± 0.060.0 ± 0.048.5 ± 0.062.1 ± 0.0
LLM Prompt1/540.1 ± 0.043.9 ± 0.070.0 ± 0.048.9 ± 0.0
Idiosyncrasies (least)1/556.5 ± 0.040.5 ± 0.067.6 ± 0.052.7 ± 0.0
Combined (most)1/549.8 ± 0.052.3 ± 0.045.3 ± 0.055.6 ± 0.0
Nested (most)2/561.7 ± 0.754.7 ± 2.071.1 ± 0.255.9 ± 1.8
Perf. Profile (least)2/563.3 ± 3.659.7 ± 0.651.6 ± 3.358.1 ± 1.2
IRT (most)2/559.8 ± 3.145.7 ± 3.367.0 ± 0.953.8 ± 2.0
SFT Classifier2/545.7 ± 8.547.7 ± 7.860.2 ± 9.654.9 ± 2.8
Top Solo Score3/560.1 ± 3.645.1 ± 2.773.3 ± 1.451.7 ± 1.9

Exp 2: Training / Sampling Seeds

SettingsUnique TeamsPR / GSM8KPR / TQABlender / GSM8KBlender / TQA
Random Baseline5/550.5 ± 10.048.0 ± 5.759.8 ± 7.148.0 ± 5.4
Description (most)1/563.4 ± 0.060.0 ± 0.048.5 ± 0.062.1 ± 0.0
LLM Prompt1/540.1 ± 0.043.9 ± 0.070.0 ± 0.048.9 ± 0.0
Idiosyncrasies (least)1/556.5 ± 0.040.5 ± 0.067.6 ± 0.052.7 ± 0.0
Combined (most)1/563.1 ± 0.049.9 ± 0.054.7 ± 0.052.3 ± 0.0
IRT (most)2/538.9 ± 0.749.5 ± 3.868.0 ± 1.059.5 ± 3.0
SFT Classifier2/555.8 ± 6.954.8 ± 8.051.2 ± 3.053.8 ± 1.0
Agentic Top-K2/539.4 ± 1.649.6 ± 0.450.4 ± 11.449.7 ± 0.3

Note: Parameter-free selectors [Single Best, Top Solo, Perf. Profile, Nested] are N/A in Exp 2, and Agentic Top-K uses no item data and is N/A in Exp 1. SD = 0.0 means all 5 replicates picked the identical team, either by construction [Description, LLM Prompt] or empirically [Idiosyncrasies, Combined, Single Best].

The results demonstrate substantial stability for several behavioral selectors. While Random produces 5/5 distinct teams, Idiosyncrasies and Combined select 1/5 distinct teams within each perturbation experiment, and IRT selects 2/5. Under development-data resampling, Nested and Perf. Profile also selects only 2/5 distinct teams, with relatively small performance variations. In contrast, SFT Classifier and Agentic Top-K exhibit larger fluctuations (SD up to 9.6 and 11.4 points, respectively). These findings provide additional evidence that several behavioral and capability-based selection strategies are reproducible across development-data perturbations and random seeds. We included these results in the revised Appendix H Selection Stability Across Multiple Seeds.

The diversity methods use different starting points. IRT and Idiosyncrasies begin with the two strongest models, while description diversity does not. It is therefore hard to tell how much of the gain comes from the diversity measure and how much comes from including strong models at the start.

Would the comparison change if all diversity methods started with the same two strongest models? Adding two random models to those two would also be a useful baseline.

Response: To isolate the contribution of our behavioral representations from the ability seeding, we conduct an ablation on Pool 1. Specifically, we fix the same two Top2 seed models used in IRT and Idiosyncrasies, and random select remaining two models over 5 seeds (mean ± std). Values in parentheses denote the relative difference from each method's respective control baseline.

SettingRouting / TQARouting / GSM8KBlender / TQABlender / GSM8K
Unconstrained Random48.0 ± 5.750.5 ± 10.048.0 ± 5.459.8 ± 7.1
IRT Control (Top-2 + Random-2)51.9 ± 5.052.5 ± 8.855.9 ± 2.467.4 ± 2.5
IRT (most-diverse)53.6 (+1.7)38.2 (−14.3)56.2 (+0.3)69.1 (+1.7)
IRT (least-diverse)56.9 (+5.0)60.6 (+8.1)55.6 (−0.3)61.8 (−5.6)
Idiosyncrasies Control (Top-2 + Random-2)46.9 ± 4.956.0 ± 5.752.5 ± 1.269.3 ± 3.3
Idiosyncrasies (most-diverse)40.7 (−6.2)51.1 (−4.9)51.5 (−1.0)70.9 (+1.6)
Idiosyncrasies (least-diverse)40.5 (−6.4)56.5 (+0.5)52.7 (+0.2)67.6 (−1.7)

The results show that ability-based seeding and behavioral diversity can provide complementary benefits. Compared with unconstrained random selection, Top-2 seeding improves performance by up to 16.5% relatively. Building on the same Top-2 seeds, behavioral selection further improves performance by up to 15.4% for IRT and 2.3% for Idiosyncrasies. These findings highlight the complementary roles of model ability and behavioral diversity in team selection. We included these ablations in the revised Appendix G Additional Ablations.

What is the cost of selection, including collecting responses and training the selectors? SFT scores all 35,960 teams in Pool 2. How practical would this be with a much larger candidate pool?

Response: We thank the reviewer for this practical question. We collected the actual GPU-hours to quantify the costs of response collection, selector training, and team search across Pool 1 (N=10) and Pool 2 (N=32). GPU-h

Selection StageMethodPool 1Pool 2Scaling
Per-model responsesIRT, Combined, Nested, Idiosyncrasies25.3 GPU-h≈66.0 GPU-hO(N))
Perf.-profilePerf. Profile20.2 GPU-h72.8 GPU-hO(N)
Solo benchmark scoresTop Solo, Single Best13.6 GPU-h30.7 GPU-hO(N)
Agent interviewsAgentic Top-K0.3 GPU-h–O(N)
Selector trainingIRT, Idiosyncrasies, SFT<0.1 / 0.7 / 0.8 GPU-h<0.1 / 1.5 / 0 (reused) GPU-hO(1)
SFT training labelsSFT Classifier≈977 GPU-h0 (reused) GPU-hO(1)
SFT Team searchSFT Classifier3.5 min (840 passes)10 GPU-h(N4)∝N4\binom{N}{4} \propto N^4 (exhaustive) or O(N)O(N) (beam/pre-filtered)
Other Team searchOther selectors< 5 sec (CPU)< 5 sec (CPU)Microseconds per candidate team

Our behavioral selectors scale linearly in candidate profiling, requiring approximately 20–73 GPU-h for individual response collection, under 1.5 GPU-h for training, and seconds for team search. For SFT Classifier, training-label collection dominates the cost (≈977 GPU-h), but the trained selector transfers zero-shot from Pool 1 to Pool 2 without additional labeling or retraining. While exhaustive SFT search scales as (O(N^4)), pre-filtering to 20 candidates reduces the search space to 4,845 teams, offering a practical scaling option. We included this analysis in the Appendix J Selection Costs and Scalability.

Reviewer_cQutOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Choosing collaborators is the central problem here: which models should enter a team before routing, refinement, weight merging, or output fusion begins? The paper compares selectors based on descriptions, behavioral representations, capability filtering, and LLM recruiters. Evaluation selects four-model teams from two pools on GSM8K, TruthfulQA, and MBPP, with additional robustness and transfer experiments. Although the second pool derives from heterogeneous source models, its deployed checkpoints are all Qwen2.5-7B distillations. The main contribution is an empirical study of team composition and its dependence on the collaboration mechanism.

Critical strengths
  1. Evaluation across four collaboration mechanisms shows that team composition matters and that effective selection depends on how the models collaborate.
  2. The SFT recruiter shows promising transfer across candidate pools and unseen tasks, exceeding Random’s mean performance in all four reported OOD settings.
Critical weaknesses
  1. The uncertainty reporting weakens the reliability claim. The first pool’s GSM8K Single Best Model should have a 95% bootstrap half-width near three percentage points, but Table 2 reports ±1.5, roughly one standard error. Also, Random’s intervals measure variation across teams, while the others measure test-item uncertainty for a fixed team. Comparing these widths does not establish more reliable selection.
  2. Behavioral diversity is confounded with capability constraints: IRT and Idiosyncrasies start with the strongest pair, while Nested removes the weaker half of the pool. Top Solo Score does not isolate these effects. Random completion of the same seeds, or random selection within the same filtered pool, is needed to establish what diversity adds beyond retaining strong candidates.
Questions for authors
  1. Are the uncertainties in Table 2 standard errors or 95% intervals? Can you reconcile the GSM8K example above with the stated bootstrap procedure?
  2. Does the stability advantage remain when the complete selection pipeline is repeated across development samples and training or generation seeds, with equivalent uncertainty reporting for Random?
  3. How do IRT, Idiosyncrasies, and Nested compare against controls that preserve their capability seeds or filters but replace diversity selection with random completion?
4 comments

Dear Reviewer cQut,

Thank you for the detailed review. Revisions in the updated manuscript are highlighted in blue.

The uncertainty reporting weakens the reliability claim. The first pool’s GSM8K Single Best Model should have a 95% bootstrap half-width near three percentage points, but Table 2 reports ±1.5, roughly one standard error. Also, Random’s intervals measure variation across teams, while the others measure test-item uncertainty for a fixed team. Comparing these widths does not establish more reliable selection.

Response: Thank you for identifying this. We checked Table 2: the ± values indeed represent standard deviations, rather than 95% confidence intervals. We corrected the uncertainty descriptions in the revised Section 4, Experiment Details. The reported numerical values remain unchanged.

Behavioral diversity is confounded with capability constraints: IRT and Idiosyncrasies start with the strongest pair, while Nested removes the weaker half of the pool. Top Solo Score does not isolate these effects. Random completion of the same seeds, or random selection within the same filtered pool, is needed to establish what diversity adds beyond retaining strong candidates.

How do IRT, Idiosyncrasies, and Nested compare against controls that preserve their capability seeds or filters but replace diversity selection with random completion?

Response: To isolate the contribution of our behavioral representations from the ability seeding, we conduct an ablation on Pool 1. Specifically, we fix the same two Top2 seed models used in IRT and Idiosyncrasies, and randomly select the remaining two models over 5 seeds (mean ± std). Similarly, for Nested, we filter to the same top half (Top-6) candidate models and randomly sample four models over 5 seeds. Values in parentheses denote the relative difference from each method's respective control baseline.

Method / Control (Pool 1)Routing / TQARouting / GSM8KBlender / TQABlender / GSM8K
Unconstrained Random (Table 1)48.0 ± 5.750.5 ± 10.048.0 ± 5.459.8 ± 7.1
IRT Control (Same Top-2 + 2 Random)51.9 ± 5.052.5 ± 8.855.9 ± 2.467.4 ± 2.5
IRT (most-diverse)53.6 (+1.7)38.2 (−14.3)56.2 (+0.3)69.1 (+1.7)
IRT (least-diverse)56.9 (+5.0)60.6 (+8.1)55.6 (−0.3)61.8 (−5.6)
Idiosyncrasies Control (Same Top-2 + 2 Random)46.9 ± 4.956.0 ± 5.752.5 ± 1.269.3 ± 3.3
Idiosyncrasies (most-diverse)40.7 (−6.2)51.1 (−4.9)51.5 (−1.0)70.9 (+1.6)
Idiosyncrasies (least-diverse)40.5 (−6.4)56.5 (+0.5)52.7 (+0.2)67.6 (−1.7)
Nested Control (Random 4 of Same Top-6)48.1 ± 4.556.8 ± 3.753.1 ± 4.970.9 ± 1.2
Nested (most-diverse)53.8 (+5.7)61.4 (+4.6)55.1 (+2.0)71.2 (+0.3)
Nested (least-diverse)48.3 (+0.2)63.2 (+6.4)55.1 (+2.0)72.7 (+1.8)

The results show that ability-based seeding and behavioral diversity can provide complementary benefits. Compared with unconstrained random selection, Top-2 seeding improves performance by up to 16.5% relatively, while Top-6 candidate filtering (Nested Control) yields gains of up to 18.6% relatively. Building on these constrained seeds, behavioral selection further improves performance by up to 15.4% for IRT, 2.3% for Idiosyncrasies, and 11.9% for Nested. These findings highlight the complementary roles of model ability and behavioral diversity in team selection. We included these ablations in the revised paper Appendix G Additional Ablations.

Are the uncertainties in Table 2 standard errors or 95% intervals? Can you reconcile the GSM8K example above with the stated bootstrap procedure?

Response: They are standard errors of accuracies across 10,000 bootstrap resamples. For GSM8K, this gives ±1.5 percentage points. A 95% bootstrap interval would indeed be approximately ±3 points, as you noted. We mistakenly labeled the former as the latter. We corrected the variance estimation description in the revised Section 4, Experiment Details. The reported numerical values remain unchanged.

Does the stability advantage remain when the complete selection pipeline is repeated across development samples and training or generation seeds, with equivalent uncertainty reporting for Random?

Response: To evaluate whether the stability advantage holds when repeating the full selection pipeline, we ran two additional experiments on Pool 1 across Prompt Routing and LLM-Blender on GSM8K and TruthfulQA (TQA), each with five replicates: (1) Dev Data Sensitivity, where we resampled selection data and refit or retrained the applicable selectors from scratch with a fixed seed; and (2) Training and Generation Seed Sensitivity, where we fixed selection data and varied initialization, data-ordering, and candidate sampling/judging seeds. We report the number of distinct teams selected and test accuracy (mean ± SD).

Exp 1: Resampled Selection Data:

SettingsUnique TeamsPR / GSM8KPR / TQABlender / GSM8KBlender / TQA
Random Baseline (5 draws)5/550.5 ± 10.048.0 ± 5.759.8 ± 7.148.0 ± 5.4
Single Best Model (lima)1/565.6 ± 0.059.3 ± 0.065.6 ± 0.059.3 ± 0.0
Description (most)1/563.4 ± 0.060.0 ± 0.048.5 ± 0.062.1 ± 0.0
LLM Prompt1/540.1 ± 0.043.9 ± 0.070.0 ± 0.048.9 ± 0.0
Idiosyncrasies (least)1/556.5 ± 0.040.5 ± 0.067.6 ± 0.052.7 ± 0.0
Combined (most)1/549.8 ± 0.052.3 ± 0.045.3 ± 0.055.6 ± 0.0
Nested (most)2/561.7 ± 0.754.7 ± 2.071.1 ± 0.255.9 ± 1.8
Perf. Profile (least)2/563.3 ± 3.659.7 ± 0.651.6 ± 3.358.1 ± 1.2
IRT (most)2/559.8 ± 3.145.7 ± 3.367.0 ± 0.953.8 ± 2.0
SFT Classifier2/545.7 ± 8.547.7 ± 7.860.2 ± 9.654.9 ± 2.8
Top Solo Score3/560.1 ± 3.645.1 ± 2.773.3 ± 1.451.7 ± 1.9

Exp 2: Training / Sampling Seeds:

SettingsUnique TeamsPR / GSM8KPR / TQABlender / GSM8KBlender / TQA
Random Baseline (5 draws)5/550.5 ± 10.048.0 ± 5.759.8 ± 7.148.0 ± 5.4
Description (most)1/563.4 ± 0.060.0 ± 0.048.5 ± 0.062.1 ± 0.0
LLM Prompt1/540.1 ± 0.043.9 ± 0.070.0 ± 0.048.9 ± 0.0
Idiosyncrasies (least)1/556.5 ± 0.040.5 ± 0.067.6 ± 0.052.7 ± 0.0
Combined (most)1/563.1 ± 0.049.9 ± 0.054.7 ± 0.052.3 ± 0.0
IRT (most)2/538.9 ± 0.749.5 ± 3.868.0 ± 1.059.5 ± 3.0
SFT Classifier2/555.8 ± 6.954.8 ± 8.051.2 ± 3.053.8 ± 1.0
Agentic Top-K2/539.4 ± 1.649.6 ± 0.450.4 ± 11.449.7 ± 0.3

Note: Parameter-free selectors [Single Best, Top Solo, Perf. Profile, Nested] are N/A in Exp 2, and Agentic Top-K uses no item data and is N/A in Exp 1. SD = 0.0 means all 5 replicates picked the identical team, either by construction [Description, LLM Prompt] or empirically [Idiosyncrasies, Combined, Single Best].

The results demonstrate substantial stability for several behavioral selectors. While Random produces 5/5 distinct teams, Idiosyncrasies and Combined select 1/5 distinct teams within each perturbation experiment, and IRT selects 2/5. Under development-data resampling, Nested and Perf. Profile also selects only 2/5 distinct teams, with relatively small performance variations. In contrast, SFT Classifier and Agentic Top-K exhibit larger fluctuations (SD up to 9.6 and 11.4 points, respectively). These findings provide additional evidence that several behavioral and capability-based selection strategies are reproducible across development-data perturbations and random seeds. We included these results in the revised Appendix H Selection Stability Across Multiple Seeds.

Dear reviewer cQut,

The author response deadline is fast approaching: would you be able to take a look? Super appreciated!

Thank you for the thoughtful rebuttal. I appreciate the authors' efforts in addressing my concerns, and I am happy to raise my score.

Reviewer_9Qa1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to select a team before running multi-model collaboration. It separates selection methods into model information and selection procedures, comparing model descriptions, behavioral fingerprints, IRT-based ability representations, performance profiles, hybrid strategies, and LLM recruiters. The experiments select teams of four from candidate pools of 10 and 32 models and evaluate them under four collaboration mechanisms: routing, refinement, weight merging, and blending. The paper also examines malicious candidates, pool size, diversity, and out-of-distribution transfer. The results show that team composition can substantially affect collaboration performance, while the best selection method depends on the task and collaboration mechanism. An important detail about the study’s scope is that, although the second pool draws on different source models, all deployed candidates are distilled Qwen2.5-7B models.

Critical strengths
  1. The paper addresses a practical problem.

Existing collaboration research often starts with a fixed set of models. By directly asking which members should be selected, this paper addresses an important part of designing collaborative systems.

  1. The comparison framework is clear.

Breaking selectors down into a signal and a procedure helps clarify how descriptions, ability estimates, and behavioral information interact with different search methods.

3。 The experiments cover a broad range of settings.

The evaluation includes two candidate pools, three main tasks, and four collaboration mechanisms, alongside practical baselines such as the strongest individual model and Top Solo. This coverage helps reveal how selection strategies interact with collaboration mechanisms.

Critical weaknesses
  1. The contribution of behavioral diversity is not sufficiently separated from ability-based selection and the search procedure.

IRT and Idiosyncrasies start their search with the two strongest candidates, while methods such as Description use different search procedures. Without controls such as keeping the two strongest models and filling the remaining slots at random, some of the gains could come from retaining strong models. The current results support comparisons of complete selection strategies, but further ablations are needed to establish the contribution of the behavioral representations themselves.

  1. The choice of fusion model in Blender may affect team comparisons.

The problem formulation treats a team as an unordered set, but the implementation uses the “first member” as the fuser. The paper does not explain how each selector’s output is ordered, so it is unclear whether the effects of team composition and fuser identity have been separated. Experiments that fix or rotate the fuser would make the conclusions more reliable.

Questions for authors

please refer to the weaknesses

1 comment

Dear Reviewer 9Qa1,

Thank you for the detailed review. Revisions in the updated manuscript are highlighted in blue.

The contribution of behavioral diversity is not sufficiently separated from ability-based selection and the search procedure. IRT and Idiosyncrasies start their search with the two strongest candidates, while methods such as Description use different search procedures. Without controls such as keeping the two strongest models and filling the remaining slots at random, some of the gains could come from retaining strong models. The current results support comparisons of complete selection strategies, but further ablations are needed to establish the contribution of the behavioral representations themselves. Values in parentheses denote the relative difference from each method's respective control baseline.

Response: To isolate the contribution of our behavioral representations from the ability seeding, we conduct an ablation on Pool 1. Specifically, we fix the same two Top2 seed models used in IRT and Idiosyncrasies, and random select remaining two models over 5 seeds (mean ± std). Values in parentheses denote the relative difference from each method's respective control baseline.

SettingRouting / TQARouting / GSM8KBlender / TQABlender / GSM8K
Unconstrained Random48.0 ± 5.750.5 ± 10.048.0 ± 5.459.8 ± 7.1
IRT Control (Top-2 + Random-2)51.9 ± 5.052.5 ± 8.855.9 ± 2.467.4 ± 2.5
IRT (most-diverse)53.6 (+1.7)38.2 (−14.3)56.2 (+0.3)69.1 (+1.7)
IRT (least-diverse)56.9 (+5.0)60.6 (+8.1)55.6 (−0.3)61.8 (−5.6)
Idiosyncrasies Control (Top-2 + Random-2)46.9 ± 4.956.0 ± 5.752.5 ± 1.269.3 ± 3.3
Idiosyncrasies (most-diverse)40.7 (−6.2)51.1 (−4.9)51.5 (−1.0)70.9 (+1.6)
Idiosyncrasies (least-diverse)40.5 (−6.4)56.5 (+0.5)52.7 (+0.2)67.6 (−1.7)

The results show that ability-based seeding and behavioral diversity can provide complementary benefits. Compared with unconstrained random selection, Top-2 seeding improves performance by up to 16.5% relatively. Building on the same Top-2 seeds, behavioral selection further improves performance by up to 15.4% for IRT and 2.3% for Idiosyncrasies. These findings highlight the complementary roles of model ability and behavioral diversity in team selection. We included these ablations in the revised Appendix G Additional Ablations.

The choice of fusion model in Blender may affect team comparisons. The problem formulation treats a team as an unordered set, but the implementation uses the “first member” as the fuser. The paper does not explain how each selector’s output is ordered, so it is unclear whether the effects of team composition and fuser identity have been separated. Experiments that fix or rotate the fuser would make the conclusions more reliable.

Response: To decouple team composition from fuser identity, we re-evaluate all 21 Pool-1 evaluations using a fixed fuser model Qwen2.5-7B-Instruct on Pool 1 and compare with original results:

Selector (Pool 1, LLM-Blender)GSM8K (Orig. Fuser)GSM8K (Fixed Fuser)TQA (Orig. Fuser)TQA (Fixed Fuser)
Nested (least / most)72.7 / 71.281.7 / 80.455.1 / 55.159.3 / 57.1
IRT-Net (least / most)61.8 / 69.181.1 / 76.455.6 / 56.260.5 / 58.8
Combined (least / most)70.5 / 59.080.8 / 71.958.4 / 56.961.9 / 50.6
Idiosyncrasies (most / least)70.9 / 67.677.1 / 75.651.5 / 52.749.8 / 51.1
Top Solo Score73.976.851.956.8
Agentic Top-K72.576.257.056.7
LLM Prompt70.076.248.954.1
SFT Classifier39.175.854.054.1
Perf. Profile (most / least)60.5 / 50.170.2 / 70.150.6 / 58.749.1 / 59.5
Description (least / most)49.7 / 48.567.9 / 66.949.1 / 62.160.5 / 57.7
Random (mean)59.866.348.052.6

The fixed-fuser results further support the effectiveness of behavioral selection. Our strongest selectors achieve 81.7% on GSM8K and 61.9% on TQA, outperforming Top Solo Score (76.8% / 56.8%) and Random (66.3% / 52.6%). Moreover, fixing the fuser substantially improves previously disadvantaged methods (e.g., SFT Classifier: 39.1% → 75.8% on GSM8K), highlighting the influence of fuser identity. We clarified the fuser selection procedure and included this ablation in the revised Appendix G Additional Ablations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.