- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how to select models for an LLM team. The authors compare methods based on model descriptions, measured capabilities, diversity, and LLM-based interviews. Experiments use two model pools and four collaboration methods, with further tests on larger pools, malicious models, and generalization to new tasks.
Critical strengths
-
Selecting models that work well together is a practical problem, and the paper offers a useful comparison of several approaches.
-
The experiments cover multiple collaboration methods. The tests involving malicious models and unseen tasks are also interesting.
Critical weaknesses
-
My main concern is how this work differs from existing ensemble-selection methods. Selecting complementary models from a candidate pool is already a well-studied problem, including greedy selection based on validation performance. What limitation of these approaches do the proposed strategies address?
-
The variance comparison in Section 5 does not seem fair. For random selection, the variation is across different teams. For the other methods, it comes from resampling test examples for a fixed selected team. These measure different sources of uncertainty, so smaller error bars do not establish that the selection method is more stable.
-
The diversity methods use different starting points. IRT and Idiosyncrasies begin with the two strongest models, while description diversity does not. It is therefore hard to tell how much of the gain comes from the diversity measure and how much comes from including strong models at the start.
Questions for authors
-
Could you add a greedy baseline that selects models based on the team’s performance on the development set?
-
Would the comparison change if all diversity methods started with the same two strongest models? Adding two random models to those two would also be a useful baseline.
-
Could you repeat selection with different development samples and training seeds to check how stable the selected teams and their performance are?
-
What is the cost of selection, including collecting responses and training the selectors? SFT scores all 35,960 teams in Pool 2. How practical would this be with a much larger candidate pool?
5 comments
The variance comparison in Section 5 does not seem fair. For random selection, the variation is across different teams. For the other methods, it comes from resampling test examples for a fixed selected team. These measure different sources of uncertainty, so smaller error bars do not establish that the selection method is more stable.
Could you repeat selection with different development samples and training seeds to check how stable the selected teams and their performance are?
Response: To evaluate whether the stability advantage holds when repeating the full selection pipeline, we ran two additional experiments on Pool 1 across Prompt Routing and LLM-Blender on GSM8K and TruthfulQA (TQA), each with five replicates: (1) Dev Data Sensitivity, where we resampled selection data and refit or retrained the applicable selectors from scratch with a fixed seed; and (2) Training and Generation Seed Sensitivity, where we fixed selection data and varied initialization, data-ordering, and candidate sampling/judging seeds. We report the number of distinct teams selected and test accuracy (mean ± SD).
Exp 1: Resampled Selection Data
| Settings | Unique Teams | PR / GSM8K | PR / TQA | Blender / GSM8K | Blender / TQA |
|---|---|---|---|---|---|
| Random Baseline | 5/5 | 50.5 ± 10.0 | 48.0 ± 5.7 | 59.8 ± 7.1 | 48.0 ± 5.4 |
| Single Best Model (lima) | 1/5 | 65.6 ± 0.0 | 59.3 ± 0.0 | 65.6 ± 0.0 | 59.3 ± 0.0 |
| Description (most) | 1/5 | 63.4 ± 0.0 | 60.0 ± 0.0 | 48.5 ± 0.0 | 62.1 ± 0.0 |
| LLM Prompt | 1/5 | 40.1 ± 0.0 | 43.9 ± 0.0 | 70.0 ± 0.0 | 48.9 ± 0.0 |
| Idiosyncrasies (least) | 1/5 | 56.5 ± 0.0 | 40.5 ± 0.0 | 67.6 ± 0.0 | 52.7 ± 0.0 |
| Combined (most) | 1/5 | 49.8 ± 0.0 | 52.3 ± 0.0 | 45.3 ± 0.0 | 55.6 ± 0.0 |
| Nested (most) | 2/5 | 61.7 ± 0.7 | 54.7 ± 2.0 | 71.1 ± 0.2 | 55.9 ± 1.8 |
| Perf. Profile (least) | 2/5 | 63.3 ± 3.6 | 59.7 ± 0.6 | 51.6 ± 3.3 | 58.1 ± 1.2 |
| IRT (most) | 2/5 | 59.8 ± 3.1 | 45.7 ± 3.3 | 67.0 ± 0.9 | 53.8 ± 2.0 |
| SFT Classifier | 2/5 | 45.7 ± 8.5 | 47.7 ± 7.8 | 60.2 ± 9.6 | 54.9 ± 2.8 |
| Top Solo Score | 3/5 | 60.1 ± 3.6 | 45.1 ± 2.7 | 73.3 ± 1.4 | 51.7 ± 1.9 |
Exp 2: Training / Sampling Seeds
| Settings | Unique Teams | PR / GSM8K | PR / TQA | Blender / GSM8K | Blender / TQA |
|---|---|---|---|---|---|
| Random Baseline | 5/5 | 50.5 ± 10.0 | 48.0 ± 5.7 | 59.8 ± 7.1 | 48.0 ± 5.4 |
| Description (most) | 1/5 | 63.4 ± 0.0 | 60.0 ± 0.0 | 48.5 ± 0.0 | 62.1 ± 0.0 |
| LLM Prompt | 1/5 | 40.1 ± 0.0 | 43.9 ± 0.0 | 70.0 ± 0.0 | 48.9 ± 0.0 |
| Idiosyncrasies (least) | 1/5 | 56.5 ± 0.0 | 40.5 ± 0.0 | 67.6 ± 0.0 | 52.7 ± 0.0 |
| Combined (most) | 1/5 | 63.1 ± 0.0 | 49.9 ± 0.0 | 54.7 ± 0.0 | 52.3 ± 0.0 |
| IRT (most) | 2/5 | 38.9 ± 0.7 | 49.5 ± 3.8 | 68.0 ± 1.0 | 59.5 ± 3.0 |
| SFT Classifier | 2/5 | 55.8 ± 6.9 | 54.8 ± 8.0 | 51.2 ± 3.0 | 53.8 ± 1.0 |
| Agentic Top-K | 2/5 | 39.4 ± 1.6 | 49.6 ± 0.4 | 50.4 ± 11.4 | 49.7 ± 0.3 |
Note: Parameter-free selectors [Single Best, Top Solo, Perf. Profile, Nested] are N/A in Exp 2, and Agentic Top-K uses no item data and is N/A in Exp 1. SD = 0.0 means all 5 replicates picked the identical team, either by construction [Description, LLM Prompt] or empirically [Idiosyncrasies, Combined, Single Best].
The results demonstrate substantial stability for several behavioral selectors. While Random produces 5/5 distinct teams, Idiosyncrasies and Combined select 1/5 distinct teams within each perturbation experiment, and IRT selects 2/5. Under development-data resampling, Nested and Perf. Profile also selects only 2/5 distinct teams, with relatively small performance variations. In contrast, SFT Classifier and Agentic Top-K exhibit larger fluctuations (SD up to 9.6 and 11.4 points, respectively). These findings provide additional evidence that several behavioral and capability-based selection strategies are reproducible across development-data perturbations and random seeds. We included these results in the revised Appendix H Selection Stability Across Multiple Seeds.
The diversity methods use different starting points. IRT and Idiosyncrasies begin with the two strongest models, while description diversity does not. It is therefore hard to tell how much of the gain comes from the diversity measure and how much comes from including strong models at the start.
Would the comparison change if all diversity methods started with the same two strongest models? Adding two random models to those two would also be a useful baseline.
Response: To isolate the contribution of our behavioral representations from the ability seeding, we conduct an ablation on Pool 1. Specifically, we fix the same two Top2 seed models used in IRT and Idiosyncrasies, and random select remaining two models over 5 seeds (mean ± std). Values in parentheses denote the relative difference from each method's respective control baseline.
| Setting | Routing / TQA | Routing / GSM8K | Blender / TQA | Blender / GSM8K |
|---|---|---|---|---|
| Unconstrained Random | 48.0 ± 5.7 | 50.5 ± 10.0 | 48.0 ± 5.4 | 59.8 ± 7.1 |
| IRT Control (Top-2 + Random-2) | 51.9 ± 5.0 | 52.5 ± 8.8 | 55.9 ± 2.4 | 67.4 ± 2.5 |
| IRT (most-diverse) | 53.6 (+1.7) | 38.2 (−14.3) | 56.2 (+0.3) | 69.1 (+1.7) |
| IRT (least-diverse) | 56.9 (+5.0) | 60.6 (+8.1) | 55.6 (−0.3) | 61.8 (−5.6) |
| Idiosyncrasies Control (Top-2 + Random-2) | 46.9 ± 4.9 | 56.0 ± 5.7 | 52.5 ± 1.2 | 69.3 ± 3.3 |
| Idiosyncrasies (most-diverse) | 40.7 (−6.2) | 51.1 (−4.9) | 51.5 (−1.0) | 70.9 (+1.6) |
| Idiosyncrasies (least-diverse) | 40.5 (−6.4) | 56.5 (+0.5) | 52.7 (+0.2) | 67.6 (−1.7) |
The results show that ability-based seeding and behavioral diversity can provide complementary benefits. Compared with unconstrained random selection, Top-2 seeding improves performance by up to 16.5% relatively. Building on the same Top-2 seeds, behavioral selection further improves performance by up to 15.4% for IRT and 2.3% for Idiosyncrasies. These findings highlight the complementary roles of model ability and behavioral diversity in team selection. We included these ablations in the revised Appendix G Additional Ablations.
What is the cost of selection, including collecting responses and training the selectors? SFT scores all 35,960 teams in Pool 2. How practical would this be with a much larger candidate pool?
Response: We thank the reviewer for this practical question. We collected the actual GPU-hours to quantify the costs of response collection, selector training, and team search across Pool 1 (N=10) and Pool 2 (N=32). GPU-h
| Selection Stage | Method | Pool 1 | Pool 2 | Scaling |
|---|---|---|---|---|
| Per-model responses | IRT, Combined, Nested, Idiosyncrasies | 25.3 GPU-h | ≈66.0 GPU-h | O(N)) |
| Perf.-profile | Perf. Profile | 20.2 GPU-h | 72.8 GPU-h | O(N) |
| Solo benchmark scores | Top Solo, Single Best | 13.6 GPU-h | 30.7 GPU-h | O(N) |
| Agent interviews | Agentic Top-K | 0.3 GPU-h | – | O(N) |
| Selector training | IRT, Idiosyncrasies, SFT | <0.1 / 0.7 / 0.8 GPU-h | <0.1 / 1.5 / 0 (reused) GPU-h | O(1) |
| SFT training labels | SFT Classifier | ≈977 GPU-h | 0 (reused) GPU-h | O(1) |
| SFT Team search | SFT Classifier | 3.5 min (840 passes) | 10 GPU-h | (exhaustive) or (beam/pre-filtered) |
| Other Team search | Other selectors | < 5 sec (CPU) | < 5 sec (CPU) | Microseconds per candidate team |
Our behavioral selectors scale linearly in candidate profiling, requiring approximately 20–73 GPU-h for individual response collection, under 1.5 GPU-h for training, and seconds for team search. For SFT Classifier, training-label collection dominates the cost (≈977 GPU-h), but the trained selector transfers zero-shot from Pool 1 to Pool 2 without additional labeling or retraining. While exhaustive SFT search scales as (O(N^4)), pre-filtering to 20 candidates reduces the search space to 4,845 teams, offering a practical scaling option. We included this analysis in the Appendix J Selection Costs and Scalability.
Dear Reviewer ahNo,
Thank you for the detailed review. Revisions in the updated manuscript are highlighted in blue.
Response: We thank the reviewer for raising this important baseline. We implemented greedy forward selection on Pool 1, starting from the strongest solo model and sequentially adding candidates based on validation performance until reaching a team size of four. We evaluated this baseline across Prompt Routing and LLM-Blender on GSM8K and TruthfulQA.
Greedy selection underperforms our best selectors in 3 of 4 settings, by up to 14.5% relatively (9.4 points), while marginally achieving the best result on Blender/GSM8K. Moreover, greedy requires 24 and 90 collaborative evaluations per setting in Pools 1 and 2, respectively, whereas our representation-based selectors reuse candidate-level information across collaboration mechanisms, avoiding repeated team-level evaluations and inference costs. These results highlight the effectiveness and efficiency of our proposed suite of methods. We included this comparison in the revised Appendix I Greedy Ensemble Selection.