acceptodds
Desk rejected at ICLR 2027

When Is a Team Worth Convening? Separating Aggregation, Sampling, and Member Selection

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-agent systems (MAS) increasingly convene multiple heterogeneous language models as a team, and the resulting performance gains are often attributed to collaboration. A team's inference, however, bundles collaboration with other mechanisms, including how candidate answers are aggregated, how much sampling budget is spent, and how the single-member alternative is selected, so an observed team gain cannot by itself identify collaboration as the source of the improvement. In this paper, we isolate these sources through controlled experiments on verifiable tasks with open non-thinking models of up to 14B parameters. We use two simple voting policies throughout, named after the answer camps (groups of responses giving the same answer) they count: CAMP-1 takes a single plurality vote, whereas CAMP adds votes only when first-pass agreement is low. Our findings show that the apparent team advantage changes with each of the three controls. The relative ranking of voting and synthesis depends on the aggregator, and with full candidate inputs, synthesis shows no average loss to an order-neutral vote. At a comparable budget, a hindsight-selected single member outperforms the tested voting teams, whereas neither side has an established advantage when selection is separated from evaluation. The relative value of routing depends strongly on label budget and pool composition: teams outperform the router at small label budgets in the development-selected pool, but this advantage is not established at larger budgets, while broader pools with a dominant member favor routing. Together, these results provide a practical decomposition for deciding whether to convene a team: before attributing a team gain to collaboration, one should determine how much comes from the aggregation rule, additional sampling, and the information available for member selection. This reframes a basic question for MAS—not only how a team should collaborate, but whether the task warrants a team at all.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-agent systems, test-time compute

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VKWwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents a controlled empirical study of multi-model answer aggregation, disentangling the effects of aggregation methods, sampling budgets, and member selection. Using six open-weight, non-reasoning models and verifiable tasks, the authors compare plurality voting with one-round synthesis, evaluate teams against individual models under matched sampling budgets, examine the impact of limited development labels on routing, and systematically explore combinations of model pools and benchmarks. The main findings suggest that aggregation performance is sensitive to input completeness, tie-breaking rules, and aggregator identity; that individual models can match or outperform the evaluated voting teams under comparable budgets; that teams may offer advantages when labeled data for member selection is scarce; and that routing tends to be preferable when one model is substantially stronger than the others.

Critical strengths

The paper makes a useful distinction among aggregation, sampling budget, and member selection. Its controlled experimental design, particularly the use of identical candidate answers and shared generation libraries, helps reduce confounding factors and facilitates more interpretable comparisons.

Critical weaknesses
  1. Limited empirical scope relative to the paper's framing. The first two studies primarily rely on fixed model pools, while the evaluated team methods are restricted to plurality voting and one-round synthesis using models from the same pool. The paper does not examine other representative multi-agent approaches, such as debate. Consequently, the findings support conclusions about the specific aggregation methods studied, but provide insufficient evidence for broader claims about multi-agent systems.

  2. Implementation choices and retrospective budget calibration complicate the interpretation of key comparisons. In the prefix-input synthesis experiment, candidate answers may be truncated before aggregation, potentially affecting synthesis performance. The full-input evaluation, however, is limited to four test suites. In addition, the near-budget comparison determines the single-member sampling configuration using completion-token costs recorded during the experiments, selecting a configuration that does not exceed the observed team budget. While this provides a useful retrospective comparison, it does not establish a budget-matching procedure that can be specified and deployed before evaluation.

Questions for authors
  1. Is the truncation of candidate answers in the prefix-input synthesis setting inherited from an established external baseline? If not, I encourage the authors to make the full-input setting the primary evaluation and extend it to all relevant test suites.

  2. Could the authors repeat the aggregation and budget-matching experiments across multiple independently constructed model pools, or otherwise quantify the sensitivity of the main findings to model-pool composition?

  3. Can budget matching be performed entirely before evaluation? Specifically, could the authors compare team and single-member methods using a pre-specified and frozen cost-calibration procedure, and clearly distinguish this deployable setting from the retrospective calibration based on observed completion-token costs?

Reviewer_3PCjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the question of where the gain from a MAD comes from. The authors designed detailed experiments to distinguish three sources of team performance: how model combine answers, the effect of additional samples, and routing selects which model to call. For each factor, authors found interesting observations: (1) Aggregator inputs and tie rules can change the ranking of voting, (2) a hindsight-selected member wins at comparable token budgets, but a separately selected member has no established advantage in the selected pool, (3) teams help the tested router when selection labels are scarce. More importantly, the authors want to call out that whether MAD is really needed for solving a task needs careful consideration.

Critical strengths
  1. The experimental design addresses all three sources of apparent team advantage: aggregation, sampling budget, and member selection. The paper compares aggregation rules on fixed candidate records, uses shared generation libraries to examine team versus single-member sampling at comparable token budgets, and varies the information available for selecting a member. This coordinated design is the paper’s strongest contribution: it shows why a team-versus-single-model accuracy difference alone cannot establish the value of collaboration. The resulting evaluation framework is useful beyond the particular CAMP policies and provides a substantive reason to support acceptance.

  2. The experiments produce actionable findings that challenge simple conclusions about which inference strategy is better. Table 1 shows that the apparent advantage of voting depends strongly on candidate visibility and tie-breaking: with full candidates and order-neutral voting, synthesis achieves a 5.22-point advantage on AIME. Conversely, the shared-library budget analysis shows that competitive single-member alternatives can substantially change the team comparison. The paper strengthens these findings by separating hindsight selection from deployable selection and explicitly reporting when uncertainty prevents a conclusion. These results contribute useful empirical knowledge rather than merely introducing another aggregation method.

Critical weaknesses
  1. The team’s advantage with ten labels may depend on how the single member is selected. The router selects a member using five samples per development question (Section 5). It then evaluates that member with a different sampling policy at a near-matched token budget. Appendix D.4 shows that selecting members under the evaluation policy can change the choices and results. However, that check does not cover the ten-label setting. It remains unclear whether the team’s advantage would persist with this alternative selector.

  2. The budget and selection comparisons cover voting teams, but not the stronger synthesis setups. Section 3 shows that some full-input synthesis hosts outperform voting. Sections 4–5 do not compare these setups with a single member at a matched budget. The paper therefore leaves open whether synthesis makes a team worth convening. Section 7 acknowledges this gap. It limits the scope of the findings, but does not invalidate the voting results.

Questions for authors
  1. Does the N=10 team advantage persist when development labels rank members at the same sampling policy used at evaluation?

  2. Would the main results change if the budget accounted for differences in per-token cost across models and the cost of processing inputs? Can the existing logs support this check? If not, what additional measurements are needed?

Reviewer_Lq5fOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the performance gains of multi-agent systems genuinely arise from collaboration among models or are influenced by answer aggregation, inference-time sampling budgets, and member selection strategies. The authors propose a controlled evaluation framework and use two voting policies CAMP-1 and CAMP as experimental baselines to examine the differences between voting and synthesis, the performance of teams versus single models under comparable sampling budgets, and the effects of different labeling budgets and model pool compositions on model routing. The experiments cover multiple verifiable tasks, heterogeneous model combinations, and 4032 model pool–task configurations. The results show that the aggregator's input, tie-breaking rules, and aggregator identity can change the relative performance of voting and synthesis. Under comparable inference budgets, a hindsight-selected optimal single model can outperform voting teams, although neither approach has an established advantage when member selection and evaluation are separated. When a model pool contains a clearly dominant member, routing is generally more accurate and less costly, whereas team voting may offer advantages in certain model pools when labeled data for member selection are limited.

Critical strengths
  1. The paper identifies several overlooked confounding factors in multi-agent evaluation. The authors do not simply compare the final accuracy of different aggregation methods. Instead, they use the same candidate responses to examine how input truncation, tie-breaking rules, and aggregator selection affect the results. The experiments show that even these seemingly minor implementation details can significantly change performance and sometimes reverse the relative ranking of Voting and Synthesis. I believe this is an important finding, as it shows that performance gains in multi-agent systems cannot always be directly attributed to collaboration. It also provides useful guidance for designing more rigorous evaluations in future research.

  2. The experimental setup is relatively rigorous and covers a broad range of settings. The experiments cover multiple verifiable tasks and combinations of language models from major model providers. The paper also uses shared generation samples, approximately budget-matched comparisons, different labeling budgets, and 4032 model pool–task configurations to examine the performance differences between teams and single models. The appendix also considers the potential bias introduced by hindsight-optimal member selection and includes additional experiments that separate model selection from evaluation. One particularly strong aspect of this paper is that it goes beyond simply reporting performance numbers and further investigates the conditions under which team-based approaches are beneficial, as well as their limitations. I think this makes the findings more informative and strengthens the credibility of the overall analysis.

Critical weaknesses
  1. The team advantage under limited labeling budgets depends heavily on how the model pool is constructed. I noticed that in Section 4.2, the authors use all available development labels to select six models with relatively similar performance, while excluding the strongest overall candidate. However, the router is then allowed to use only a small number of labels for model selection. I think this setup is somewhat different from real-world deployment scenarios, and the additional experiments also show that including a stronger model can change the results. Therefore, it remains unclear whether the team advantage would still hold in more naturally constructed model pools.

  2. The three main factors are not evaluated together in a unified experimental setting. The paper investigates Aggregation, Sampling Budget, and Member Selection separately, but the experimental settings are not fully consistent. For example, the full-input Synthesis experiments cover only four benchmarks, while the budget-matched comparisons with single models mainly focus on Voting. As a result, I am still not sure whether a Synthesis-based team can outperform a properly selected single model under the same inference budget. I think this limitation somewhat weakens the evidence supporting the paper's central conclusion.

  3. The generalizability of the findings is still somewhat limited. Although the paper includes 4032 model pool–task configurations, this is already a substantial number, but many of these configurations overlap. Most experiments also focus on relatively small non-thinking models and tasks with verifiable answers. It remains unclear whether the findings would hold for stronger reasoning models or more complex multi-agent scenarios involving multi-round collaboration and tool use. In addition, the cost analysis mainly focuses on generated tokens and does not fully account for inference latency and other practical deployment costs.

Questions for authors
  1. Model Pool Construction: In Section 4.2, the authors use all development labels to construct the model pool while excluding the strongest model. Could the authors explain the reasoning behind this setup? Would the team advantage under limited labeling budgets still hold if the strongest model were included or if the pool were not specifically selected?

  2. Unified Experimental Comparison: Could the authors provide a direct comparison between a Full-input Synthesis Team and a development-selected single model under a matched inference budget? Are the current separate experiments sufficient to support the paper's main conclusions about the value of teams?

  3. Inference Cost: As I mentioned earlier, the paper mainly measures cost using completion tokens. Could considering input tokens, inference latency, and parallel execution across multiple models change the relative advantages of Team and Routing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.