- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents a controlled empirical study of multi-model answer aggregation, disentangling the effects of aggregation methods, sampling budgets, and member selection. Using six open-weight, non-reasoning models and verifiable tasks, the authors compare plurality voting with one-round synthesis, evaluate teams against individual models under matched sampling budgets, examine the impact of limited development labels on routing, and systematically explore combinations of model pools and benchmarks. The main findings suggest that aggregation performance is sensitive to input completeness, tie-breaking rules, and aggregator identity; that individual models can match or outperform the evaluated voting teams under comparable budgets; that teams may offer advantages when labeled data for member selection is scarce; and that routing tends to be preferable when one model is substantially stronger than the others.
Critical strengths
The paper makes a useful distinction among aggregation, sampling budget, and member selection. Its controlled experimental design, particularly the use of identical candidate answers and shared generation libraries, helps reduce confounding factors and facilitates more interpretable comparisons.
Critical weaknesses
-
Limited empirical scope relative to the paper's framing. The first two studies primarily rely on fixed model pools, while the evaluated team methods are restricted to plurality voting and one-round synthesis using models from the same pool. The paper does not examine other representative multi-agent approaches, such as debate. Consequently, the findings support conclusions about the specific aggregation methods studied, but provide insufficient evidence for broader claims about multi-agent systems.
-
Implementation choices and retrospective budget calibration complicate the interpretation of key comparisons. In the prefix-input synthesis experiment, candidate answers may be truncated before aggregation, potentially affecting synthesis performance. The full-input evaluation, however, is limited to four test suites. In addition, the near-budget comparison determines the single-member sampling configuration using completion-token costs recorded during the experiments, selecting a configuration that does not exceed the observed team budget. While this provides a useful retrospective comparison, it does not establish a budget-matching procedure that can be specified and deployed before evaluation.
Questions for authors
-
Is the truncation of candidate answers in the prefix-input synthesis setting inherited from an established external baseline? If not, I encourage the authors to make the full-input setting the primary evaluation and extend it to all relevant test suites.
-
Could the authors repeat the aggregation and budget-matching experiments across multiple independently constructed model pools, or otherwise quantify the sensitivity of the main findings to model-pool composition?
-
Can budget matching be performed entirely before evaluation? Specifically, could the authors compare team and single-member methods using a pre-specified and frozen cost-calibration procedure, and clearly distinguish this deployable setting from the retrospective calibration based on observed completion-token costs?