- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to allocate a fixed test-time compute budget among LLM agents on long-horizon tasks with automatically verifiable solutions. It proposes parallel agent teams: the budget is divided into multiple independent flat teams of k agents, agents within each team share a forum, each team is scored by its best verified submission, and the best team is selected. This interpolates between best-of-N independent agents when k=1 and one large collaborative team when k equals the full budget.
The authors evaluate the approach primarily with Qwen3.8-27B on 12 tasks from several benchmarks and domains, using approximately 100 independent runs and 10 teams of ten per task. At a budget of 100 agents, teams of ten outperform independent agents on 10 of 12 tasks, while limited large-team experiments report that parallel teams outperform a single 100-agent team on all four tested tasks. Additional experiments study team size, longer sessions, another model, orchestrated teams, cost, and forum behavior. A normal order-statistics model is used to interpret the results as a trade-off between the higher mean of collaborative samples and the variance gain from sampling multiple independent samples.
Critical strengths
-
The proposed construction is simple, modular, and practically relevant. It applies best-of-N selection at the level of small collaborative teams, requires no training or modification of the base model, and exposes an intuitive trade-off between improving the quality of each sample through communication and preserving diversity across samples. This makes the method easy to deploy on tasks with reliable verifiers.
-
The empirical study is broad and unusually ambitious in scale. It spans 12 heterogeneous long-horizon tasks and includes experiments on team size, another model, longer sessions, orchestrated teams, cost, and forum behavior. The task cards and infrastructure appendices also make the experimental protocol more concrete than is common in many agent papers. The large gains on difficult tasks such as Carleson Formalization and Graph Node Classification make the underlying empirical question worth studying.
Critical weaknesses
-
The main numerical claims are statistically under-supported. The paper has roughly 100 solo-agent runs but only about 9–11 team runs per task, while the large-team experiments at sizes 16–100 often have only 1–5 repetitions and sometimes only one run. At a budget of 100 agents, the “best of ten teams” is therefore often just the maximum of a very small team pool, whereas the solo-agent curve is estimated from a much larger pool by averaging over many subsets. The paper reports no run-level confidence intervals, paired tests, task-level bootstrap intervals, or systematic accounting of dropped runs. This is particularly problematic because the scores are often bounded, heavy-tailed, bimodal, or affected by evaluator noise. Consequently, claims such as “10/12 tasks,” “all four large-team tasks,” and the unweighted average gain of approximately 29% may be sensitive to sampling variation and outliers. The cross-task average also mixes heterogeneous metrics and appears to be dominated by large relative gains on a small number of tasks, such as the reported +218% on Carleson Formalization.
-
Several comparisons and mechanistic interpretations are confounded. Team members use 1.0–2.2 times as many tokens as independent agents, team members receive an additional forum instruction that solo agents do not receive, and teams of 64 or 100 have a truncated forum index whereas smaller teams can inspect the full board. The Qwen cost comparison uses a flat GPU-share estimate while the GPT comparison uses token pricing, and the DeepSeek solo arm has a shorter effective run because of a turn cap. The 6-hour solo comparison is also obtained by reading hour-6 checkpoints from agents planned for 12 hours rather than by running independent 6-hour agents. There is no no-communication, delayed-communication, or matched-information baseline, so the team-versus-agent gap cannot be attributed specifically to communication. The orchestrator ablation also changes the number of active problem-solving agents and prevents workers from communicating with one another, so it does not establish that managers are generally harmful. Finally, the paper motivates the setting using per-attempt verifiers, but two MLS-Bench tasks do not provide running scores and are graded only at the end, which raises a scope and interpretation issue.
Questions for authors
The following questions are material to my assessment:
-
Can the authors provide a complete run-level accounting for every task, budget, and team size, including the number of completed runs, infrastructure failures, evaluator failures, reward-exploit exclusions, and all other dropped runs? Please report task-level confidence intervals or hierarchical/bootstrap uncertainty for the main curves and gains, resampling independent runs rather than only recombining subsets of the same small team pool. How sensitive are the conclusions to the single KDA 100-agent run and other visible outliers?
-
Can the authors provide a consistent compute accounting based on actual wall-clock time, GPU/CPU hours, input/output/cache tokens, and forum overhead? In particular, do the main conclusions survive token-normalized or FLOP-normalized budgets? Please also clarify the fairness of the DeepSeek and 6-hour comparisons, and whether large teams can be evaluated with the same forum visibility and information bandwidth as smaller teams.
-
What evidence separates the benefit of within-team communication from simply taking the maximum of k members, the extra forum prompt, or the different information-access regime? A no-forum or delayed-forum control, together with a matched-information orchestrator or shared-memory baseline, would help establish the causal interpretation. Please also clarify why the two MLS-Bench tasks, which have no running verifier, are included in a setting motivated by per-attempt verification, and whether the results are consistent when online-verifier and end-of-run tasks are analyzed separately.
I have read the response posted by the authors to my review and updated my ratings.