acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Compute Scales Better with Parallel Agent Teams

Abstract

Test-time scaling improves the performance of a fixed model by spending more computation at inference. There are two common ways to spend it on agents when a verifier can score every attempt. Best-of- sampling runs many independent agents and keeps the best result but saturates quickly. One large team of collaborating agents can score higher but stops improving as it grows. Can we get the best of both? Our method, parallel agent teams, does so by sampling many small teams in parallel and keeping the best team's result. We test it on 12 long-horizon tasks from GPU kernels and formal proofs to exploit development. With a 27B model and a budget of 100 agents, parallel teams of ten beat parallel agents on 10 of 12 tasks and beat one team of 100 on all four tasks where we ran such large teams. At equal dollar cost the same parallel teams beat a frontier model on half the tasks. A simple statistical model shows that individual agents lose because their many samples have a low mean and that one large team loses because a single sample gets no gain from variance. Sampling many small teams in parallel balances a high mean with the gain from variance. Parallel agent teams thus offer a more effective way to scale test-time compute.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi Agent, Long Horizon, Continuous Optimization Problem, Autoresearch

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nsWMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to allocate a fixed test-time compute budget among LLM agents on long-horizon tasks with automatically verifiable solutions. It proposes parallel agent teams: the budget is divided into multiple independent flat teams of k agents, agents within each team share a forum, each team is scored by its best verified submission, and the best team is selected. This interpolates between best-of-N independent agents when k=1 and one large collaborative team when k equals the full budget.

The authors evaluate the approach primarily with Qwen3.8-27B on 12 tasks from several benchmarks and domains, using approximately 100 independent runs and 10 teams of ten per task. At a budget of 100 agents, teams of ten outperform independent agents on 10 of 12 tasks, while limited large-team experiments report that parallel teams outperform a single 100-agent team on all four tested tasks. Additional experiments study team size, longer sessions, another model, orchestrated teams, cost, and forum behavior. A normal order-statistics model is used to interpret the results as a trade-off between the higher mean of collaborative samples and the variance gain from sampling multiple independent samples.

Critical strengths
  1. The proposed construction is simple, modular, and practically relevant. It applies best-of-N selection at the level of small collaborative teams, requires no training or modification of the base model, and exposes an intuitive trade-off between improving the quality of each sample through communication and preserving diversity across samples. This makes the method easy to deploy on tasks with reliable verifiers.

  2. The empirical study is broad and unusually ambitious in scale. It spans 12 heterogeneous long-horizon tasks and includes experiments on team size, another model, longer sessions, orchestrated teams, cost, and forum behavior. The task cards and infrastructure appendices also make the experimental protocol more concrete than is common in many agent papers. The large gains on difficult tasks such as Carleson Formalization and Graph Node Classification make the underlying empirical question worth studying.

Critical weaknesses
  1. The main numerical claims are statistically under-supported. The paper has roughly 100 solo-agent runs but only about 9–11 team runs per task, while the large-team experiments at sizes 16–100 often have only 1–5 repetitions and sometimes only one run. At a budget of 100 agents, the “best of ten teams” is therefore often just the maximum of a very small team pool, whereas the solo-agent curve is estimated from a much larger pool by averaging over many subsets. The paper reports no run-level confidence intervals, paired tests, task-level bootstrap intervals, or systematic accounting of dropped runs. This is particularly problematic because the scores are often bounded, heavy-tailed, bimodal, or affected by evaluator noise. Consequently, claims such as “10/12 tasks,” “all four large-team tasks,” and the unweighted average gain of approximately 29% may be sensitive to sampling variation and outliers. The cross-task average also mixes heterogeneous metrics and appears to be dominated by large relative gains on a small number of tasks, such as the reported +218% on Carleson Formalization.

  2. Several comparisons and mechanistic interpretations are confounded. Team members use 1.0–2.2 times as many tokens as independent agents, team members receive an additional forum instruction that solo agents do not receive, and teams of 64 or 100 have a truncated forum index whereas smaller teams can inspect the full board. The Qwen cost comparison uses a flat GPU-share estimate while the GPT comparison uses token pricing, and the DeepSeek solo arm has a shorter effective run because of a turn cap. The 6-hour solo comparison is also obtained by reading hour-6 checkpoints from agents planned for 12 hours rather than by running independent 6-hour agents. There is no no-communication, delayed-communication, or matched-information baseline, so the team-versus-agent gap cannot be attributed specifically to communication. The orchestrator ablation also changes the number of active problem-solving agents and prevents workers from communicating with one another, so it does not establish that managers are generally harmful. Finally, the paper motivates the setting using per-attempt verifiers, but two MLS-Bench tasks do not provide running scores and are graded only at the end, which raises a scope and interpretation issue.

Questions for authors

The following questions are material to my assessment:

  1. Can the authors provide a complete run-level accounting for every task, budget, and team size, including the number of completed runs, infrastructure failures, evaluator failures, reward-exploit exclusions, and all other dropped runs? Please report task-level confidence intervals or hierarchical/bootstrap uncertainty for the main curves and gains, resampling independent runs rather than only recombining subsets of the same small team pool. How sensitive are the conclusions to the single KDA 100-agent run and other visible outliers?

  2. Can the authors provide a consistent compute accounting based on actual wall-clock time, GPU/CPU hours, input/output/cache tokens, and forum overhead? In particular, do the main conclusions survive token-normalized or FLOP-normalized budgets? Please also clarify the fairness of the DeepSeek and 6-hour comparisons, and whether large teams can be evaluated with the same forum visibility and information bandwidth as smaller teams.

  3. What evidence separates the benefit of within-team communication from simply taking the maximum of k members, the extra forum prompt, or the different information-access regime? A no-forum or delayed-forum control, together with a matched-information orchestrator or shared-memory baseline, would help establish the causal interpretation. Please also clarify why the two MLS-Bench tasks, which have no running verifier, are included in a setting motivated by per-attempt verification, and whether the results are consistent when online-verifier and end-of-run tasks are analyzed separately.

Reviewer_pVZfOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a parallel agent team approach, in which multiple agents are organized into independent collaborative groups, with the best outcome selected via automated evaluation. This method leverages collaboration to enhance the quality of individual solutions while ensuring the diversity of parallel samples. Across 12 long-horizon tasks, it consistently outperforms both independent sampling and a single large-scale team. A statistical model is provided to explain its computational scaling advantages.

Critical strengths
  • Conducted extensive experiments to analyze the patterns of agent collaboration.
  • Performed experiments and analyzed results across different models.
Critical weaknesses
  • The experiments in the paper yield only qualitative conclusions—specifically, that "multiple smaller teams are better"—a finding that is fairly intuitive and has already been presented in numerous other papers. Furthermore, the authors fail to provide a more detailed analysis or offer practical guidelines for determining the optimal number and size of teams.
  • The theoretical analysis is overly simplistic ("toy-like") and relies on many unrealistic assumptions; though, admittedly, developing a theoretical framework that aligns well with real-world application is a challenging task.
  • There is a lack of ablation analysis regarding the total budget.
Questions for authors
  • Would the conclusions presented in the paper change if the total budget were different?
  • In practical application, once the task, model, and total budget have been determined, is there any guidance available for setting the number of teams and their respective sizes?
Reviewer_pFH5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how to allocate test-time compute between independent sampling and multi-agent collaboration. It proposes running multiple small teams independently and selecting the best verified result. Experiments across twelve optimization tasks suggest that this allocation often outperforms independent agents and a single large team under the evaluated budgets.

Critical strengths
  1. Empirical comparison of compute allocation: The study compares independent sampling, single-team collaboration, and parallel teams under a common budget framework.

  2. Evidence for collaboration benefits: Comparisons with independent sampling at matched agent counts and durations support the value of information sharing on some tasks.

  3. Experimental documentation: The task descriptions, communication protocols, and example prompts help readers understand and assess the experimental setup.

Critical weaknesses
  1. Configuration guidance needs further validation: Although the statistical framework is evaluated against observed results, its reliability for choosing configurations from a small number of pilot runs remains unclear.

  2. Mechanistic attribution could be clearer: Some organizational comparisons change multiple communication and coordination choices simultaneously, limiting conclusions about the specific effects of team size or management.

  3. Statistical uncertainty warrants further attention: Although limited replication is acknowledged, uncertainty in the main performance differences could be more clearly quantified.

Questions for authors
  1. Configuration selection: Could the authors clarify whether configurations selected from limited pilot runs retain their advantage in independent evaluations?

  2. Organizational effects: Could the authors clarify which conclusions about team size and management remain supported after accounting for differences in communication design and worker autonomy?

  3. Statistical reliability: Beyond the reported run ranges, could the authors quantify uncertainty in the key performance differences?

1 comment

I have read the response posted by the authors to my review and updated my ratings.

Comments on the paper

1 comment

For all revierers: as the discussion period is approaching its end, please join the discussion and put acknowledge or feedback on authors' rebuttal, thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.