acceptodds
Under review as a conference paper at ICLR 2027

Scaling Inference-Time Compute with Teams of Coding Agents

Abstract

Multi-agent systems promise a new axis for scaling inference: rather than having one agent think longer or sample more attempts, a task is split across many agents that work in parallel and coordinate. However, in practice, more agents can easily produce duplicate work, coordination failures, and higher costs. We therefore formulate team design as a compute allocation problem: starting from a solo agent, where should additional inference compute go to reach the reward-cost Pareto frontier? We build a minimal multi-agent harness from the ground up and trace this frontier by varying, one at a time, the workflow stage at which a lead delegates to its workers (planning, implementation, or verification), the models that form the team, and the number of workers. On long-horizon challenging software-engineering tasks (e.g., ProgramBench), teams outperform solo agents but cost three to seven times as much. With the right compute allocation, however, a heterogeneous 16-agent team comes within 1.4 points of a strong homogeneous 4-agent team at 70% lower cost. This efficiency stems from three findings. First, planning is the most valuable use of workers: in a homogeneous team whose lead decides how many workers to spawn, a lead that delegates only planning to its workers, and then implements and verifies the code itself, matches the reward of a lead that delegates to workers throughout planning, implementation, and verification, at half the cost. Second, cheap models pay off as planners but not as implementers: a strong lead with cheap planners recovers 47.5% of the reward gain of an all-strong team for only 13.9% of its additional cost, whereas cheap implementers lower reward. Third, planners keep improving reward as the team grows, whereas implementers and verifiers plateau beyond four workers: additional implementers fragment the code into more files for the lead to integrate, and additional verifiers add little new information about the program. Together, our results offer a guide to spending an agent budget where it counts, given the task, cost, and latency constraints a user has.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent evaluation, llm agents, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pJJxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate inference-time compute in teams of coding agents. It frames multi-agent team design as a compute allocation problem and builds a controlled framework to study delegation stages, model choices, and team size. Experiments across ProgramBench and Commit0-Lite show that where workers help most depends on the task. On ProgramBench, a strong lead can benefit from cheaper planning workers but adding more implementation or verification workers brings limited gains, while delegating implementation is more effective on Commit0-Lite.

Critical strengths
  1. The paper addresses an important question of how to allocate additional inference compute in multi-agent teams. It examines where to delegate, which models to use, how many workers to add, and whether communication helps.
  2. The experiments give a detailed empirical picture of how team design affects performance and cost.
Critical weaknesses
  1. The main findings rely heavily on ProgramBench where discovering the behavior of a black-box program is central to the task. This may explain why planning workers are especially useful. However, Commit0-Lite only has 16 tasks and each configuration is run once, and also the strongest model already solves most tasks perfectly. It is therefore hard to tell how well the findings carry over to other coding tasks.
  2. The authors did not compare teams with single agents under the same compute budget. It is clear that multi-agent teams generally cost more, but the experiments do not establish if a single agent given a comparable budget for further exploration or repeated attempts could achieve similar gains.
Questions for authors
  1. On ProgramBench, are there types of tasks where implementation or verification workers help more than planning workers?
  2. Since the configurations use different budgets, how much of the improvement can be attributed to delegation rather than simply spending more compute? Do you have any comparisons with a solo agent at a similar cost? (referring to weakness 2).
  3. The analysis suggests that planning helps because workers gather more information about the target program. I am curious if there is trace evidence showing that different workers discover different behaviors, rather than repeat similar investigations?
  4. On Commit0-Lite, is the advantage of delegating implementation consistent across tasks or driven by a few libraries?
1 comment
AuthorsOpenReview ↗

Response to Reviewer pJJx

Thanks for your detailed review and thoughtful questions. Please find our responses below, with updates marked in blue in the revised PDF.

The authors did not compare teams with single agents under the same compute budget.

To evaluate at matched compute levels, we compare against an oracle best-of-N approach, where a configuration is run N times and an oracle selects the highest-scoring attempt on each task. The cost is then N times that of a single run (new Appendix C.3, Figure 12). Because the oracle selects using ProgramBench's hidden test rewards, we view it as an upper bound on this sampling approach, since it selects perfectly and costs nothing.

We ran Solo PIV ten times on all 200 tasks for each of Sol, Terra, and Luna, roughly enough for solo sampling to span the cost of a single run of any team. We then fit saturating curves (reward=c−ANb\text{reward} = c - AN^b, where cc is the reward the curve approaches and bb sets how quickly it gets there) through best-of-1 to best-of-10, which lets us estimate the solo reward at any budget. Teams have three runs each, so we also report best-of-3 for them.

  • Repeated solo attempts have diminishing returns. For all leads, each doubling of attempts adds less than the last, and going from four to eight adds only about half as much as going from one to two.
  • In a single run, the planning teams already beat solo at the same cost, but not every team does. Both planning teams (table below) are ahead by 0.6 points, while Team PIV, which uses workers at every stage, costs twice as much as Team P and falls 1.8 points below.
  • With best-of-3 for the teams as well, all three are ahead of solo at the same cost, and the planning teams' lead grows from +0.6 to +1.5 and +1.9.
Sol-lead teamTeamCostTeam rewardEstimated solo reward at same cost
Sol lead + 16 Luna plannerssingle run (mean of 3)$1570.6190.613 (~2 attempts)
best-of-3$4720.6620.648 (~6 attempts)
Team P (Sol planners)single run (mean of 3)$2490.6360.630 (~3 attempts)
best-of-3$7470.6760.657 (~10 attempts)
Team PIV (Sol workers at all stages)single run (mean of 3)$5100.6320.649 (~7 attempts)
best-of-3$1,5290.6730.669 (~20 attempts)

The main findings rely heavily on ProgramBench [...] Commit0-Lite only has 16 tasks and each configuration is run once.

We have now run every Commit0-Lite setup three times. Implementation remains the strongest single stage in nine of ten settings (five model setups, each with blocking and asynchronous scheduling), while planning is never clearly best (Appendix C.15). We agree that 16 tasks, with Sol near saturation (Solo PIV solves 10 of 16 tasks perfectly in all three runs), cannot establish generality on their own, so we use Commit0-Lite as a contrast to ProgramBench. When the repository skeleton and unit tests are given, the bottleneck shifts from discovering the specification to implementing it, and the stage that benefits most from extra workers shifts with it.

Are there types of tasks where implementation or verification workers help more than planning workers?

Across three Sol runs, implementation or verification beats planning by >5 points on 23 of 200 tasks, whereas planning beats both by >5 points on 47. No single label separates those 23 tasks, but the balance shifts with difficulty (Appendix C.8). Planning's lead over implementation is widest on easy and medium utilities (+8.6 vs. +1.0 lift on easy, +6.0 vs. +1.6 on medium). On hard tasks, which are large multi-subsystem programs such as duckdb, ffmpeg, php-src, and typst, planning's lift narrows (+2.7 vs. +1.9 for implementation) as implementation volume becomes a larger share of the work.

Is there trace evidence showing that different workers discover different behaviors?

Yes. With sixteen planners, 104 of the 108 reference-executable runs per task are distinct commands, and pairwise similarity between planner reports drops from 0.26 with two planners to 0.17 with sixteen. Each planner runs the executable about seven times regardless of team size, so total coverage grows with the team (Appendix C.10).

On Commit0-Lite, is the advantage of delegating implementation consistent across tasks or driven by a few libraries?

It holds across most non-saturated libraries, though two hard libraries drive most of the mean lift. Across the ten settings and three runs, implementation beats planning by >1 point on 9 of 16 libraries, is within 1 point on 5 (all near ceiling), and trails on 2 (deprecated -7.4, minitorch -4.5). Gains are largest on babel (+56.7) and jinja (+44.6), where solo scores are lowest (Appendix C.15).


Thanks again for your feedback and questions, which prompted us to add the budget-matched comparison and per-library Commit0 breakdown. We hope our responses address your concerns, and we welcome any further discussion.

Reviewer_kzdBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate inference-time compute across teams of coding agents by varying delegation stages, model choices, and team sizes. Experiments mainly on ProgramBench show that using cheaper workers for planning while leaving implementation to a strong lead offers a better cost–performance tradeoff than delegating throughout the workflow.

Critical strengths
  1. The paper studies a practical question: where additional compute is most useful in a team of coding agents. Considering both performance and cost makes the study relevant to users deciding whether and how to use multiple agents.
  2. The experiments systematically examine delegation stages, model choices, team sizes, scheduling, and communication within a common framework. This makes the comparisons easier to interpret, and the repeated runs and task-level confidence intervals strengthen the main results.
  3. The paper provides useful findings beyond whether teams outperform solo agents. On ProgramBench, cheaper workers help with planning but hurt when assigned implementation, while additional communication does not reliably improve performance. These results give concrete guidance on team design.
Critical weaknesses
  1. The gains over a strong solo agent are modest relative to the added cost. Using 16 cheap planning workers improves mean reward from about 57.7% to 61.8% while roughly doubling the cost. The highlighted 70% cost reduction is relative to an expensive team, leaving the practical benefit over a solo agent less convincing.
  2. Most evidence comes from ProgramBench. Commit0-Lite contains only 16 tasks, with one run per configuration, and the strong solo model already achieves full reward on 12 tasks. This provides limited support for broader conclusions about coding tasks.
  3. The team-size scaling experiment uses only a Sol lead with Luna workers and one run per configuration. The finding that planning continues to benefit from more workers while implementation and verification plateau needs validation across other model combinations and repeated runs.
Questions for authors

See the weakness part.

1 comment
AuthorsOpenReview ↗

Response to Reviewer kzdB

Thank you for your careful review. We are glad you found the question practical for users deciding whether and how to use multi-agent teams, and that our results offer concrete guidance on team design. Our responses to your concerns are listed in turn below, with updates marked in blue in the revised PDF.

The gains over a strong solo agent are modest relative to the added cost. [...] leaving the practical benefit over a solo agent less convincing.

To evaluate at matched compute levels, we compare against an oracle best-of-N approach, where a configuration is run N times and an oracle selects the highest-scoring attempt on each task. The cost is then N times that of a single run (new Appendix C.3, Figure 12). Because the oracle selects using ProgramBench's hidden test rewards, we view it as an upper bound on this sampling approach, since it selects perfectly and costs nothing.

We ran Solo PIV ten times on all 200 tasks for each of Sol, Terra, and Luna, roughly enough for solo sampling to span the cost of a single run of any team. We then fit saturating curves (reward=c−ANb\text{reward} = c - AN^b, where cc is the reward the curve approaches and bb sets how quickly it gets there) through best-of-1 to best-of-10, which lets us estimate the solo reward at any budget. Teams have three runs each, so we also report best-of-3 for them.

  • Repeated solo attempts have diminishing returns. For all leads, each doubling of attempts adds less than the last, and going from four to eight adds only about half as much as going from one to two.
  • In a single run, the planning teams already beat solo at the same cost, but not every team does. Both planning teams (table below) are ahead by 0.6 points, while Team PIV, which uses workers at every stage, costs twice as much as Team P and falls 1.8 points below.
  • With best-of-3 for the teams as well, all three are ahead of solo at the same cost, and the planning teams' lead grows from +0.6 to +1.5 and +1.9.
Sol-lead teamTeamCostTeam rewardEstimated solo reward at same cost
Sol lead + 16 Luna plannerssingle run (mean of 3)$1570.6190.613 (~2 attempts)
best-of-3$4720.6620.648 (~6 attempts)
Team P (Sol planners)single run (mean of 3)$2490.6360.630 (~3 attempts)
best-of-3$7470.6760.657 (~10 attempts)
Team PIV (Sol workers at all stages)single run (mean of 3)$5100.6320.649 (~7 attempts)
best-of-3$1,5290.6730.669 (~20 attempts)

Commit0-Lite contains only 16 tasks, with one run per configuration [...].

We have now run every Commit0-Lite setup three times. Implementation remains the strongest single stage in nine of ten settings (five model setups, each with blocking and asynchronous scheduling), while planning is never clearly best (Appendix C.15). We agree that 16 tasks, with Sol near saturation (Solo PIV solves 10 of 16 tasks perfectly in all three runs), cannot establish generality on their own, so we use Commit0-Lite as a contrast to ProgramBench. When the repository skeleton and unit tests are given, the bottleneck shifts from discovering the specification to implementing it, and the stage that benefits most from extra workers shifts with it.

The team-size scaling experiment uses only a Sol lead with Luna workers and one run per configuration.

We have now run the Sol + Luna fixed-size experiment three times, and the pattern replicates closely. Planning gains +2.1 points (95% CI [+1.4, +2.8]) from two to sixteen workers, while implementation and verification stay flat beyond four workers (Figure 8, Section 7).

To test other model pairings, we also ran all 16 fixed-size configurations with a Terra lead and with a Luna lead, again with Luna workers over three runs each (new Appendix C.9). Planning keeps improving from two to sixteen workers in both (+2.7 [+1.7, +3.7] and +4.1 [+2.7, +5.5] points), while implementation and verification again plateau after four workers.


Thanks again for your thoughtful comments, which led us to add the repeated-sampling comparison and the additional fixed-size experiments across model pairings. We hope our responses address your concerns, and we are happy to discuss further.

Reviewer_LPx5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to spend inference-time compute in multi-agent coding systems, asking where extra agents actually help instead of simply adding more workers. They introduce a controlled lead–worker framework that varies the delegation stage, model strength, and team size, allowing the authors to directly compare different team designs on the reward–cost tradeoff. On ProgramBench, their empircal results show that the most efficient setup is a strong lead with cheap workers focused on planning: planning continues to benefit from more workers, while implementation and verification quickly saturate.

Critical strengths
  1. The paper studies multi-agent systems from a very interesting and important angle: when using multiple agents for inference-time scaling, how should additional compute be allocated within the team to get the biggest performance gains. The authors clearly recognize that adding more agents also leads to duplicated work and higher coordination and integration costs, so team performance does not automatically improve as the number of agents increases.
  2. The experiments are well controlled. The authors change one design choice at a time. This makes the results much easier to interpret than comparisons between completely different agent systems.
Critical weaknesses
  1. The most impressive result (with 16 workers), comes from just one run of the fixed-size configuration. Yet the authors’ own variance analysis shows substantial noise in individual task results.
Questions for authors
  1. CAID paper (https://arxiv.org/abs/2603.21489) also study some similar problem in their analysis. They show how many subagents a multi-agent system needs. Could you summarize specifically how your key findings advance our understanding beyond the analysis in that paper? I'd like to see more about how inference compute should be allocated across a team.
  2. Can we predict where the bottleneck will be before the task starts? For example, could features such as how complete the specification is, how well the repository is modularized, and whether tests are available help predict whether to assign workers to planning or implementation?
  3. If planners are useful because they uncover different information, could we explicitly assign them different areas to explore? For example, one could focus on the CLI, another on edge cases, and another on error handling. Would this use compute more efficiently?
  4. Why does implementation performance stop improving beyond four workers?
1 comment
AuthorsOpenReview ↗

Response to Reviewer LPx5

Thanks for your careful reading of our paper and for highlighting our controlled experimental design as well as the value in framing team design around compute allocation. Please find our responses to your questions below, with updates marked in blue in the revised PDF.

The most impressive result (with 16 workers), comes from just one run of the fixed-size configuration.

We agree this needed additional validation, and we have since run the fixed-size sweep two more times across all 200 tasks with a Sol lead, as well as three times each with a Terra lead and with a Luna lead. Planning with a Sol lead and 16 Luna workers scores 0.618, 0.623, and 0.616 across the three runs, coming within 1.2 points of the all-Sol PIV team at 69% lower cost (Figure 8, Section 7). From two to sixteen workers, planning teams gain 2.1 points (95% CI [+1.4, +2.8]), whereas implementation and verification teams flatten after four (+0.3 and -0.1 points from four to sixteen). The same pattern holds with the Terra and Luna leads (new Appendix C.9).

Could you summarize specifically how your key findings advance our understanding beyond the analysis in [CAID]?

CAID coordinates asynchronous engineers over shared repositories via software-engineering primitives (dependency graphs, git worktrees, and test-gated merges). The authors observe in an ablation that scaling parallel engineers from 2 to 4 helps on Commit0-Lite before dropping at 8 from integration overhead. Using a simple shared-workspace harness, we instead ask how compute should be allocated across stages, model tiers, and team sizes.

Two findings extend CAID's team-size ablation. First, scaling depends on what workers do. On ProgramBench, implementers and verifiers plateau around four workers just as in CAID, whereas planners keep raising reward up to sixteen workers (the largest team size we tested). Second, capability does not need to be uniform across roles. Pairing a strong lead with cheaper planners can preserve some of the gain at a fraction of the cost, whereas cheap implementers lower reward below the solo agent.

Can we predict where the bottleneck will be before the task starts?

We hypothesize that yes, a lead trained on similar tasks could learn to anticipate the bottleneck and allocate compute accordingly. In our experiments the allocation is set through stage guidance rather than chosen by the lead, so we have not tested this directly, but it is a direction we believe is promising to pursue in future work.

Could we explicitly assign [planners] different areas to explore?

Even without predefined sub-roles, leads already divide exploration across planners. Each planner runs the reference executable ~7 times, so sixteen planners make 108 runs per task, 104 of them distinct, and their reports overlap less as the team grows (Appendix C.10). Prescribing areas like CLI flags, input formats, and error handling could help by specializing individual planners, though it may also bake task-specific structure into the prompt. Longer term, we think the lead could learn to divide the search space itself, and we view this as a promising direction for future work.

Why does implementation performance stop improving beyond four workers?

Two things change as the number of implementers grows.

  1. The lead's understanding of what the reference program does stays roughly the same, so each added implementer just gets a smaller piece of the same plan (each worker's assignment shrinks from 1,400 to 400 characters on average).
  2. The less capable Luna workers write more of the code (55% to 86% of edits), but the lead still checks the combined program against the reference only five or six times. The team may be working more in parallel without getting better at catching mistakes.

Planners, by contrast, leave the code with the more capable lead and widen what the team learns about the specification.


Thank you once again for the thoughtful questions and feedback, which helped us strengthen the fixed-size results and sharpen our discussion on compute allocation. We hope our responses address the concerns you raised, and we are happy to discuss any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Summary

We would like to thank all three reviewers for their feedback. Reviewers noted the importance and practicality of studying how to allocate inference compute within agent teams (LPx5, kzdB, pJJx), our controlled experimental design (LPx5, kzdB), and the guidance and detailed empirical picture our results offer on team design (kzdB, pJJx).

In response to the questions and concerns raised, we conducted substantial additional experiments to test the robustness and scope of our findings, and updated the revised PDF (with revisions highlighted in blue). We summarize the most important of these below.

  1. Budget-matched comparison between solo and teams (kzdB, pJJx). We compared each team with oracle best-of-N sampling of the solo agent at the same cost, using ten Solo PIV runs per model. Even against this upper bound, planning teams pairing a Sol or Terra lead with Luna workers beat the solo agent in a single run. When the oracle also picks the best of three team runs, every team except the Luna PIV team comes out ahead.
  2. Fixed-size scaling across runs and model pairings (LPx5, kzdB). We repeated the Sol + Luna fixed-size sweep two more times (three runs in total) and also added Terra + Luna and Luna + Luna pairings, with three runs each. In all three pairings, planning keeps improving from 2 to 16 workers (+2.1, +2.7, and +4.1 points) while implementation and verification plateau after 4, and Sol + 16 Luna planners comes within 1.2 points of the all-Sol PIV team at 69% lower cost.
  3. Additional Commit0 runs and per-library analysis (kzdB, pJJx). Across three runs for all five model setups, implementation guidance is the strongest single stage in 9 of 10 settings, with gains concentrated on the libraries where solo agents leave the most headroom.

We thank the reviewers again for their constructive comments, and we hope these revisions address the main concerns raised. Together, the new results strengthen our central finding that whether additional compute pays off depends on where a team spends it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.