- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to allocate inference-time compute in teams of coding agents. It frames multi-agent team design as a compute allocation problem and builds a controlled framework to study delegation stages, model choices, and team size. Experiments across ProgramBench and Commit0-Lite show that where workers help most depends on the task. On ProgramBench, a strong lead can benefit from cheaper planning workers but adding more implementation or verification workers brings limited gains, while delegating implementation is more effective on Commit0-Lite.
Critical strengths
- The paper addresses an important question of how to allocate additional inference compute in multi-agent teams. It examines where to delegate, which models to use, how many workers to add, and whether communication helps.
- The experiments give a detailed empirical picture of how team design affects performance and cost.
Critical weaknesses
- The main findings rely heavily on ProgramBench where discovering the behavior of a black-box program is central to the task. This may explain why planning workers are especially useful. However, Commit0-Lite only has 16 tasks and each configuration is run once, and also the strongest model already solves most tasks perfectly. It is therefore hard to tell how well the findings carry over to other coding tasks.
- The authors did not compare teams with single agents under the same compute budget. It is clear that multi-agent teams generally cost more, but the experiments do not establish if a single agent given a comparable budget for further exploration or repeated attempts could achieve similar gains.
Questions for authors
- On ProgramBench, are there types of tasks where implementation or verification workers help more than planning workers?
- Since the configurations use different budgets, how much of the improvement can be attributed to delegation rather than simply spending more compute? Do you have any comparisons with a solo agent at a similar cost? (referring to weakness 2).
- The analysis suggests that planning helps because workers gather more information about the target program. I am curious if there is trace evidence showing that different workers discover different behaviors, rather than repeat similar investigations?
- On Commit0-Lite, is the advantage of delegating implementation consistent across tasks or driven by a few libraries?
Response to Reviewer pJJx
Thanks for your detailed review and thoughtful questions. Please find our responses below, with updates marked in blue in the revised PDF.
To evaluate at matched compute levels, we compare against an oracle best-of-N approach, where a configuration is run N times and an oracle selects the highest-scoring attempt on each task. The cost is then N times that of a single run (new Appendix C.3, Figure 12). Because the oracle selects using ProgramBench's hidden test rewards, we view it as an upper bound on this sampling approach, since it selects perfectly and costs nothing.
We ran Solo PIV ten times on all 200 tasks for each of Sol, Terra, and Luna, roughly enough for solo sampling to span the cost of a single run of any team. We then fit saturating curves (reward=c−ANb, where c is the reward the curve approaches and b sets how quickly it gets there) through best-of-1 to best-of-10, which lets us estimate the solo reward at any budget. Teams have three runs each, so we also report best-of-3 for them.
We have now run every Commit0-Lite setup three times. Implementation remains the strongest single stage in nine of ten settings (five model setups, each with blocking and asynchronous scheduling), while planning is never clearly best (Appendix C.15). We agree that 16 tasks, with Sol near saturation (Solo PIV solves 10 of 16 tasks perfectly in all three runs), cannot establish generality on their own, so we use Commit0-Lite as a contrast to ProgramBench. When the repository skeleton and unit tests are given, the bottleneck shifts from discovering the specification to implementing it, and the stage that benefits most from extra workers shifts with it.
Across three Sol runs, implementation or verification beats planning by >5 points on 23 of 200 tasks, whereas planning beats both by >5 points on 47. No single label separates those 23 tasks, but the balance shifts with difficulty (Appendix C.8). Planning's lead over implementation is widest on easy and medium utilities (+8.6 vs. +1.0 lift on easy, +6.0 vs. +1.6 on medium). On hard tasks, which are large multi-subsystem programs such as duckdb, ffmpeg, php-src, and typst, planning's lift narrows (+2.7 vs. +1.9 for implementation) as implementation volume becomes a larger share of the work.
Yes. With sixteen planners, 104 of the 108 reference-executable runs per task are distinct commands, and pairwise similarity between planner reports drops from 0.26 with two planners to 0.17 with sixteen. Each planner runs the executable about seven times regardless of team size, so total coverage grows with the team (Appendix C.10).
It holds across most non-saturated libraries, though two hard libraries drive most of the mean lift. Across the ten settings and three runs, implementation beats planning by >1 point on 9 of 16 libraries, is within 1 point on 5 (all near ceiling), and trails on 2 (deprecated -7.4, minitorch -4.5). Gains are largest on babel (+56.7) and jinja (+44.6), where solo scores are lowest (Appendix C.15).
Thanks again for your feedback and questions, which prompted us to add the budget-matched comparison and per-library Commit0 breakdown. We hope our responses address your concerns, and we welcome any further discussion.