acceptodds
Under review as a conference paper at ICLR 2027

Scaling Discovery through Test-Time Communication

Abstract

Science advances not in isolation but through collaboration, yet existing agentic systems capture little of this. Whether communicating agents help remains an open question with mixed prior results. We show that test-time communication can substantially outperform independent parallel attempts on challenging tasks, where sharing a breakthrough can push the whole group forward. We first study the effect of scaling multi-agent test-time communication, where agents have no predefined roles and communicate via a shared directory, on ARC-AGI-3, a benchmark requiring novel problem solving. We find that a team of k communicating agents, team@k, matches the success rate of 4k independent agents, and this advantage grows with k, suggesting gains compound with scale. The effect is not merely efficiency: a task that no single agent can solve, a team of agents can solve reliably. Furthermore, these gains transfer to research-oriented tasks, given sufficient compute. On polyomino packing, communicating agents outperform best@k and exceed the prior best-known score. On MNIST classifier compression, a team of four communicating agents produced a 1,957-byte classifier achieving 99.4% test accuracy, smaller than both the best-known human solution and the best single-agent result. These gains are not unconditional. Independent agents may outperform communication when compute is limited or when a clear measure of progress is absent. However, under sufficient compute and clear verifier feedback, multi-agent communication consistently yields stronger results.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model, multi-agent systems, test-time discovery, test-time communication

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_31PSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether test-time communication among LLM agents can improve open-ended discovery beyond independent parallel sampling. The authors introduce a shared-workspace communication framework in which homogeneous agents exchange intermediate findings, experimental outcomes, and reusable artifacts without predefined roles or centralized orchestration.

The central comparison is between a team of (k) communicating agents (team@k) and the best outcome from (k) independent agents (best@k). Experiments are conducted on ARC-AGI-3, Frontier-CS polyomino packing, and MNIST classifier compression, with additional negative results on Terminal-Bench 2.0.

On ARC-AGI-3, team@5 achieves an 8.0% solve rate compared with 2.2% for best@5, approximately matching the estimated performance of 33 independent agents. The communicating agents also achieve a packing score of 0.945 and produce a 1,957-byte MNIST classifier satisfying the benchmark's accuracy requirement.

The paper further introduces the concept of verified progress sharing and presents a simplified probabilistic analysis to illustrate how communication can accumulate discoveries across successive stages.

Critical strengths
  1. A relevant research question supported by substantial empirical results.

The paper addresses an important question in multi-agent LLM systems: whether communication provides benefits beyond simply increasing the number of independent attempts. The comparison between team@k and best@k is meaningful, and the reported gains on ARC-AGI-3 are substantial in several challenging games.

The results on polyomino packing and MNIST classifier compression are particularly interesting because they demonstrate that communicating agents can produce competitive optimization artifacts rather than merely improve performance on conventional reasoning benchmarks. These findings provide useful empirical evidence for the potential of collaborative test-time discovery.

  1. Useful analysis of communication dynamics and failure conditions.

The paper goes beyond reporting final benchmark scores by investigating token consumption, action efficiency, compute allocation, and intermediate discovery trajectories. The qualitative case studies provide concrete examples of agents adopting and extending one another's ideas.

I also appreciate that the authors acknowledge negative results under limited compute budgets and on Terminal-Bench 2.0. These observations make the empirical investigation more informative and suggest that communication benefits depend on task structure and the availability of useful intermediate feedback.

Critical weaknesses
  1. The experimental design does not sufficiently isolate the causal contribution of communication itself.

Although the paper aims to establish the benefits of test-time communication, the communicating condition introduces several mechanisms beyond information exchange.

In particular, the communication prompt in Appendix B.3 explicitly encourages agents to pursue distinct approaches, continuously execute experiments, record failures, challenge leading hypotheses, and switch algorithmic strategies after repeated unsuccessful attempts. These instructions may independently improve exploration efficiency, even without communication between agents.

Furthermore, the ARC-AGI-3 setup introduces synchronization constraints for communicating teams that are not applied to independent agents. Consequently, the two experimental conditions differ in their search protocol and execution dynamics, not merely in whether information is shared.

The paper does not provide a sufficiently controlled ablation that preserves the same exploration instructions, execution environment, and synchronization policy while selectively disabling cross-agent information exchange.

As a result, the current experiments demonstrate the effectiveness of the complete collaborative framework, but do not conclusively establish how much of the improvement is attributable to communication rather than the additional search guidance and coordination mechanisms.

This distinction is important because identifying the independent contribution of communication is central to the paper's stated research question.

  1. The evidence supporting general scaling and consistent discovery advantages remains limited.

The paper makes strong claims about the scaling benefits of communication, but the empirical evidence is concentrated in a relatively narrow set of configurations and successful cases.

On ARC-AGI-3, only four of the 25 games have nonzero final success rates under team@5. In fact, FT09 and LP85 account for 33 of the 40 successful team-game trials, meaning that a large fraction of the aggregate improvement comes from only two games. While these results are impressive, they do not necessarily establish a general scaling pattern across diverse tasks.

Moreover, the reported equivalence between team@3 and 13 independent agents, or team@5 and 33 independent agents, is estimated from a finite pool of 64 single-agent trials per game. The paper does not provide sufficiently detailed uncertainty estimates for these equivalences. The conclusion that communication gains compound with scale is also based primarily on a small number of team-size configurations.

Similar concerns apply to the longer-horizon experiments. The 72-hour polyomino experiments include only two team@4 trials per model, while the MNIST compression result is based on a single team@4 run. Reporting best-performing trajectories demonstrates the potential of collaborative discovery, but does not adequately characterize its expected performance or reliability.

Finally, although the paper reports output-token efficiency, output tokens alone do not capture total inference costs, including input tokens, repeated context processing, tool execution, and evaluator queries. Therefore, the strength of the compute-efficiency conclusions remains somewhat uncertain.

These limitations do not invalidate the reported achievements, but they weaken the broader claims regarding reproducible, generalizable, and increasingly advantageous communication at scale.

Questions for authors

Q1. Isolating the contribution of communication.

Could the authors introduce a communication-disabled control that preserves the same team size, exploration instructions, synchronization mechanisms, and computational resources, while preventing agents from accessing one another's findings?

Additionally, how does a single agent perform when given the structured exploration instructions from Appendix B.3 without the multi-agent communication component?

These comparisons would help determine whether the observed improvements are attributable specifically to information sharing or to the broader exploration framework.

Q2. Statistical reliability of the reported scaling advantage.

Can the authors provide confidence intervals or bootstrap estimates for the reported equivalences between team@3 and best@13, and between team@5 and best@33?

Since the independent-agent equivalences are estimated from only 64 single-agent trials per game, it would be useful to quantify uncertainty, particularly for games where no independent successes were observed.

Could the authors also report whether the improvement remains substantial when the two dominant games, FT09 and LP85, are excluded from the aggregate analysis?

Q3. More comprehensive compute-matched comparisons.

Figure 3 reports performance against cumulative output tokens, which is useful. However, communicating agents may also incur additional input-token costs from reading shared messages and artifacts.

Could the authors provide comparisons under matched total inference costs, including both input and output tokens, tool calls, evaluator queries, and wall-clock resource usage?

This would help establish whether the observed communication advantages remain substantial under a more complete accounting of test-time compute.

Q4. Robustness of the long-horizon discovery results.

Can the authors report the distribution of final scores across all polyomino trials rather than primarily the best-performing trajectories?

For MNIST compression, could the authors clarify how many queries were made to the sealed test oracle during the 96-hour search? Since the final classifier achieves 99.41% accuracy against a 99.40% qualification threshold, repeated adaptive evaluation on the same test set may influence artifact selection.

Additional independent runs or evaluation on a genuinely untouched holdout set would strengthen the reliability of the reported compression result.

Q5. Evidence for verified progress sharing as the underlying mechanism.

The paper argues that communication is particularly effective when agents have access to reliable intermediate verification. However, this hypothesis is mainly supported by comparisons across different benchmark environments.

Could the authors perform a controlled experiment within the same task that varies the availability, frequency, or reliability of intermediate verifier feedback while keeping the remaining conditions fixed?

Such an experiment would provide stronger causal evidence for the proposed mechanism and help distinguish the benefits of verified progress sharing from those of communication more generally.

9 comments

We thank the reviewer for the detailed and constructive review, and for recognizing that the results are "impressive" and that the limitations "do not invalidate the reported achievements." The feedback led us to report test oracle use, confidence intervals, and full score distributions, which strengthened the paper. Our main answers are summarized below.

  1. Synchronization has no consistent direction of effect. The substantial team@k advantage over best@k persists without it.
  2. Single agents receive comparable exploration guidance through identical continuation prompts on packing and MNIST, and the MNIST solo protocol covers the reviewer's full list. On MNIST, single agents received more exploration guidance than teams, and they still never beat the human solution.
  3. Without the anti-herding instructions, a minimal protocol still beats independent agents on ARC-AGI-3 (SB26, 15% vs. 7.8% for best@5). We also expanded Appendix B.3 and supplied an anonymous codebase.
  4. Team@k beats best@k with non-overlapping 95% intervals. Three communicating agents beat five independent ones, and five beat thirteen.
  5. Across all long-horizon settings, none of the 88 independent runs beat the prior best. The best team did in every setting.
  6. On MNIST, the team's advantage holds at matched oracle use and fewer output tokens. At the same number of oracle queries as all four independent runs combined, the team's classifier was already 34% smaller than the best independent result, with 36% fewer output tokens.

For reproducibility, we expanded Appendix B.3 with more prompt details (L1030–1207) and have updated our paper with an anonymous codebase (L186–187, https://anonymous.4open.science/r/test-time-communication-210D/) that keeps our main experiments fully reproducible.


W1. [...] the ARC-AGI-3 setup introduces synchronization constraints for communicating teams that are not applied to independent agents. [...]

Through additional experiments, we show that synchronization does not explain the improvement and that teams without synchronization still beat independent agents.

We implemented action synchronization because agents in a team run at different speeds because of API latency and uneven request throughput. Without pacing, an agent whose requests return faster can race several levels ahead. Synchronization keeps teammates on the same part of the task so that what they share stays relevant. However, synchronization grants no additional game actions or compute, and only shows each teammate's current level and action count (L1218–1228).

To study its effect, we ran teams with and without synchronization for 20 trials per configuration (Appendix C.1, Table 5, and in Section 3, L219–221). We synchronize five times per level to make any effect of synchronization easier to observe. We also vary the per-level action budget to vary the number of steps in between syncs. We also compare two protocols.

  • Slot broadcast: Agents only claim slots and share a findings log (L1411–1413).
  • Anti-herding (our main protocol): Agents also declare an approach distinct from claimed slots, log evidence against the leading hypothesis, adopt a peer's approach only after a clearly better measured result, and keep a variation after adopting (L1042–1082).
ARC gameTeamProtocolNo syncSync
FT093-agentanti-herding60.0%80.0%
FT09 (500)3-agentanti-herding75.0%70.0%
SB263-agentslot broadcast0.0%0.0%
SB263-agentanti-herding0.0%0.0%
SB26 (1000)3-agentanti-herding30.0%20.0%
SB265-agentslot broadcast15.0%0.0%
SB265-agentanti-herding10.0%10.0%
SB26 (1000)5-agentanti-herding25.0%50.0%

(500) and (1000) denote fixed per-level action budgets.

Overall, synchronization has no consistent direction of effect. It is better in two settings, worse in three, and tied in three. Yet, teams without synchronization still clearly beat independent agents. On FT09, the unsynchronized team@3 solves 60% of trials, against 25.9% for best@3 and 9.4% for a single agent (L119–120, L218–219). On SB26, the hardest game here, the unsynchronized team@5 matches or exceeds best@5's 7.8% (10–15%, Figure 9b). Therefore, our observed improvement does mainly come from information sharing rather than action synchronization.

W1. [...] the communication prompt in Appendix B.3 explicitly encourages agents to pursue distinct approaches, continuously execute experiments, record failures, challenge leading hypotheses, and switch algorithmic strategies after repeated unsuccessful attempts. These instructions may independently improve exploration efficiency, even without communication between agents.

On packing and MNIST, single agents receive these same exploration instructions, so the instructions cannot explain the team advantage there. On ARC-AGI-3, removing these instructions does not remove the advantage. The synchronization control above also compares our anti-herding protocol with the minimal slot broadcast protocol, which drops the instructions to pursue distinct approaches, record evidence against the leading hypothesis, and adopt a peer's approach only after a measured improvement. On SB26 with five agents and no synchronization, both protocols match or exceed best@5's 7.8% (15% and 10%), and neither dominates. Even the minimal protocol beats independent agents, so the gain comes from sharing progress rather than from the wording of our rules.

Single agents also receive comparable exploration guidance through the continuation prompts. Whenever an agent ends its process, the harness re-prompts it. On packing and MNIST, this prompt is identical for single agents and teams, and it serves as a single-agent exploration protocol. These prompts and the MNIST solo protocol address the reviewer's list as follows.

Reviewer's itemWhat single agents receive (code)
Continuously execute experimentsRe-prompted every time a turn ends. Packing: "Keep iterating and keep submitting for as long as you can make progress." ARC (task instruction, both settings): "Do not end your session just because you have a partial solution, a theory, a promising state, a shared note, or a dead end."
Switch strategies after unsuccessful attemptsPacking: "Hitting a plateau is not a reason to stop; it is a signal to try a structurally DIFFERENT approach." MNIST: "A plateau is not a reason to stop; it is a signal to change approach."
Pursue distinct approachesPacking: "Try a fundamentally different algorithm or strategy than the ones you have already tried." MNIST: "try a structurally DIFFERENT way to shrink it rather than another small variation […]"
Record failuresPacking: the scorer logs every attempt and score for both settings (L1249–1250). MNIST: "Append each distinct intervention and its measured result to IDEAS.md outside /app/submission."
Challenge hypothesesMNIST solo protocol: run "the cheapest experiment that could FALSIFY" an idea before investing in it.

On MNIST, the single agent also received a dedicated solo protocol with more exploration guidance than the team's. Single agents received guidance that tells the agent to rotate among structurally distinct approaches, run cheap tests to falsify an idea, log every attempt, and change approach on a plateau. Even so, none of the four 96-hour independent runs beat the 2,461-byte human solution, whereas the team reached 1,957 bytes.


Q1. Could the authors introduce a communication-disabled control that preserves the same team size, exploration instructions, synchronization mechanisms, and computational resources [...]? Additionally, how does a single agent perform when given the structured exploration instructions from Appendix B.3 without the multi-agent communication component?

The requested communication-disabled team reduces to a single agent with structured exploration instructions, which is the reviewer's second request. For example, action synchronization without multiple agents amounts to nothing. On MNIST, single agents received a solo protocol covering every relevant instruction the reviewer lists, plus the identical continuation prompt. Even so, none of the four 96-hour independent runs beat the 2,461-byte human solution, whereas the team reached 1,957 bytes.

The remaining parts of the requested control are covered by best@k, our central baseline, and the two points above.

  • Same team size and resources. Best@k uses k agents that cannot exchange findings. Both use the same model, maximum reasoning effort, longest context, per-agent resources, task, tools, and base filesystem (L969, L990–992). Teams do not receive extra compute compared with best@k. On ARC-AGI-3 and packing, a team shares one container with the same CPU and memory a single agent gets (Table 3, L1008–1020).
  • Comparable exploration guidance. On packing and MNIST, single agents receive the identical continuation prompts, and on MNIST the solo protocol also covers every exploration instruction the reviewer lists.

W2. [...] the empirical evidence is concentrated in a relatively narrow set of configurations and successful cases.

We respectfully disagree that our evaluations are "concentrated in a relatively narrow set." A small set of configurations is not, by itself, a criterion for insufficient empirical evidence. The appropriate question is whether the experiments substantiate the contribution being claimed.

We test 15 configurations across four benchmarks and three model backends, and its long-horizon experiments alone use up to 4,000 agent-hours. If mode refers to model backbone, we use Claude Sonnet 4.6, Claude Opus 4.6, and GPT-5.6 Sol across benchmarks spanning interactive gameplay, algorithm design, ML compression, and terminal tasks. If experimental configurations are intended, each task is run under two to six configurations, varying team size, resource allocation, and search horizon, as well as running 20-64 trials for ARC and packing to ensure a robust conclusion.

TaskConfigsDescription
ARC-AGI-35Solo, team@3, and team@5; additionally, solo with 5× budget and team@5 with 0.2× budget per agent on two games
Packing6Sonnet, 3 hours: solo/team@3; Sonnet, 72 hours: solo/team@4; Opus, 72 hours: solo/team@4
MNIST2Four independent 96-hour runs and one 96-hour team@4 run
TerminalBench2Four solo trials and two team@2 trials per task, across all 89 tasks

Each task plays a distinct role. ARC-AGI-3 and packing provide the central comparisons in both width and depth. MNIST asks whether the benefits extend to realistic multi-day engineering. Moreover, our experiments also cover depth as they run much longer than most works. In MNIST alone, the team trial produces about 6M output tokens across four agents for 96 hours, while a single ARC-AGI-3 team@5 trial uses up to about 23M output tokens. On the other hand, Terminal-Bench deliberately shows that communication is not a silver bullet, and one counterexample suffices for that purpose without an exhaustive model sweep.

Q2. Can the authors provide confidence intervals or bootstrap estimates for the reported equivalences between team@3 and best@13, and between team@5 and best@33? [...] The conclusion that communication gains compound with scale is also based primarily on a small number of team-size configurations.

We have added 95% confidence intervals for ARC-AGI-3 to Figure 1 and Appendix C (Table 4, L1296–1307), discussed at L213–216. They use a parametric bootstrap with games held fixed, 20,000 replicates, and resampling of 64 single-agent trials and 20 team trials per game. The comparisons team@3 ≈ best@13 and team@5 ≈ best@33 describe approximate matches between observed solve rates, rather than strict equivalence.

ConfigurationAverage95% CI
team@34.6%3.6–5.6%
team@58.0%6.8–9.2%
best@31.4%0.6–2.3%
best@52.2%0.9–3.4%
best@134.7%2.3–6.7%
best@338.1%3.8–10.6%

For the same number of agents, team@k and best@k have non-overlapping intervals. In fact, under these CIs, we can claim that team@3 outperforms best@5 and team@5 outperforms best@13 reliably. For team@5, the 95% bootstrap interval for the matching number of independent agents starts at 18 agents (3.6×) and extends beyond our 64 single-agent trials. Therefore, we can reliably claim that communication gains compound with team size.

For games with zero independent successes, we bound these in the worst case for our claim. LP85 has 0 successes in 64 single-agent trials, so the 95% upper bound on the single-agent rate is 5.6% (Clopper–Pearson). Even at that bound, best@5 is at most 25%, while team@5 solves LP85 in 65% of trials (95% CI 41–85%).

Reviewer_aYyFOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies scaling test-time computation through independent instances of the same language model that are allowed to communicate with each other. These language-model based agents are given a basic communication protocol and can write to a shared directory. Crucially, no fixed hierarchy is enforced, leaving agents to self-organize their actions. The authors compare the performance of such agentic “teams” to independently operating agents on several benchmarks. Specifically, they compare team@k to best@k, where k is the number of agents. Results suggest that agentic teams have better scaling properties than individual agents in domains for which verifiable rewards are available. Furthermore, teams appear capable of solving task instances that remain out of reach for collections of individual agents. Reported results are mixed in the absence of verifiable rewards or for limited token budgets, where “communication overhead costs” appear detrimental.

Critical strengths

[significance] Finding scalable recipes has been one of the most fruitful strategies for modern machine learning progress. Scaling test-time compute through, e.g., reasoning or multiple, parallel samples, has shown impressive results, making agentic “teams” a logical extension. The authors fill in a useful gap in the literature by attempting to cleanly disentangle the benefits of collaboration from simply scaling the number of parallel agents.

[quality] The experimental setup is clear and well done. However, baselines and ablations are minimal (see weaknesses)

[clarity] The results are generally presented clearly. Yet, some sections could use work (see weaknesses)

Critical weaknesses

High-level comment: The idea and setup are simple (which is great!) – but severely under tested and ablated (see below). The core contributions all rely on empirical evaluation of the simple idea. However, these evaluations are so limited in scope that this author is not sufficiently convinced of the results.

[quality] Confidence intervals are missing everywhere.

[quality] The paper makes strong empirical claims, but only evaluates 1 or 2 modes per task. This is simply not enough for a main conference submission.

[quality] Many details about the independent-agent baseline are missing (see questions).

[significance] The first three contributions [l113-125] make unconditional statements. These are only partially qualified by contribution [4], and seem to conflict with the ending of the abstract “These gains are not unconditional. Independent agents may outperform communication when compute is limited or when a clear measure of progress is absent.” In light of this, I would suggest a cleared presentation of the facts.

Questions for authors
  1. The setup appears to rely heavily on a single prompt: what ablations were done to decide on this?

  2. [l158-160] “An agent [...] meaningful variations” : this is a pure prompt-based instruction. What checks were done to ensure any of this actually happens?

  3. What was the prompt and the setup for the single agents? The claim is that teams of agents that can communicate outperform single agents. The teams of agents had additional tools, e.g., scratch spaces, did the individual agents receive similar tools and instructions?

  4. One recurring argument for better “team” performance is improved diversity and exploration. How were the individual agents seeded? Did they simply get the same prompt?

  5. Related work: the field of “ad-hoc” teamwork is relevant to this work [1] as is this recent work on agentic collaboration gaps [2]. The latter is a nice contrast to the results presented here. Other work worth looking into is [3], which explores different failure modes of agentic systems.

  6. [l955] “Do not write propose whose only purpose is to make peers copy you” – Could you please explain this?

  7. Any reason no open-source models were used to ensure future reproducibility?

  8. Small nit, feel free to ignore: currently has “Under review as conference paper at ICLR 2026”

  9. The future work section (Appendix F), largely provides a summary of the work, only l1579-l1583 appear forward looking?

[1] Mirsky, Reuth, et al. "A survey of ad hoc teamwork research." European conference on multi-agent systems. Cham: Springer International Publishing, 2022.

[2] Davidson, Tim R., et al. "The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation." arXiv preprint arXiv:2511.02687 (2025).

[3] Cemri, Mert, et al. "Why do multi-agent LLM systems fail?, 2025." NeurIPS (2025).

9 comments

Thank you for your careful and constructive feedback, which helped us sharpen the scope of our claims and strengthen the empirical evaluation. We address each concern below.

The idea and setup are simple (which is great!) – but severely under tested and ablated (see below). [...] these evaluations are so limited in scope

The paper makes strong empirical claims, but only evaluates 1 or 2 modes per task. [...]

We thank the reviewer for recognizing the value of simplicity within our setup. However, we respectfully disagree that our evaluations are "limited in scope." One or two modes per task is not, by itself, a criterion for insufficient empirical evidence. The appropriate question is whether the experiments substantiate the contribution being claimed.

We test 15 configurations across four benchmarks and three model backends, and its long-horizon experiments alone use up to 4,000 agent-hours. If mode refers to model backbone, we use Claude Sonnet 4.6, Claude Opus 4.6, and GPT-5.6 Sol across benchmarks spanning interactive gameplay, algorithm design, ML compression, and terminal tasks. If experimental configurations are intended, each task is run under two to six configurations, varying team size, resource allocation, and search horizon, as well as running 20-64 trials for ARC and packing to ensure a robust conclusion.

TaskConfigsDescription
ARC-AGI-35Solo, team@3, and team@5; additionally, solo with 5× budget and team@5 with 0.2× budget per agent on two games
Packing6Sonnet, 3 hours: solo/team@3; Sonnet, 72 hours: solo/team@4; Opus, 72 hours: solo/team@4
MNIST2Four independent 96-hour runs and one 96-hour team@4 run
TerminalBench2Four solo trials and two team@2 trials per task, across all 89 tasks

Each task plays a distinct role. ARC-AGI-3 and packing provide the central comparisons in both width and depth. MNIST asks whether the benefits extend to realistic multi-day engineering. Moreover, our experiments also cover depth as they run much longer than most works. In MNIST alone, the team trial produces about 6M output tokens across four agents for 96 hours, while a single ARC-AGI-3 team@5 trial uses up to about 23M output tokens. On the other hand, Terminal-Bench deliberately shows that communication is not a silver bullet, and one counterexample suffices for that purpose without an exhaustive model sweep.

Confidence intervals are missing everywhere.

We agree and have added 95% confidence intervals for ARC-AGI-3 to Figure 1 and Appendix C (Table 4, L1296–1307), discussed at L213–216. ARC estimates success probabilities from 64 single-agent and 20 team trials per game (L942–944), so uncertainty belongs alongside those rates.

ConfigurationAverage95% CI
team@34.6%3.6–5.6%
team@58.0%6.8–9.2%
best@31.4%0.6–2.3%
best@52.2%0.9–3.4%
best@134.7%2.3–6.7%
best@338.1%3.8–10.6%

For the same number of agents, team@k and best@k have non-overlapping intervals for both k = 3 and k = 5. In fact, under these CIs, we can claim that team@3 outperforms best@5 and team@5 outperforms best@13 reliably.

However, for packing and MNIST, we report the best discovered solution, because these are discovery tasks. When searching for a faster GPU kernel or a tighter mathematical bound, only the best solution ever found matters, and the average can hide the rare outcomes that move the frontier. This follows the reporting convention of prior discovery work such as AlphaEvolve (Novikov et al., 2025) and TTT-Discover (Yuksekgonul et al., 2026). The run-level distribution supports this framing. Below are the final 3-hour packing scores for Sonnet 4.6, the setting with the most trials.

Packing (3 hr)nMeanMedian90%Max
Solo600.7890.7920.8370.883
Best@3-0.8260.8280.8810.883
Team@3200.8220.8190.8880.945

Both setups use 60 agents in total. The team's 90th-percentile run (0.888) beats the best of all 60 solo runs (0.883), so at least two teams exceed every solo run, and the best team surpasses the prior best-known score of 0.894. While mean performance is similar, the advantage lies in the strongest discovered solutions, which is exactly what discovery depends on.

Q9. The future work section (Appendix F), largely provides a summary of the work, only l1579-l1583 appear forward looking?

Thank you, we have renamed the section and expanded its forward-looking part. Appendix F is now Concluding Remarks and Future Directions. After a short recap, it discusses communication under sparse, noisy, or subjective feedback (L1889–1891), reward hacking that can spread between communicating agents (L1891–1895), and how roles, model diversity, topology, and orchestration should be designed (newly added in L1896–1902).

The first three contributions make unconditional statements. [...] seem to conflict with the ending of the abstract "These gains are not unconditional. [...]" In light of this, I would suggest a cleared presentation of the facts.

The contributions were written to agree with the abstract. Each bold claim is followed by its benchmark and setting. Contribution 2 is specific to FT09 and LP85 (L118–121), and Contribution 3 to packing and MNIST (L122–126). Contribution 4 is the qualification of the first three. It states that the benefits depend on verifiable progress and sufficient compute, and that team@k pays a coordination tax at low budgets (L127–130). The paper is transparent that communication has conditions and report where communication loses:

  • The coordination tax. Independent agents lead at low compute (Figure 4, L250–254).
  • The budget-splitting ablation. A team that divides one agent's budget loses to that single agent (Figure 5, L269–289).
  • Terminal-Bench 2.0. Communication fails to beat pass@2 here (L105–111, Table 7).

Per reviewer's feedback, however, we have now stated the condition in the bold text itself. Contribution 1 opens with "Given sufficient compute" (L113) and reports its 95% confidence intervals (L114–115).

Many details about the independent-agent baseline are missing (see questions).

We have added specifications of what each agent receives, covering prompts, continuation prompts, and tools (Appendix B.1, L966–968, and B.3, L1030–1207), summarized in Section 2 (L188–191).

Q1. The setup appears to rely heavily on a single prompt: what ablations were done to decide on this?

We did not tune the prompt and kept one protocol fixed by design, because tuning it per benchmark would confound communication with prompt engineering. The same protocol is used unchanged on ARC-AGI-3 and polyomino packing (L362–363), and the MNIST variant adds rules for sharing CPU and GPU.

In a new experiment (performed for sync ablation), we compared our protocol with a much lighter one on ARC-AGI-3 (Appendix C.1, Table 5, L1392–1422) with 20 trials per configuration.

  • Broadcast: Agents only claim slots and share a findings log.
  • Anti-herding (our main protocol): Agents also declare an approach distinct from claimed slots, log evidence against the leading hypothesis, adopt a peer's approach only after a clearly better measured result, and keep a variation after adopting (L1042–1082).
SB26Broadcast, no syncBroadcast, syncAnti-herding, no syncAnti-herding, sync
3 agents0%0%0%0%
5 agents15%0%10%10%

With five agents and no synchronization, both protocols match or exceed best@5's 7.8%, and neither dominates. (Sync paces teammates so they stay on the same level, Appendix B.4.) Even the minimal protocol beats independent agents, so the gain comes from sharing progress rather than from the wording of our rules. (For a complete table of results, see our response to W1 of reviewer 31PS.)

Q2. [l158-160] "An agent [...] meaningful variations" : this is a pure prompt-based instruction. What checks were done to ensure any of this actually happens?

Separation is enforced by the filesystem. Agents race to create numbered slot directories with an atomic mkdir, which acts as a lock, so no two agents can hold the same slot (L142–143, L1049–1051). Each agent declares its approach in its slot and reads the others' declarations, and if a lower-numbered slot already has the same approach, the agent is told to change its own (L1053–1055). Every agent therefore starts in a distinct slot with a private workspace.

Beyond that, we do not ensure diversity. Whether an agent adopts only after a clearly better result and keeps a meaningful variation is left to the agent. Agents can still converge, and our analysis of verified progress sharing states this limitation directly (Appendix E, L1766–1770). We believe that maintaining diversity during test-time communication is an important avenue of research.

Q6. [l955] "Do not write propose whose only purpose is to make peers copy you" – Could you please explain this?

"propose" should read "prose." The instruction asks agents to report evidence objectively rather than argue for their approach, because persuasion without evidence can make agents herd toward a single approach.

Q7. Any reason no open-source models were used to ensure future reproducibility?

We have updated our paper with an anonymous codebase (L186–187, https://anonymous.4open.science/r/test-time-communication-210D/) that keeps our main experiments fully reproducible. The repo provides the task environments, harness, agent prompts, and model configurations, and the evaluated models are publicly available through GitHub Copilot.

Q8. currently has "Under review as conference paper at ICLR 2026"

Thank you for catching this! We have fixed it to the correct 2027 style in our revision.

Q3. What was the prompt and the setup for the single agents? [...] The teams of agents had additional tools, e.g., scratch spaces, did the individual agents receive similar tools and instructions?

Yes. Single agents receive the same tools and comparable instructions, and on MNIST they receive more exploration guidance than teams. Tools and resources are identical or favor independent agents.

  • Both use the same per-agent resources, task, tools, and base filesystem (L969, L990–992).
  • The tools are bash, file viewing, search, and patching. Copilot's subagent tools are disabled for every agent (L966–968).
  • The team's "scratch spaces" are ordinary directories that a single agent can equally create.
  • Teams do not receive extra compute compared with best@k. On ARC-AGI-3 and packing, a team shares one container with the same CPU and memory a single agent gets (Table 3, L1008–1020).

Single agents also receive comparable exploration guidance through the continuation prompts. Whenever an agent ends its process, the harness re-prompts it. On packing and MNIST, this prompt is identical for single agents and teams, and it serves as a single-agent exploration protocol.

  • Packing: "Hitting a plateau is not a reason to stop; it is a signal to try a structurally DIFFERENT approach … Try a fundamentally different algorithm or strategy than the ones you have already tried."
  • MNIST: "try a structurally DIFFERENT way to shrink it rather than another small variation … Append each distinct intervention and its measured result to IDEAS.md outside /app/submission."
  • On ARC-AGI-3, both conditions share the rule "Do not end your session just because you have a partial solution, a theory, a promising state, a shared note, or a dead end."

On MNIST, single agents received guidance that tells the agent to rotate among structurally distinct approaches, run cheap tests to falsify an idea, log every attempt, and change approach on a plateau. Even so, none of the four 96-hour independent runs beat the 2,461-byte human solution, whereas the team reached 1,957 bytes.

All prompts appear in Appendix B.3 (L1030–1207), and the task instructions and MNIST team variant are in the released code (L186–187).

Q4. One recurring argument for better "team" performance is improved diversity and exploration. How were the individual agents seeded? Did they simply get the same prompt?

Yes, independent agents receive the same prompt, as in standard best@k. Each trial runs in a fresh container and shares no files or messages with other trials (L972–973).

We do not argue that improved diversity explains better team performance. Our explanation is verified progress sharing. Here, some diversity is necessary, but it is not where the team advantage comes from. Independent agents are already as diverse as a team. What a team adds is gathering on verified breakthroughs and continuing from there (L98–104, Proposition 1). Consider kk agents searching a binary tree for a leaf at depth mm, with mm steps each. At every step, each agent picks a random child with equal chance and can verify whether it chose correctly.

k=5k=5, m=10m=10DiverseGathers at the verified nodeSuccess
Single attemptNoYes or no2−m2^{-m} = 0.1%
best@kkYesNo≤k⋅2−m\le k \cdot 2^{-m} = 0.5%
team@kkYesYes(1−2−k)m(1-2^{-k})^m = 73%

The above example depicts very different outcomes despite the same diversity. Diversity lets some agent find the right branch at each level, which is why our protocol encourages it and why herding hurts (L1766–1770). Gathering is what turns those finds into cumulative progress.

We would be grateful if the reviewer could point us to the passage that suggests improved diversity drives team performance, so that we can revise its wording to state our explanation consistently.

Q5. Related work: the field of "ad-hoc" teamwork is relevant to this work [1] as is this recent work on agentic collaboration gaps [2]. The latter is a nice contrast to the results presented here. Other work worth looking into is [3], which explores different failure modes of agentic systems.

Thank you for these references. We have added them to the Related Work section in Appendix A, L821–825 and L889–891.

  • Mirsky et al. also asks agents to collaborate without prior coordination, but there an agent must model unknown teammates, while our agents are identical and LLM-based.
  • Davidson et al., 2026 finds that strong models often degrade when paired. We agree this is a useful contrast. Our results suggest that whether agents can verify progress is one condition that separates the two outcomes (L831–834).
  • Cemri et al. attribute most multi-agent failures to system design, inter-agent misalignment, and inadequate task verification (L822–824). The last is consistent with our account.

Comments on the paper

7 comments

We thank the reviewer for their valuable feedback. We ran the synchronization control the reviewer suggested, and it shows that the team advantage persists without synchronization. We also added the requested framework figure and full prompts. Several concerns are already addressed in the manuscript, and we point to the relevant lines below.

Also, only the team group in ARC used action synchronization, so the observed improvement cannot be directly attributed to information sharing alone.

explain whether ARC's synchronization arrangement affects the comparison. A small-scale synchronization control can help explain this.

Through additional experiments, we show that synchronization does not explain the improvement and that teams without synchronization still beat independent agents.

We implemented action synchronization because agents in a team run at different speeds because of API latency and uneven request throughput. Without pacing, an agent whose requests return faster can race several levels ahead. Synchronization keeps teammates on the same part of the task so that what they share stays relevant. However, synchronization grants no additional game actions or compute, and only shows each teammate's current level and action count (L1218–1228).

To study its effect, we ran teams with and without synchronization for 20 trials per configuration (Appendix C.1, Table 5, and in Section 3, L219–221). We synchronize five times per level to make any effect of synchronization easier to observe. We also vary the per-level action budget so vary the number of steps in between syncs.

ARC gameTeamProtocolNo syncSync
FT093-agentanti-herding60.0%80.0%
FT09 (500)3-agentanti-herding75.0%70.0%
SB263-agentslot broadcast0.0%0.0%
SB263-agentanti-herding0.0%0.0%
SB26 (1000)3-agentanti-herding30.0%20.0%
SB265-agentslot broadcast15.0%0.0%
SB265-agentanti-herding10.0%10.0%
SB26 (1000)5-agentanti-herding25.0%50.0%

(500) and (1000) denote fixed per-level action budgets. Slot broadcast is a minimal protocol that keeps only slot claims and a shared findings log (L1411–1413), whereas anti-herding (our main protocol) instructs agents to declare an approach distinct from claimed slots, log evidence against the leading hypothesis, adopt a peer's approach only after a clearly better measured result, and keep a variation after adopting (L1042–1082).

Overall, synchronization has no consistent direction of effect. It is better in two settings, worse in three, and tied in three. Yet, teams without synchronization still clearly beat independent agents. On FT09, the unsynchronized team@3 solves 60% of trials, against 25.9% for best@3 and 9.4% for a single agent (L119–120, L218–219). On SB26, the hardest game here, the unsynchronized team@5 matches or exceeds best@5's 7.8% (10–15%, Figure 9b). Therefore, our observed improvement does mainly come from information sharing rather than action synchronization.

The architecture is close to shared-memory methods like CORAL, so method novelty is limited.

We do not claim that our communication protocol is our novelty. Section 1 introduces "a minimal form of test-time communication" (L80–85). The related-work section states that we "do not study orchestration or multi-agent harness optimization" and use "a fixed shared-workspace scaffold … to isolate communication from parallel exploration" (L886–889).

Our contribution is a controlled answer to a question on which prior work is divided (L66–71). When, and why, does communication beat independent parallel attempts? The paper provides the following.

  • Compute-matched comparisons at scale. We ran 64 single-agent and 20 team trials per game, on all 25 ARC-AGI-3 games, with 95% confidence intervals (Table 4). Team@k and best@k have non-overlapping intervals at both k = 3 and k = 5, and team@3 outperforms best@5.
  • Scaling with team size. The point-estimate multiplier grows from 4.3× (team@3) to 6.6× (team@5).
  • Failure conditions. These are the coordination tax, the budget-splitting ablation, and Terminal-Bench 2.0.
  • A mechanism. Verified progress sharing is formalized in Proposition 1.

The comparison with CORAL shows why a minimal protocol is worth studying. The prior best packing score of 0.894 was CORAL's (Qu et al., 2026), obtained with four Opus 4.6 agents (L360–361, L950). Our minimal protocol with Copilot harness reached 0.945 with three Sonnet 4.6 agents in three hours, achieving state-of-the-art results with less capable models.

Figure 2 is a concept sketch, and the paper still lacks an overall framework figure showing how the shared records, scoring, and adoption of approaches connect.

Thank you for this suggestion. We have added a new Figure 3 (L144–161) that shows the overall framework and flow. It shows the agent lanes, the communication log, how measured results feed adoption, and how the team's best result passes between agents, using logs from the MNIST team@4 run.

The polyomino packing results only show the best run trajectory under each setting, and the MNIST experiment only has one team trial, so whether the long-term effect is stable is still unclear;

The long-term effect holds in every long-horizon setting we ran. None of the 88 independent runs across packing and MNIST exceeded the prior best, while the best team did so in every setting.

SettingModelBest teamBest independentPrior best
Packing, 3 h (20 teams vs. 60 solo)Sonnet 4.60.9450.8830.894
Packing, 72 h (2 teams vs. 12 solo)Sonnet 4.60.9100.8910.894
Packing, 72 h (2 teams vs. 12 solo)Opus 4.60.9220.8930.894
MNIST, 96 h (1 team vs. 4 solo)GPT-5.6 Sol1,957 B3,160 B2,461 B

This covers two tasks and three models, and in both 72-hour settings, the independent side even had more agents (12 solo vs. 2 teams of 4).

Furthermore, we deliberately report the best run because these are discovery tasks, where a single breakthrough matters even when other attempts are unsuccessful. For instance, when searching for a faster GPU kernel or a tighter mathematical bound, only the best solution ever found matters, and the average can hide the rare outcomes that move the frontier. Communicating agents found a packing program scoring 0.945 and a 1,957-byte classifier that clears 99.4% on the sealed test set, and one verified instance settles that such a solution exists. Reporting the best discovered solution follows the convention of AlphaEvolve (Novikov et al., 2025) and TTT-Discover (Yuksekgonul et al., 2026), and it is how CORAL reported the 0.894 packing record we surpass.

The long-horizon run distribution supports the same conclusion. Below are the final 3-hour packing scores for Sonnet 4.6, the setting with the most trials.

Packing (3 hr)TrialsMeanMedian90%Max
Solo600.7890.7920.8370.883
Best@3-0.8260.8280.8810.883
Team@3200.8220.8190.8880.945

The team's 90th-percentile run (0.888) beats the best of all 60 solo runs (0.883), so at least two of the 20 teams exceed every solo run, and the best team (0.945) surpasses the prior best of 0.894. Mean performance is similar for team@3 and best@3 (0.822 vs. 0.826), so the advantage lies in the strongest discovered solutions, which is exactly what discovery depends on.

On MNIST, we agree that one team run cannot measure run-to-run variance. The run below does show that the team's lead is sustained and does not come from extra budget. Each test oracle query returns only aggregate test accuracy, so we compare the team with the four independent runs combined at matched query counts (Appendix C.3, Table 6).

RunOracle queriesOutput tokensBest size
All four independent runs, final (96 h)6867.29M3,160 B
Team@4 (3.7 h)1000.65M3,177 B
Team@4 (49.9 h)6864.66M2,076 B
Team@4, final1,6145.85M1,957 B

At the same 686 queries, the team's classifier was already 34% smaller than the best independent one, with 36% fewer output tokens. After only 100 queries, it was within 1% of best@4's final size. The improvements also came from all four agents rather than one (Appendix D.2). Thus, although one run cannot establish variance, the team's lead held for nearly the entire 96 hours rather than coming from a single lucky jump (L416–417), and packing reproduces the effect in all three of its settings.

The result differences across ARC-AGI-3, polyomino packing, MNIST compression, and Terminal-Bench 2.0 are not enough to prove that dense verification feedback is the reason the team wins.

We do not claim verification is the sole reason teams win. Within each task, we present evidence in our results and qualitative analysis that show the verifier scores guided the agents. In particular, both case studies in Appendix D trace the final solutions through successive verified improvements contributed by different agents.

Our claim is that verified progress sharing is an important factor, supported by a formal model (Proposition 1) and by qualitative traces showing how agents build on measured improvements. We have revised the wording to say this explicitly (L832–834).

We highlight the factor of verified progress sharing because it receives much less attention in prior explanations, which emphasize decomposition and parallelism:

  • Kim et al. (2025) find that multi-agent coordination helps on parallelizable tasks (about +81% on Finance-Agent) but degrades every multi-agent variant they test on sequential PlanCraft, by 39–70%.
  • Gu et al. (2025) argue that divide-and-conquer is what LLM multi-agent systems need.
  • Anthropic (2025) (Hadfield et al., How We Built Our Multi-Agent Research System, Anthropic Engineering Blog, June 2025) attributes the gains of its multi-agent research system to parallel, breadth-first exploration. Token usage alone explains about 80% of performance variance, and the authors note that most coding tasks involve fewer truly parallelizable tasks than research.

Verified progress sharing provides a complementary explanation. When agents can check which partial solution is better, each verified improvement becomes everyone's new starting point, and a different agent can supply the next breakthrough. Consider kk agents searching a binary tree for a leaf at depth mm, with mm steps each, where each agent picks a random child at every step and can verify whether it chose correctly. Independent agents succeed with probability at most k⋅2−mk \cdot 2^{-m}, while a team that gathers at each verified node succeeds with probability (1−2−k)m(1-2^{-k})^m. For k=5k=5 and m=10m=10, that is 0.5% vs. 73%, with the same diversity. We see our explanation as an addition to the decomposition account, not a replacement for it.

Please list the actual task prompts, extra instructions, and instructions for continuing runs used by each group

We have updated our paper with an anonymous codebase (L186–187, https://anonymous.4open.science/r/test-time-communication-210D/) that keeps our main experiments fully reproducible. The repo provides the task environments, harness, agent prompts, and model configurations, and the evaluated models are publicly available through GitHub Copilot. The table below summarizes what each setting receives.

TaskSingle agent receivesTeam agent receivesContinuation (both conditions)
ARC-AGI-3Task instructionRole line + anti-herding protocol + task instruction + sync noteRe-prompt until the first valid action
PackingTask instructionRole line + anti-herding protocol + task instructionIdentical "beat your best; try a structurally different approach" prompt, up to 1,000×
MNISTRole line + solo protocol + task instructionRole line + team protocol + task instructionIdentical "beat your best; change approach on a plateau" prompt, up to 1,000×

The continuation prompts, which are identical for both conditions on packing and MNIST, already give single agents comparable exploration guidance. On MNIST, the single agent also received a dedicated solo protocol with more exploration guidance than the team's.

  • Packing: "Hitting a plateau is not a reason to stop; it is a signal to try a structurally DIFFERENT approach … Try a fundamentally different algorithm or strategy than the ones you have already tried."
  • MNIST: "try a structurally DIFFERENT way to shrink it rather than another small variation … Append each distinct intervention and its measured result to IDEAS.md outside /app/submission."
  • On ARC-AGI-3, both conditions share the rule "Do not end your session just because you have a partial solution, a theory, a promising state, a shared note, or a dead end."

On MNIST, single agents received guidance that tells the agent to rotate among structurally distinct approaches, run cheap tests to falsify an idea, log every attempt, and change approach on a plateau. Even so, none of the four 96-hour independent runs beat the 2,461-byte human solution, whereas the team reached 1,957 bytes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.