- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether test-time communication among LLM agents can improve open-ended discovery beyond independent parallel sampling. The authors introduce a shared-workspace communication framework in which homogeneous agents exchange intermediate findings, experimental outcomes, and reusable artifacts without predefined roles or centralized orchestration.
The central comparison is between a team of (k) communicating agents (team@k) and the best outcome from (k) independent agents (best@k). Experiments are conducted on ARC-AGI-3, Frontier-CS polyomino packing, and MNIST classifier compression, with additional negative results on Terminal-Bench 2.0.
On ARC-AGI-3, team@5 achieves an 8.0% solve rate compared with 2.2% for best@5, approximately matching the estimated performance of 33 independent agents. The communicating agents also achieve a packing score of 0.945 and produce a 1,957-byte MNIST classifier satisfying the benchmark's accuracy requirement.
The paper further introduces the concept of verified progress sharing and presents a simplified probabilistic analysis to illustrate how communication can accumulate discoveries across successive stages.
Critical strengths
- A relevant research question supported by substantial empirical results.
The paper addresses an important question in multi-agent LLM systems: whether communication provides benefits beyond simply increasing the number of independent attempts. The comparison between team@k and best@k is meaningful, and the reported gains on ARC-AGI-3 are substantial in several challenging games.
The results on polyomino packing and MNIST classifier compression are particularly interesting because they demonstrate that communicating agents can produce competitive optimization artifacts rather than merely improve performance on conventional reasoning benchmarks. These findings provide useful empirical evidence for the potential of collaborative test-time discovery.
- Useful analysis of communication dynamics and failure conditions.
The paper goes beyond reporting final benchmark scores by investigating token consumption, action efficiency, compute allocation, and intermediate discovery trajectories. The qualitative case studies provide concrete examples of agents adopting and extending one another's ideas.
I also appreciate that the authors acknowledge negative results under limited compute budgets and on Terminal-Bench 2.0. These observations make the empirical investigation more informative and suggest that communication benefits depend on task structure and the availability of useful intermediate feedback.
Critical weaknesses
- The experimental design does not sufficiently isolate the causal contribution of communication itself.
Although the paper aims to establish the benefits of test-time communication, the communicating condition introduces several mechanisms beyond information exchange.
In particular, the communication prompt in Appendix B.3 explicitly encourages agents to pursue distinct approaches, continuously execute experiments, record failures, challenge leading hypotheses, and switch algorithmic strategies after repeated unsuccessful attempts. These instructions may independently improve exploration efficiency, even without communication between agents.
Furthermore, the ARC-AGI-3 setup introduces synchronization constraints for communicating teams that are not applied to independent agents. Consequently, the two experimental conditions differ in their search protocol and execution dynamics, not merely in whether information is shared.
The paper does not provide a sufficiently controlled ablation that preserves the same exploration instructions, execution environment, and synchronization policy while selectively disabling cross-agent information exchange.
As a result, the current experiments demonstrate the effectiveness of the complete collaborative framework, but do not conclusively establish how much of the improvement is attributable to communication rather than the additional search guidance and coordination mechanisms.
This distinction is important because identifying the independent contribution of communication is central to the paper's stated research question.
- The evidence supporting general scaling and consistent discovery advantages remains limited.
The paper makes strong claims about the scaling benefits of communication, but the empirical evidence is concentrated in a relatively narrow set of configurations and successful cases.
On ARC-AGI-3, only four of the 25 games have nonzero final success rates under team@5. In fact, FT09 and LP85 account for 33 of the 40 successful team-game trials, meaning that a large fraction of the aggregate improvement comes from only two games. While these results are impressive, they do not necessarily establish a general scaling pattern across diverse tasks.
Moreover, the reported equivalence between team@3 and 13 independent agents, or team@5 and 33 independent agents, is estimated from a finite pool of 64 single-agent trials per game. The paper does not provide sufficiently detailed uncertainty estimates for these equivalences. The conclusion that communication gains compound with scale is also based primarily on a small number of team-size configurations.
Similar concerns apply to the longer-horizon experiments. The 72-hour polyomino experiments include only two team@4 trials per model, while the MNIST compression result is based on a single team@4 run. Reporting best-performing trajectories demonstrates the potential of collaborative discovery, but does not adequately characterize its expected performance or reliability.
Finally, although the paper reports output-token efficiency, output tokens alone do not capture total inference costs, including input tokens, repeated context processing, tool execution, and evaluator queries. Therefore, the strength of the compute-efficiency conclusions remains somewhat uncertain.
These limitations do not invalidate the reported achievements, but they weaken the broader claims regarding reproducible, generalizable, and increasingly advantageous communication at scale.
Questions for authors
Q1. Isolating the contribution of communication.
Could the authors introduce a communication-disabled control that preserves the same team size, exploration instructions, synchronization mechanisms, and computational resources, while preventing agents from accessing one another's findings?
Additionally, how does a single agent perform when given the structured exploration instructions from Appendix B.3 without the multi-agent communication component?
These comparisons would help determine whether the observed improvements are attributable specifically to information sharing or to the broader exploration framework.
Q2. Statistical reliability of the reported scaling advantage.
Can the authors provide confidence intervals or bootstrap estimates for the reported equivalences between team@3 and best@13, and between team@5 and best@33?
Since the independent-agent equivalences are estimated from only 64 single-agent trials per game, it would be useful to quantify uncertainty, particularly for games where no independent successes were observed.
Could the authors also report whether the improvement remains substantial when the two dominant games, FT09 and LP85, are excluded from the aggregate analysis?
Q3. More comprehensive compute-matched comparisons.
Figure 3 reports performance against cumulative output tokens, which is useful. However, communicating agents may also incur additional input-token costs from reading shared messages and artifacts.
Could the authors provide comparisons under matched total inference costs, including both input and output tokens, tool calls, evaluator queries, and wall-clock resource usage?
This would help establish whether the observed communication advantages remain substantial under a more complete accounting of test-time compute.
Q4. Robustness of the long-horizon discovery results.
Can the authors report the distribution of final scores across all polyomino trials rather than primarily the best-performing trajectories?
For MNIST compression, could the authors clarify how many queries were made to the sealed test oracle during the 96-hour search? Since the final classifier achieves 99.41% accuracy against a 99.40% qualification threshold, repeated adaptive evaluation on the same test set may influence artifact selection.
Additional independent runs or evaluation on a genuinely untouched holdout set would strengthen the reliability of the reported compression result.
Q5. Evidence for verified progress sharing as the underlying mechanism.
The paper argues that communication is particularly effective when agents have access to reliable intermediate verification. However, this hypothesis is mainly supported by comparisons across different benchmark environments.
Could the authors perform a controlled experiment within the same task that varies the availability, frequency, or reliability of intermediate verifier feedback while keeping the remaining conditions fixed?
Such an experiment would provide stronger causal evidence for the proposed mechanism and help distinguish the benefits of verified progress sharing from those of communication more generally.
9 comments
W1. [...] the communication prompt in Appendix B.3 explicitly encourages agents to pursue distinct approaches, continuously execute experiments, record failures, challenge leading hypotheses, and switch algorithmic strategies after repeated unsuccessful attempts. These instructions may independently improve exploration efficiency, even without communication between agents.
On packing and MNIST, single agents receive these same exploration instructions, so the instructions cannot explain the team advantage there. On ARC-AGI-3, removing these instructions does not remove the advantage. The synchronization control above also compares our anti-herding protocol with the minimal slot broadcast protocol, which drops the instructions to pursue distinct approaches, record evidence against the leading hypothesis, and adopt a peer's approach only after a measured improvement. On SB26 with five agents and no synchronization, both protocols match or exceed best@5's 7.8% (15% and 10%), and neither dominates. Even the minimal protocol beats independent agents, so the gain comes from sharing progress rather than from the wording of our rules.
Single agents also receive comparable exploration guidance through the continuation prompts. Whenever an agent ends its process, the harness re-prompts it. On packing and MNIST, this prompt is identical for single agents and teams, and it serves as a single-agent exploration protocol. These prompts and the MNIST solo protocol address the reviewer's list as follows.
| Reviewer's item | What single agents receive (code) |
|---|---|
| Continuously execute experiments | Re-prompted every time a turn ends. Packing: "Keep iterating and keep submitting for as long as you can make progress." ARC (task instruction, both settings): "Do not end your session just because you have a partial solution, a theory, a promising state, a shared note, or a dead end." |
| Switch strategies after unsuccessful attempts | Packing: "Hitting a plateau is not a reason to stop; it is a signal to try a structurally DIFFERENT approach." MNIST: "A plateau is not a reason to stop; it is a signal to change approach." |
| Pursue distinct approaches | Packing: "Try a fundamentally different algorithm or strategy than the ones you have already tried." MNIST: "try a structurally DIFFERENT way to shrink it rather than another small variation […]" |
| Record failures | Packing: the scorer logs every attempt and score for both settings (L1249–1250). MNIST: "Append each distinct intervention and its measured result to IDEAS.md outside /app/submission." |
| Challenge hypotheses | MNIST solo protocol: run "the cheapest experiment that could FALSIFY" an idea before investing in it. |
On MNIST, the single agent also received a dedicated solo protocol with more exploration guidance than the team's. Single agents received guidance that tells the agent to rotate among structurally distinct approaches, run cheap tests to falsify an idea, log every attempt, and change approach on a plateau. Even so, none of the four 96-hour independent runs beat the 2,461-byte human solution, whereas the team reached 1,957 bytes.
Q1. Could the authors introduce a communication-disabled control that preserves the same team size, exploration instructions, synchronization mechanisms, and computational resources [...]? Additionally, how does a single agent perform when given the structured exploration instructions from Appendix B.3 without the multi-agent communication component?
The requested communication-disabled team reduces to a single agent with structured exploration instructions, which is the reviewer's second request. For example, action synchronization without multiple agents amounts to nothing. On MNIST, single agents received a solo protocol covering every relevant instruction the reviewer lists, plus the identical continuation prompt. Even so, none of the four 96-hour independent runs beat the 2,461-byte human solution, whereas the team reached 1,957 bytes.
The remaining parts of the requested control are covered by best@k, our central baseline, and the two points above.
- Same team size and resources. Best@k uses k agents that cannot exchange findings. Both use the same model, maximum reasoning effort, longest context, per-agent resources, task, tools, and base filesystem (L969, L990–992). Teams do not receive extra compute compared with best@k. On ARC-AGI-3 and packing, a team shares one container with the same CPU and memory a single agent gets (Table 3, L1008–1020).
- Comparable exploration guidance. On packing and MNIST, single agents receive the identical continuation prompts, and on MNIST the solo protocol also covers every exploration instruction the reviewer lists.
W2. [...] the empirical evidence is concentrated in a relatively narrow set of configurations and successful cases.
We respectfully disagree that our evaluations are "concentrated in a relatively narrow set." A small set of configurations is not, by itself, a criterion for insufficient empirical evidence. The appropriate question is whether the experiments substantiate the contribution being claimed.
We test 15 configurations across four benchmarks and three model backends, and its long-horizon experiments alone use up to 4,000 agent-hours. If mode refers to model backbone, we use Claude Sonnet 4.6, Claude Opus 4.6, and GPT-5.6 Sol across benchmarks spanning interactive gameplay, algorithm design, ML compression, and terminal tasks. If experimental configurations are intended, each task is run under two to six configurations, varying team size, resource allocation, and search horizon, as well as running 20-64 trials for ARC and packing to ensure a robust conclusion.
| Task | Configs | Description |
|---|---|---|
| ARC-AGI-3 | 5 | Solo, team@3, and team@5; additionally, solo with 5× budget and team@5 with 0.2× budget per agent on two games |
| Packing | 6 | Sonnet, 3 hours: solo/team@3; Sonnet, 72 hours: solo/team@4; Opus, 72 hours: solo/team@4 |
| MNIST | 2 | Four independent 96-hour runs and one 96-hour team@4 run |
| TerminalBench | 2 | Four solo trials and two team@2 trials per task, across all 89 tasks |
Each task plays a distinct role. ARC-AGI-3 and packing provide the central comparisons in both width and depth. MNIST asks whether the benefits extend to realistic multi-day engineering. Moreover, our experiments also cover depth as they run much longer than most works. In MNIST alone, the team trial produces about 6M output tokens across four agents for 96 hours, while a single ARC-AGI-3 team@5 trial uses up to about 23M output tokens. On the other hand, Terminal-Bench deliberately shows that communication is not a silver bullet, and one counterexample suffices for that purpose without an exhaustive model sweep.
Q2. Can the authors provide confidence intervals or bootstrap estimates for the reported equivalences between team@3 and best@13, and between team@5 and best@33? [...] The conclusion that communication gains compound with scale is also based primarily on a small number of team-size configurations.
We have added 95% confidence intervals for ARC-AGI-3 to Figure 1 and Appendix C (Table 4, L1296–1307), discussed at L213–216. They use a parametric bootstrap with games held fixed, 20,000 replicates, and resampling of 64 single-agent trials and 20 team trials per game. The comparisons team@3 ≈ best@13 and team@5 ≈ best@33 describe approximate matches between observed solve rates, rather than strict equivalence.
| Configuration | Average | 95% CI |
|---|---|---|
| team@3 | 4.6% | 3.6–5.6% |
| team@5 | 8.0% | 6.8–9.2% |
| best@3 | 1.4% | 0.6–2.3% |
| best@5 | 2.2% | 0.9–3.4% |
| best@13 | 4.7% | 2.3–6.7% |
| best@33 | 8.1% | 3.8–10.6% |
For the same number of agents, team@k and best@k have non-overlapping intervals. In fact, under these CIs, we can claim that team@3 outperforms best@5 and team@5 outperforms best@13 reliably. For team@5, the 95% bootstrap interval for the matching number of independent agents starts at 18 agents (3.6×) and extends beyond our 64 single-agent trials. Therefore, we can reliably claim that communication gains compound with team size.
For games with zero independent successes, we bound these in the worst case for our claim. LP85 has 0 successes in 64 single-agent trials, so the 95% upper bound on the single-agent rate is 5.6% (Clopper–Pearson). Even at that bound, best@5 is at most 25%, while team@5 solves LP85 in 65% of trials (95% CI 41–85%).
We thank the reviewer for the detailed and constructive review, and for recognizing that the results are "impressive" and that the limitations "do not invalidate the reported achievements." The feedback led us to report test oracle use, confidence intervals, and full score distributions, which strengthened the paper. Our main answers are summarized below.
For reproducibility, we expanded Appendix B.3 with more prompt details (L1030–1207) and have updated our paper with an anonymous codebase (L186–187, https://anonymous.4open.science/r/test-time-communication-210D/) that keeps our main experiments fully reproducible.
Through additional experiments, we show that synchronization does not explain the improvement and that teams without synchronization still beat independent agents.
We implemented action synchronization because agents in a team run at different speeds because of API latency and uneven request throughput. Without pacing, an agent whose requests return faster can race several levels ahead. Synchronization keeps teammates on the same part of the task so that what they share stays relevant. However, synchronization grants no additional game actions or compute, and only shows each teammate's current level and action count (L1218–1228).
To study its effect, we ran teams with and without synchronization for 20 trials per configuration (Appendix C.1, Table 5, and in Section 3, L219–221). We synchronize five times per level to make any effect of synchronization easier to observe. We also vary the per-level action budget to vary the number of steps in between syncs. We also compare two protocols.
(500) and (1000) denote fixed per-level action budgets.
Overall, synchronization has no consistent direction of effect. It is better in two settings, worse in three, and tied in three. Yet, teams without synchronization still clearly beat independent agents. On FT09, the unsynchronized team@3 solves 60% of trials, against 25.9% for best@3 and 9.4% for a single agent (L119–120, L218–219). On SB26, the hardest game here, the unsynchronized team@5 matches or exceeds best@5's 7.8% (10–15%, Figure 9b). Therefore, our observed improvement does mainly come from information sharing rather than action synchronization.