- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TeamGym, an environment in which multiple AI agents can make decisions collaboratively and be trained to do so. In this setup, each agent initially possesses only a portion of the relevant information, called PIU (Pivotal Information Units).
Twelve datasets spanning the fields of law, finance, medicine, and software development were used. The PIUs are sufficient to reach a correct decision, with each individual unit being essential to that outcome. The authors evaluate the agents performance under varying conditions of information distribution and different modes of communication.
Across the models tested, the team-based success rate is on average 22.9 percentage points lower than that of a single model provided with all the necessary information ("Communication Gap"). If the agents first needs to extract this information from documents, the rate drops by an additional 19 percentage points ("Retrieval Gap"). The effectiveness of various communication methods depends on how the information is distributed. Two reinforcement learning experiments demonstrate that both decision quality and collaboration can be improved through training.
Critical strengths
-
The use of PIUs makes it possible to keep the underlying decision-making task and its requirements constant, while simultaneously investigating different information distributions, communication structures, and resource constraints. This is methodologically valuable, as many existing benchmarks compare different tasks or system configurations without controlling the distribution of necessary information in a similarly systematic way.
-
Evaluation across twelve source corpora, four application domains, and eight language models represents another strength. The use of real-world documents makes the setting more relevant than tasks that are purely synthetic.
-
The reinforcement learning (RL) experiments demonstrate that TeamGym can serve as an evaluation benchmark and also a training environment.
-
The authors explicitly state the limitations of their results, for example, the use of only a single training run per experiment and the limited generalizability of specific training effects.
Critical weaknesses
- The scientific added value compared to existing benchmarks is not clearly articulated. Works like HiddenBench and SILO-BENCH already investigate the collaboration of AI agents under distributed information, TeamGym distinguishes itself, through features like source-grounded PIUs, the separate analysis of communication and retrieval, and the capability for policy training. The specific new scientific insights gained from this approach could be highlighted more clearly.
Questions for authors
- In Section 4.4, a reversal in the ranking of different communication structures is reported in relation to varying PIU distributions. However, the number of agents also differs between the conditions "One common PIU" and "Extra participant without a PIU."
Could the observed ranking reversal be explained by the difference in team size and the associated communication possibilities and does this effect persist if the team size is held constant?
7 comments
Response to Reviewer ovUG (2/2): Q1
Q1. Could team size explain the reversal, and does it persist at fixed size?
The original named conditions use different constructions:
- One common PIU: one PIU is initially common to all participants, while the remaining PIUs stay local.
- Extra participant without a PIU: each PIU has a different private owner, with one additional participant assigned no PIU.
Under our original construction rules, these conditions have different numbers of communicating participants. Matching their rosters requires adjusting the PIU-assignment rules. The reported legal reversal (+3.9 versus −9.8 percentage points for Current-speaker selection minus Simultaneous responses) compares configurations that jointly specify information availability and team size. To investigate specific communication mechanisms, we added more tightly controlled intervention experiments that change coordinator queries or the messages reaching the final decision maker while holding the other settings fixed within each comparison.
Finance: query content with speaker selection held fixed (§4.3.1). We use 59 instances (30 FinDVer and 29 FinLongDocQA), pooling Qwen3-32B and Qwen3-235B-A22B in Multi-Agent Full-Context. The design compares One PIU per participant and All-but-one PIU concentration under Round robin and Coordinator-selected speaker. Within an episode, participants and the coordinator or separate final decision maker use the same model.
The interventions operate within Coordinator-selected speaker. We replay the original selected participants, their order, and response count, holding PIU assignment, private source material, model, resource budgets, and final-decision procedure fixed. We retain the original task-specific query or replace it with "It is your turn to contribute." Independently, we append or omit public PIU-category labels plus an instruction to report supporting evidence for each category. The labels identify topics without supplying private facts or answers; the original participant descriptions remain unchanged. We regenerate messages and final decisions for three repetitions, with 600-token message caps and a 1,200-token final rationale budget, and retain the original Round robin outcomes.
The replay interaction—the Round robin advantage over Coordinator-selected speaker under concentration minus its advantage under One PIU per participant—is +15.5 points with the original query and no additions. It falls to +5.9 with the labels and instruction, and becomes −5.4 or −9.3 when the task-specific query is removed without or with those additions. All three changes from the original-query replay are statistically significant; the resulting interactions are not. Substantive PIU communication also improves selectively under concentration. These results show how query content affects evidence sharing and decision success with speaker selection held fixed. The added-category treatment tests the labels and reporting instruction together.
SARA: source access and messages from participants without assigned PIUs (§4.3.2). We use 44 primary SARA instances (one additional instance is reserved for development), Qwen3-32B, Qwen3-235B-A22B, and GPT-5.5, with three repetitions. Participants receive their assigned PIUs explicitly, with or without added private source material. This differs from ordinary Multi-Agent Full-Context, where participants must retrieve their PIUs. Across One PIU per participant and All-but-one PIU concentration and five communication structures, adding source material changes the Simultaneous-versus-Coordinator PIU-distribution interaction from +10.6 to −5.1 points, a statistically significant −15.7-point change.
We then independently vary source-material access for participants assigned PIUs and those assigned none, keeping exact PIUs and public descriptions fixed. To test the contribution of the latter group's messages, we use All-but-one PIU concentration and Simultaneous responses, freeze all participant messages separately within each source-access condition, and regenerate only final decisions. The final decision maker sees either the complete transcript or only messages from the two PIU holders; those holders' messages and assignments remain unchanged.
When source material goes only to participants assigned no PIU, removing their messages improves success from 21.2% to 68.9%. Without added source material, filtering changes success from 69.9% to 68.9%. The +48.7-point interaction between source access and message removal is statistically significant. Thus, in this setting those messages can impair the final decision even while the evidence communicated by the PIU holders is retained.
Thank you again for helping us clarify TeamGym's scientific contribution and deepen the mechanistic analysis. We hope the revised explanations and additional controlled experiments address your concerns, and we would welcome any further feedback.
Thank you for the detailed response and the substantial new content, including the additional controlled studies. I will take these into account in my final assessment. I note that the ranking reversal is acknowledged to be confounded with team size and was not tested at fixed size
Response to Reviewer ovUG (1/2): W1
W1. Scientific contribution relative to HiddenBench and SILO-BENCH
Thank you for encouraging us to make the scientific added value clearer. We see two connected contributions: a procedure for constructing source-grounded PIUs and the corresponding distributed source observations, and the scientific findings enabled by those evidence controls. In the original submission, we devoted Appendix C.1, "Closest Benchmark Comparisons," to individual comparisons with HiddenBench and SILO-BENCH. We summarize that comparison below and explain more explicitly how our construction procedure enables the scientific findings.
PIU construction and source-document division enable controlled comparisons and systematic scientific analysis. This procedure (§3.1–3.2; Appendix F.2) supports comparisons with the same underlying decision requirements and evidence. Using these controls, we estimate separate logistic regressions for Multi-Agent PIU-Only and Multi-Agent Full-Context across 11 problem families and eight models, with problem-family and model fixed effects and standard errors clustered by problem instance. The PIU-distribution–communication-structure interactions are jointly significant in both conditions (p < .001), providing systematic evidence that relative protocol performance depends on the tested PIU distribution (§4.3). Appendices G.5–G.6 report probability-scale estimates and robustness checks with problem-instance fixed effects. These findings motivate the targeted mechanism studies below.
The comparison above draws on HiddenBench v4 and SILO-BENCH, which provide valuable settings for studying distributed collaboration. To make the scientific insights enabled by TeamGym clearer, we highlight three findings: