acceptodds
Under review as a conference paper at ICLR 2027

TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information

Abstract

Many real-world decisions require teams to integrate information distributed across participants. Yet AI agents are still primarily trained and benchmarked in isolation. We introduce TeamGym, an environment for evaluating and training language model-based agents on team decision-making problems constructed from real-world legal, finance, medical, and software engineering corpora. TeamGym introduces a method to identify a minimal set of facts, each element of which is necessary and all of which are collectively sufficient for a correct final decision, which we call pivotal information. This enables TeamGym to both control the distribution of pivotal information and track how well agents identify and integrate pivotal information when making team decisions. Examining 8 language models on 12 source corpora, we find that current AI agents often fail to effectively share information: Distributing pivotal information across participants lowers success rate by 22.9 percentage points on average relative to giving the same information to a single model (a communication gap). We also document that participants do not effectively retrieve decision-relevant information on their own: A team’s success rate is lowered by a further 19.9 points when participants must identify and extract pivotal information from source material rather than receiving it directly (a retrieval gap). To help remedy these deficits, we use TeamGym to train policies through reinforcement learning on team decision loss. Across two policy-training experiments, we find that training improves final-decision quality and induces emergent coordination in participation, speaker selection, and communication, even though the reward evaluates only the final decision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, language-model agents, team decision making, distributed information, communication, information retrieval, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ovUGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TeamGym, an environment in which multiple AI agents can make decisions collaboratively and be trained to do so. In this setup, each agent initially possesses only a portion of the relevant information, called PIU (Pivotal Information Units).

Twelve datasets spanning the fields of law, finance, medicine, and software development were used. The PIUs are sufficient to reach a correct decision, with each individual unit being essential to that outcome. The authors evaluate the agents performance under varying conditions of information distribution and different modes of communication.

Across the models tested, the team-based success rate is on average 22.9 percentage points lower than that of a single model provided with all the necessary information ("Communication Gap"). If the agents first needs to extract this information from documents, the rate drops by an additional 19 percentage points ("Retrieval Gap"). The effectiveness of various communication methods depends on how the information is distributed. Two reinforcement learning experiments demonstrate that both decision quality and collaboration can be improved through training.

Critical strengths
  1. The use of PIUs makes it possible to keep the underlying decision-making task and its requirements constant, while simultaneously investigating different information distributions, communication structures, and resource constraints. This is methodologically valuable, as many existing benchmarks compare different tasks or system configurations without controlling the distribution of necessary information in a similarly systematic way.

  2. Evaluation across twelve source corpora, four application domains, and eight language models represents another strength. The use of real-world documents makes the setting more relevant than tasks that are purely synthetic.

  3. The reinforcement learning (RL) experiments demonstrate that TeamGym can serve as an evaluation benchmark and also a training environment.

  4. The authors explicitly state the limitations of their results, for example, the use of only a single training run per experiment and the limited generalizability of specific training effects.

Critical weaknesses
  1. The scientific added value compared to existing benchmarks is not clearly articulated. Works like HiddenBench and SILO-BENCH already investigate the collaboration of AI agents under distributed information, TeamGym distinguishes itself, through features like source-grounded PIUs, the separate analysis of communication and retrieval, and the capability for policy training. The specific new scientific insights gained from this approach could be highlighted more clearly.
Questions for authors
  1. In Section 4.4, a reversal in the ranking of different communication structures is reported in relation to varying PIU distributions. However, the number of agents also differs between the conditions "One common PIU" and "Extra participant without a PIU."

Could the observed ranking reversal be explained by the difference in team size and the associated communication possibilities and does this effect persist if the team size is held constant?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ovUG (1/2): W1

W1. Scientific contribution relative to HiddenBench and SILO-BENCH

Thank you for encouraging us to make the scientific added value clearer. We see two connected contributions: a procedure for constructing source-grounded PIUs and the corresponding distributed source observations, and the scientific findings enabled by those evidence controls. In the original submission, we devoted Appendix C.1, "Closest Benchmark Comparisons," to individual comparisons with HiddenBench and SILO-BENCH. We summarize that comparison below and explain more explicitly how our construction procedure enables the scientific findings.

PIU construction and source-document division enable controlled comparisons and systematic scientific analysis. This procedure (§3.1–3.2; Appendix F.2) supports comparisons with the same underlying decision requirements and evidence. Using these controls, we estimate separate logistic regressions for Multi-Agent PIU-Only and Multi-Agent Full-Context across 11 problem families and eight models, with problem-family and model fixed effects and standard errors clustered by problem instance. The PIU-distribution–communication-structure interactions are jointly significant in both conditions (p < .001), providing systematic evidence that relative protocol performance depends on the tested PIU distribution (§4.3). Appendices G.5–G.6 report probability-scale estimates and robustness checks with problem-instance fixed effects. These findings motivate the targeted mechanism studies below.

DimensionHiddenBenchSILO-BENCHTeamGym
Task basisAdapted/generated Hidden Profile decisionsAlgorithmic distributed-computation tasksSource-grounded decisions across 12 corpora and four domains
Evidence presentationExplicit shared/private factsGenerated local data shardsDirect PIUs versus source material containing the same facts
Controlled comparisonsFull/Hidden Profile; prompting, depth, group-size and protocol ablationsProtocol × agent scale × model; global input scales with team sizePIU distribution × communication structure with fixed problem and evidence requirements
DiagnosticsReveal-All and Secretary interventionsSingle-agent baseline, outcome metrics and annotated logsPIU tracing, frozen-transcript interventions and trajectory replay
Improving coordinationPrompted structured communicationEvaluation of pretrained agentsGRPO training of participant and coordinator policies

The comparison above draws on HiddenBench v4 and SILO-BENCH, which provide valuable settings for studying distributed collaboration. To make the scientific insights enabled by TeamGym clearer, we highlight three findings:

  • The same source-grounded decisions reveal two performance gaps. Holding the decision, reference answer, and PIUs fixed, we compare one model given all PIUs, a team given distributed PIUs, and a team given source material containing its assigned PIUs. Across eight models and 12 corpora, success drops by 22.9 percentage points when moving to distributed team decision-making, and by a further 19.9 points when participants must find and use evidence in source material. These comparisons characterize the demands of distributed decision-making and the additional source-processing burden, including distraction and context length.
  • PIU distribution influences which communication structures perform better. The regression analysis motivates the new mechanism studies in §4.3. Across 59 finance tasks, broadening or removing narrow coordinator queries reduced the PIU-distribution–communication-structure interaction while holding the selected speakers and their order fixed. Across 44 SARA tasks, we examined messages from participants assigned no PIU. Under All-but-one PIU concentration and Simultaneous responses, when only those participants received added source material, removing their messages raised final-decision success from 21.2% to 68.9%. These interventions show how query content and the messages reaching the final decision maker can affect decision quality in the tested settings.
  • Training with a final-decision reward can improve the communication trajectory. On 40 finance validation tasks where both base and trained trajectories reached a decision, replaying the trained trajectory improved scores by 6.95 percentage points, averaged across the two final-decision policies. Each policy was held fixed when comparing trajectories; changing only the final-decision policy had a statistically inconclusive effect. The trajectory includes the messages, speaker selections, stopping decisions, and the final decision maker's identity and assigned PIU. This provides evidence that training can improve the team's communication and coordination under the evaluated setup.
AuthorsOpenReview ↗

Response to Reviewer ovUG (2/2): Q1

Q1. Could team size explain the reversal, and does it persist at fixed size?

The original named conditions use different constructions:

  • One common PIU: one PIU is initially common to all participants, while the remaining PIUs stay local.
  • Extra participant without a PIU: each PIU has a different private owner, with one additional participant assigned no PIU.

Under our original construction rules, these conditions have different numbers of communicating participants. Matching their rosters requires adjusting the PIU-assignment rules. The reported legal reversal (+3.9 versus −9.8 percentage points for Current-speaker selection minus Simultaneous responses) compares configurations that jointly specify information availability and team size. To investigate specific communication mechanisms, we added more tightly controlled intervention experiments that change coordinator queries or the messages reaching the final decision maker while holding the other settings fixed within each comparison.

Finance: query content with speaker selection held fixed (§4.3.1). We use 59 instances (30 FinDVer and 29 FinLongDocQA), pooling Qwen3-32B and Qwen3-235B-A22B in Multi-Agent Full-Context. The design compares One PIU per participant and All-but-one PIU concentration under Round robin and Coordinator-selected speaker. Within an episode, participants and the coordinator or separate final decision maker use the same model.

The interventions operate within Coordinator-selected speaker. We replay the original selected participants, their order, and response count, holding PIU assignment, private source material, model, resource budgets, and final-decision procedure fixed. We retain the original task-specific query or replace it with "It is your turn to contribute." Independently, we append or omit public PIU-category labels plus an instruction to report supporting evidence for each category. The labels identify topics without supplying private facts or answers; the original participant descriptions remain unchanged. We regenerate messages and final decisions for three repetitions, with 600-token message caps and a 1,200-token final rationale budget, and retain the original Round robin outcomes.

The replay interaction—the Round robin advantage over Coordinator-selected speaker under concentration minus its advantage under One PIU per participant—is +15.5 points with the original query and no additions. It falls to +5.9 with the labels and instruction, and becomes −5.4 or −9.3 when the task-specific query is removed without or with those additions. All three changes from the original-query replay are statistically significant; the resulting interactions are not. Substantive PIU communication also improves selectively under concentration. These results show how query content affects evidence sharing and decision success with speaker selection held fixed. The added-category treatment tests the labels and reporting instruction together.

SARA: source access and messages from participants without assigned PIUs (§4.3.2). We use 44 primary SARA instances (one additional instance is reserved for development), Qwen3-32B, Qwen3-235B-A22B, and GPT-5.5, with three repetitions. Participants receive their assigned PIUs explicitly, with or without added private source material. This differs from ordinary Multi-Agent Full-Context, where participants must retrieve their PIUs. Across One PIU per participant and All-but-one PIU concentration and five communication structures, adding source material changes the Simultaneous-versus-Coordinator PIU-distribution interaction from +10.6 to −5.1 points, a statistically significant −15.7-point change.

We then independently vary source-material access for participants assigned PIUs and those assigned none, keeping exact PIUs and public descriptions fixed. To test the contribution of the latter group's messages, we use All-but-one PIU concentration and Simultaneous responses, freeze all participant messages separately within each source-access condition, and regenerate only final decisions. The final decision maker sees either the complete transcript or only messages from the two PIU holders; those holders' messages and assignments remain unchanged.

When source material goes only to participants assigned no PIU, removing their messages improves success from 21.2% to 68.9%. Without added source material, filtering changes success from 69.9% to 68.9%. The +48.7-point interaction between source access and message removal is statistically significant. Thus, in this setting those messages can impair the final decision even while the evidence communicated by the PIU holders is retained.

Thank you again for helping us clarify TeamGym's scientific contribution and deepen the mechanistic analysis. We hope the revised explanations and additional controlled experiments address your concerns, and we would welcome any further feedback.

Thank you for the detailed response and the substantial new content, including the additional controlled studies. I will take these into account in my final assessment. I note that the ranking reversal is acknowledged to be confounded with team size and was not tested at fixed size

Reviewer_ChTmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces TeamGym, a framework for evaluating and training multi-agent decision-making under distributed information. It constructs real-world tasks using pivotal information units, which form an inclusion-minimal set of evidence sufficient for a correct decision. By controlling information allocation, source presentation, and communication structures, the authors quantify communication and retrieval gaps across 8 LLMs and four domains. They further show that reinforcement learning can improve decision quality and induce coordination behaviors without explicitly rewarding them.

Critical strengths
  1. The introduction of PIUs is a valuable methodological contribution. Identifying a decision-sufficient and inclusion-minimal evidence set allows the authors to construct tasks where successful decisions require combining information held by different participants. The complete-set and leave-one-out checks provide a systematic, albeit operational, procedure for verifying evidence relevance and necessity.
  2. A major strength is the explicit separation of the decision problem, information allocation, source presentation, communication structure, and resource constraints. The communication-gap and retrieval-gap comparisons provide intuitive and useful diagnostic measurements, while the PIU-distribution–communication-structure interaction analysis goes beyond simple comparisons of multi-agent frameworks.
  3. The RL experiments demonstrate that optimizing final decision quality can change communication behaviors without explicitly rewarding participant coverage or speaker selection. In particular, the cross-replay analysis provides evidence that improvements in communication trajectories can contribute to better performance. This is a promising direction for training collaborative agents.
Critical weaknesses
  1. The comparison between Single-Agent All-PIU and Multi-Agent PIU-Only changes not only information availability but also the interaction process, the identity and role of the final decision maker, and the way evidence must be communicated under resource constraints. Consequently, the communication gap may reflect multiple interacting factors, including coordination, information transmission, synthesis, and decision-making difficulties.
  2. Similarly, Multi-Agent Full-Context introduces longer source materials, additional irrelevant context, and potentially different effective information burdens. The retrieval gap therefore captures more than information extraction alone.
  3. TeamGym defines multiple dimensions, including five PIU-allocation conditions, five communication structures, source-presentation settings, token budgets, and model policies. However, different experiments use different subsets of these dimensions, making the overall experimental design difficult to follow.
  4. Figure 1 may confuse readers by placing the label “inclusion-minimal” beside the four “no answer” leave-one-out subsets. On line 197 of the paper, there is a spelling error: “trace” is misspelled as “traace.”
Questions for authors
  1. To what extent do the RL gains come from improved information sharing rather than increased participation or final-decision completion?
  2. Does learned coordination generalize across tasks, models, and communication structures? The current training results use a single base model and one training run per demonstration, with relatively limited evidence of transfer. Could the authors provide multi-seed evaluations and experiments with additional model initializations?
6 comments
AuthorsOpenReview ↗

Response to Reviewer ChTm (1/3): W1–W4

Thank you for the thoughtful review and for recognizing our methodological values.

References below use the current manuscript numbering.

Weakness [W1, W2]. Scope of the communication and retrieval gaps.

As defined in the original §4.2 (now §4.1), the communication gap concerns communicating and integrating distributed PIUs, while the retrieval gap is the additional decrease in success from Multi-Agent PIU-Only to Multi-Agent Full-Context. The revision makes the combined demands captured by these comparisons more explicit (Appendix D).

For the communication gap, Single-Agent All-PIU gives one model the complete pivotal evidence directly; Multi-Agent PIU-Only distributes that same evidence across participants who must exchange and integrate it under the specified team protocol. The underlying problem, reference decision, PIU set, model, and evaluator are matched. The transition changes local evidence access, the interaction process, and final-decision roles, with communication occurring through bounded messages. Coordination, transmission, synthesis, and final decision making are therefore all relevant to the measured 22.9-point difference. We examine these processes separately:

  1. Coordination. Original analyses: the RL experiments track changes in coordinator queries, participant coverage, speaker selection, and self-handoffs (§§5.1–5.2; Tables 5 and 8). Added in the revision: three joint-training histories, three finance shared-policy runs, and an additional Llama initialization (Appendices I.2.3, I.3.6, and I.3.7).
  2. Information transmission. Original analyses: transcript PIU coverage and a common final-decision policy show how much pivotal evidence reaches the conversation and how useful those transcripts are (§5.2; Table 9). Added in the revision: the finance query interventions test how query content changes evidence sharing with speaker selection fixed (§4.3.1), and a paired CUAD-to-MAUD audit traces evidence from its owner through queries, replies, and final use (Table 53; Appendix I.2.4).
  3. Synthesis. Original analyses: the correct-synthesis and error-uptake interventions add supported inferences while holding communicated PIUs fixed, or remove later uses of an earlier error (§4.2; Table 2; Appendix G.4).
  4. Final decision making. Original analyses: we compare final-decision models in the mechanism study (§4.2) and replay base and trained communication trajectories with both final-decision policies (§5.2; Table 7). Added in the revision: in SARA, assigned PIUs remain explicitly supplied while source-material access varies by participant group. Within each source-access condition, we freeze participant messages and regenerate final decisions from full or filtered transcripts (§4.3.2).

The retrieval gap compares Multi-Agent PIU-Only with Multi-Agent Full-Context. These conditions hold the underlying problem, PIU set and distribution, communication structure, model, evaluator, and nominal resource budgets fixed, while changing the presentation from directly supplied PIUs to source material containing those facts. The 19.9-point drop captures the combined demands of finding and using evidence within source material, including longer inputs, irrelevant context, and potentially unequal information burdens across participants.

We also report Single-Agent Full-Context as a supplementary condition, in which one model receives the combined source material assigned to all participants (Tables 30 and 32; Appendix G.3, as in the original submission). Of the 2,632 scheduled problem-instance–model inputs across 11 corpora, 2,099 were evaluated normally, 50 had output failures counted as unsuccessful episodes, 460 exceeded a serving context window, and 23 had other unavailable failures. Context-window failures are disclosed separately from the 2,149-input success denominator, so readers can distinguish the reported success rates from the coverage of the evaluation.

W3. Which settings does each experiment use?

We added an overview table in Appendix B (Table 10). Each row states what varies and what stays fixed, including the evaluation condition, PIU distribution, communication structure, model/policy, and budgets. We also added an experiment roadmap in Appendix A (Figure A1) that maps each experiment to its main-text section and detailed appendix. Section 3.3 points to both.

W4. Figure 1 and the typo on line 197.

Figure 1 now presents the framework overview, and PIU construction has its own diagram in Figure 7 (Appendix F.2). The retained set is labeled "jointly sufficient; each PIU necessary," which the caption identifies as the inclusion-minimal pivotal information, and each leave-one-out row shows what becomes unresolved when that PIU is removed. We also corrected "traace" to "trace."

AuthorsOpenReview ↗

Response to Reviewer ChTm (2/3): Q1

Q1. Do RL gains come from evidence sharing, participation, or completion?

We appreciate it for asking us to distinguish these contributions. The original submission included the three analyses below: trajectory × final-decision-policy cross-replay, common-final-decision-policy scoring with transcript PIU coverage, and participation/completion diagnostics. Their current locations are §5.2 and Appendix A, Tables 7–9.

In the revision, we expand the explanation and presentation of these existing results. We clarify the cohorts and controls in each comparison, bring the supporting Tables 7–9 together in Appendix A, revise their captions, and add Figure 5 to illustrate evidence sharing and coordination through actual base/trained messages. The repeated-training and additional-model experiments are described separately in Q2 below.

First, the original cross-replay analysis (current Table 7) uses 40 validation instances on which both the base and trained policies produce a final decision. Each recorded trajectory is paired with both final-decision policies. Replacing the base trajectory with the trained trajectory improves replay score by 6.95 percentage points (paired 95% CI [0.49, 13.78]), whereas changing only the final-decision policy gives −0.20 points (CI [−0.99, 0.61]). Because both original episodes complete on these instances, this comparison provides evidence of a trajectory contribution beyond the difference in original completion rates within this subset. The trajectory includes messages, speaker selections, stopping, and the final decision maker's identity and assigned PIU; the result therefore concerns the communication trajectory as a whole.

Second, the original common-final-decision-policy analysis (current Table 9) applies one common GPT-5.5 final-decision policy to the valid transcripts from all 39 FinDVer validation and 40 FinLongDocQA test instances, including originally incomplete episodes. This policy receives the task and transcript, with the original final answer and participants' private PIUs withheld. On FinDVer, trained transcripts improve its score from 34.10% to 46.72%, a gain of 12.62 points (95% CI [4.38, 21.59]). A separate transcript audit also finds that the proportion of PIUs communicated rises from 38.0% to 53.9%. These results support an information-sharing contribution: the trained team's conversation supplies more pivotal evidence and enables better decisions under the same final-decision policy.

Third, the original participation and completion diagnostics (current Tables 8–9) report these behaviors separately. On held-out FinLongDocQA, participant coverage rises from 64.4% to 90.0%, final-decision completion rises from 65.0% to 92.5%, and end-to-end score rises from 33.50% to 41.13%. Transcript PIU coverage remains at 63.1%, and the fixed-decider score change is inconclusive (−4.78 points, 95% CI [−17.75, 8.20]). In this cohort, the end-to-end gain accompanies broader participation and more completed decisions, with flat measured PIU coverage.

Together, the replay and coverage analyses link the FinDVer gain to better communicated evidence and improved communication trajectories, while the held-out results show gains in participation and completion with flat PIU coverage. Participation can itself enable evidence sharing, so we report these linked changes alongside the controlled replay effects.

AuthorsOpenReview ↗

Response to Reviewer ChTm (3/3): Q2

Q2. Does learned coordination generalize across tasks, models, or protocols?

We distinguish the task and team-size transfer already evaluated in the paper from the new repeated-training and model-initialization results, and from transfer between communication structures.

Behavioral transfer across tasks and team sizes.

The original experiments evaluate CUAD-to-MAUD transfer and held-out four-participant FinLongDocQA teams. Current Table 53 in Appendix I.2.4 makes the distinction between behavioral and decision-quality transfer explicit. On held-out MAUD, the number of instances on which every participant is queried and replies increases from 20/26 to 26/26, and final-answer completion increases from 21/26 to 26/26. Decision score changes only from 5.12% to 5.58% (+0.46 percentage points, paired 95% agreement-cluster interval [−1.97, 3.32]). Thus, we observe transfer of participation and completion behavior more clearly than improvement in decision quality. Required-evidence communication changes little, from 12.31% to 12.65%.

Tables 8–9 in Appendix A show a related pattern on the held-out four-participant FinLongDocQA test: participant coverage increases from 64.4% to 90.0% and final-decision completion from 65.0% to 92.5%, while transcript PIU coverage remains at 63.1%. Finance training includes both FinDVer and two- or three-participant FinLongDocQA instances; the four-participant FinLongDocQA test is held out, although four-participant FinDVer teams occur in training. These results provide evidence of task/team-size generalization of coordination behavior within the evaluated settings. The behavioral diagnostics concern the original detailed runs; the new multi-run endpoint results below are reported separately.

Repeated Qwen training.

Figure 4 in §5 now shows three-run means with across-run SD shading for both joint coordinator/participant training and finance shared-policy training. Appendix I.2.3 describes the three recorded joint-training histories.

For finance, current Table 6 and Appendix I.3.6 report three Qwen2.5-7B-Instruct runs. Mean scores ± sample SD are 38.79 ± 3.70% on FinDVer validation, 67.70 ± 3.16% on FinLongDocQA validation, and 40.83 ± 8.82% on the held-out four-participant FinLongDocQA test. The corresponding endpoint baseline scores are 25.95%, 54.05%, and 25.53%. Individual test gains are +15.60, +23.98, and +6.35 percentage points.

Additional model initialization.

Appendix I.3.7 reports one Llama-3.1-8B-Instruct finance run from its own frozen starting checkpoint, using the same finance tasks and 80-update budget. Scores rise from 15.46% to 34.64% on FinDVer validation, 27.62% to 50.95% on FinLongDocQA validation, and 17.50% to 36.33% on the held-out test. The test gain is +18.83 points (paired 95% source-group bootstrap interval [9.83, 27.95]). Each model is compared with its own baseline. This independently trained Llama policy supplies initial evidence of trainability in another model family alongside the repeated Qwen experiments.

Communication structures and future work.

We demonstrate learned coordination under two separately trained structures: coordinator-controlled legal teams and finance teams using Current-speaker selection. Transfer of a fixed trained policy between communication structures remains untested. A natural future direction is to train over a mixture of structures, or allow a policy to choose or switch structures during training, to test whether more flexible coordination can be learned.

Thank you again for helping us clarify the diagnostic comparisons and examine learned coordination more closely. We hope the settings overview, additional training runs, and expanded analyses address your questions, and we would welcome any further feedback.

Reviewer_pRgnOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents TeamGym, a set of tasks across multiple scenarios (e.g. legal, medical, finance). TeamGym is used for two purposes - first to assess the effect of multiple important axes in multi agent systems, e.g. hidden information integration as a function of communication structure and ability to properly retrieve said hidden information when context containing it is distributed amongst agents. The authors perform an interesting logistic analysis to model which variables drive performance to make comments on multi-agent systems broadly, which is useful. The authors then include an RL experiment to show that teams can learn policies to better integrate distributed information, which is a useful illustration of how to train better multi agent teams.

Critical strengths
  • Comprehensive variation and characterization of choices that underlie multiagent systems. The axes they choose to vary are general enough to represent a large number of multiagent systems, and also reflect choices that are often assumed and not ablated in multiagent system papers. Moreover, they test across a variety of tasks so the reader can gain an an understanding of what trends are robust
  • Analysis: The use of fixed effects logit model is useful for analyzing multiagent systems as an object as opposed to analyzing one instantiation of a multiagent system.
  • RL: The authors show a useful illustration of how to train for team performance, which is still an open and somewhat understudied question in the multiagent teamwork literature.

Overall, the question, analysis, and results are very interesting.

Critical weaknesses

Unfortunately, and my reject rating rests on this: the paper is extremely difficult to read. There are many tables and figures that seem crammed into the paper, and it is not clear whether they are all necessary. Moreover, odd coloring (such as blue text) makes the paper look unprofessional. Paragraphs are long, with poor spacing between them, and many sentences are written unclearly, with results being hard to extract. The main text includes variation of several variables and axes but often does not clearly explain takeaways - one example is error uptake vs synthesis analysis.

At present, I believe the paper presentation is under the standard for acceptance to ICLR.

I include a list of concrete examples in the 'Questions for Authors' section.

Questions for authors

The question and results are very interesting, and studying team performance as a system with a logistic model is a useful approach. Most of my questions below are about clarity, plus a few missing controls.

Clarity and presentation

  1. The main text has no conclusion, and the RL section ends abruptly. Please add a short conclusion that states the main findings.
  2. Several figures and tables are hard to read. Tables 3 and 4 are crammed into the text, and page 9 places five figures and tables close together. There is already more than enough interesting content: could you dedicate the main figures to quantitative analysis of the main trends, plus qualitative examples of learned behaviors, and move the rest to the appendix?
  3. What is the takeaway of Figure 2? And why does Table 3 need to be in the main text?
  4. Figure 3 changes several variables at once: the communication structure (current-speaker selection vs. coordinator) and the training scheme (two-stage vs. joint). Could you restructure it around a single comparison so the takeaway is clear?
  5. Table captions (for example, Tables 5–8) describe what is shown but not the point. Please make captions standalone and state the finding each table supports.
  6. Blue text is used to highlight takeaways in Sections 4.2–4.4 and elsewhere. Consider stating takeaways in the text or captions instead.
  7. Section 5.2, among others, mixes implementation details with results, and some takeaways explain little. For example, "broader participation and higher end-to-end performance do not necessarily coincide with a larger proportion of PIUs appearing in the communication transcript" does not say why this happens or what is going on. Could you separate methods from results and explain what drives this finding? This comment applies to the rest of the paper as well.
  8. The RL section would benefit from clearer writing focused on the main takeaways, with one clear figure or results table and a representative qualitative example of the learned behaviors in the main text. Other RL results could move to the appendix.
  9. How are the embedded-retrieval (Full-Context) observations built, and how are the documents constructed? Section 3 does not explain this; please describe it there.

Experimental design and analysis

  1. Why is the token-budget condition (Section 4.5) necessary or interesting? Truncated messages seem like a systems issue rather than a behavior inherent to teams of language agents. Shouldn't the main experiments be run with budgets large enough that no cutoffs occur?
  2. Did you check that a single agent given all of the source material can solve each Full-Context instance? The solvability check gives the model the PIUs directly, and the Single-Agent Full-Context results in Table 30 exclude context-window failures and omit DeepSWE. Without this check, the retrieval gap does not separate retrieval loss from a long context issue, i.e., there being too much context for even one agent with full information to process correctly.
  3. Lack of human annotation / rating: as the PIU construction and env construction is all model-based, the paper needs human rater agreement, for example on PIU necessity and sufficiency for a random sample of instances.
  4. Why do the CUAD-trained gains not transfer to MAUD? An analysis would help show whether the CUAD environment misses difficulties specific to MAUD.
  5. Why include both joint and two-stage training? Is there an analysis contrasting the two? Otherwise the second setup mainly complicates the paper.
  6. What is the main takeaway of the correct-synthesis and error-uptake analysis about where teams fail? Please state it explicitly.
  7. Beyond the single selected episode in Appendix G.7, could you provide case studies that illustrate the interaction between PIU distribution and communication structure?
  8. Will the benchmark/env be released?
8 comments
AuthorsOpenReview ↗

Response to Reviewer pRgn (1/4): W1 and Q1–Q9 (presentation)

Thank you for the careful review. We are glad you found the work interesting, and we agree that the paper’s organization made the main findings hard to pick out. We went through every point you have mentioned and reorganized both the main text and the appendix. We redrew every figure in the main part and added three new figures to the appendix. We hope these revisions make the paper easier to follow and the main findings clearer.

Below, we respond to W1 and Q1–Q9 by topic. Section, figure, and table numbers refer to the revised PDF unless marked “original.” We’ve highlighted revised text in orange for this review and will remove the markup in the final version.

Structure and conclusion (W1, Q1, Q7, Q8)

  • The main text now covers the environment (§3), evaluation (§4), and policy training (§5) in separate sections. We added §6 to summarize the main findings (Q1).

  • We reorganized §5.2 to separate methods from results. The methods appear under Training setup and Analysis setup, followed by three results headings: Training results, Trajectory results, and Evidence coverage and completion (Q7).

  • We rewrote the sentence quoted in Q7 to clarify that repeating the same evidence leaves the number of distinct PIUs covered unchanged. On the held-out FinLongDocQA set, participation broadened and the share of public messages that were exact repeats rose from 23.1% to 48.6%. Mean transcript PIU coverage stayed at 63.1%. The final-decision completion rate rose from 65.0% to 92.5%. These diagnostics help interpret the end-to-end gain and leave open what drove it.

  • The RL section now focuses on the training curves (Figure 4), the scores (Table 6), and a before-and-after conversation example (Figure 5). We kept the comparison of coordinator query behavior in Table 5 and moved the replay, participation, and coverage diagnostics to Appendix A (Tables 7–9) (Q8).

  • For the synthesis and error-uptake analysis you mention in W1, the takeaway is now stated in bold beside Table 2. We say more under Q15.

Figures and tables (W1, Q2, Q3, Q4)

  • Each main-text page now has at most two figures and tables in total. The token-budget section (original §4.5) moved to Appendix G.8.

  • We replaced the original Figure 2 heatmap with Figure 3, which plots the three conditions on a shared success-rate axis within each domain. Its caption states the takeaway: across eight models and 12 corpora, success drops by 22.9 percentage points when PIUs are distributed across agents and by a further 19.9 points when agents must retrieve them from source material (Q3).

  • The instance counts in the original Table 3 are merged into Table 1, so readers can see the sample size underlying each corpus in the cross-domain comparison (Q3).

  • The original Table 2 (PIU distributions) and the appendix diagrams of communication structures became Figure 2.

  • We redrew the original Figure 3 as Figure 4. Panel (a) reports CUAD validation with a coordinator (§5.1); panel (b) reports FinDVer validation for the shared participant policy (§5.2). These are separate training demonstrations, not a controlled comparison of communication structures. The caption now specifies that each panel has its own pre-training baseline. Panel (a) retains two-stage and joint training, both evaluated on the same 36 CUAD validation instances (Q4).

We retain Tables 2–6 beside their discussion within the 10-page main text. Each has its finding in the caption or in an adjacent bold sentence.

Captions and color (Q5, Q6)

  • The captions of Tables 5–9 and Figures 3–5 now state the finding, for example "Joint training shortens queries and reaches every participant" (Table 5) and "Trained trajectories score higher" (Table 7). For Tables 2–4, the finding is in bold beside the table.

  • We replaced the blue italic takeaways with bold sentences in the text. Orange text is only for revisions temporarily.

How Full-Context observations are built (Q9)

Section 3.2 describes source-material distribution. We divide the source material into units without dividing a PIU, assign the units that contain PIUs according to the fixed PIU distribution, and distribute the remaining units across participants. We then verify that no participant can produce a correct final decision from its own initial observation and the common observation. We added a SARA example in §3.3 in which the same payment fact appears once as a directly supplied PIU and once inside a source excerpt. The new Figure 7 (Appendix F.2) illustrates the construction and distribution procedure, and Appendix F gives the source-specific rules.

Navigation

An experiment roadmap (Figure A1, Appendix A) maps every study to its section, and Table 10 (Appendix B) lists what varies and what stays fixed in each comparison. Section 3.3 points to both.

AuthorsOpenReview ↗

Response to Reviewer pRgn (2/4): Q10–Q11 (evaluation design)

Q10. Token budgets.

The main comparisons use large token budgets. The separate budget sweep deliberately restricts private notes, messages, and transcript reading to study how communication resources affect team decisions. The motivation comes from §2: A3 idealizes communication resources as nonbinding, whereas attention and communication constraints limit what a team can transmit and read. The sweep makes this departure explicit. We hold the problem instances, PIU distribution, communication structure, participant-response count, coordinator policy, and final-decision cap fixed. For GPT-5.5 on 24 CUAD and 29 Multi-LexSum instances, increasing the message cap from 32 to 128 tokens raises success from 55.3% to 78.6%, with no further average gain at 256. This tests sensitivity to communication bandwidth under controlled constraints. We moved the sweep to Appendix G.8 to keep the main text focused.

Q11. Single-Agent Full-Context.

Our solvability check establishes that the PIUs are sufficient: a model given all PIUs solves each evaluated instance (§3.1), and each participant's source material contains its assigned PIUs (§3.3). Single-Agent Full-Context was evaluated in the original submission as a supplementary condition, with its results and coverage reported in aggregate (Appendices G.2–G.3), rather than used as a per-instance filter.

The retrieval gap compares Multi-Agent PIU-Only with Multi-Agent Full-Context (§4.1), holding the problem, model, PIU distribution, communication structure, and resource budgets fixed. The comparison follows from the direct-report idealization in §2 (A2), which treats each participant's private information as something it knows and can report. For language agents, this reporting capability is not automatic. Decision-relevant information may be embedded in long source material, and a participant must first identify it, interpret it, and express it in a message useful to teammates whose material it cannot see. PIU-Only supplies the relevant facts directly, and Full-Context adds this step. The revised Appendix D explains this motivation.

Your question highlights a distinction in how "retrieval" can be understood. We use the term in a broader sense: our gap measures how team performance changes when participants must identify and use evidence within source material. Under this definition, long-input processing and irrelevant content are part of the measured burden together with evidence selection, and Appendix D now states this scope explicitly.

Single-Agent Full-Context gives one model the source material of all participants, so it shows how a single model handles the pooled material when no communication is needed. It differs from the team condition in two ways. First, the single model reads all the material at once, whereas each participant reads only its own share. Second, the model that finds the evidence also makes the decision, whereas a participant must report what it finds to the team. We therefore report this condition alongside the two gaps, and the retrieval gap itself is computed from the two multi-agent conditions. Its success rates in Table 30 are computed on the inputs that could be evaluated, and Table 32 reports the 460 of 2,632 inputs that exceeded a context window. To examine individual components of the team setting, the interventions in §4.3 vary what the coordinator asks participants to report (§4.3.1), and participants' access to source material together with the messages that reach the final decision maker (§4.3.2).

This supplementary condition covers 11 corpora. DeepSWE uses a different evaluation protocol based on repository repair (§F.6.4). Its outcome combines evidence use, patch construction, execution feedback, and verification, making it difficult to separate retrieval from these other stages using a single-agent score. Its gap is interpreted within the repair protocol shared by the three primary conditions.

AuthorsOpenReview ↗

Response to Reviewer pRgn (3/4): Q12–Q13 (validation and transfer)

Q12. Human PIU ratings.

Following your review, we developed an annotation protocol and tool and recruited two raters with relevant expertise for each of the four domains. We randomly sample instances within each source corpus in each domain. The raters assess PIU sufficiency and necessity so we can measure inter-rater agreement.

We’re still waiting for ratings from three raters. As soon as we’ve compiled all the ratings, we’ll add the inter-rater agreement results for each domain and an Ethics Statement to the manuscript. We expect to make this update by the end of the weekend.

Q13. CUAD-to-MAUD transfer.

We added a paired information-flow audit of the joint-training run on 36 CUAD validation instances and 26 held-out MAUD instances (Appendix I.2.4, Table 53). On MAUD, the number of instances in which every participant was queried and replied rose from 20/26 to 26/26 after training. The score changed only from 5.12% to 5.58%.

In the model-based audit, the mean partial-credit score for communicating required evidence rose from 24.77% to 37.37% on CUAD (+12.60 points, 95% interval [3.39, 22.21]) and from 12.31% to 12.65% on MAUD (+0.35 points, 95% interval [−2.19, 4.15]). The improvement in participation transferred to MAUD more clearly than the improvement in evidence transmission.

Both corpora contain conditions, exceptions, and party directionality. Under the same short-message settings, the MAUD instances in the audit had longer sources and reference evidence, more widely distributed PIUs, and more required answer components.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: summary of revisions

We thank all reviewers for the time and care they put into our paper. Your comments on presentation, on how to interpret our comparisons, and on the training evidence were very helpful, and we worked hard to address them in the revision. The main text is now 10 pages, using the additional page allowed during the discussion period. Revised and added text is shown in orange so the changes are easy to find, and we will remove this markup in the final version.

This comment covers changes relevant to more than one review. Our replies under each review respond to each point.

1. Presentation and navigation (pRgn, ChTm)

We reorganized the paper so that each section does one thing.

  • §1 Introduction. The introduction now previews the full paper, including the main findings. We redrew the overview in Figure 1 with essential information.
  • §3 Environment. This section is now solely about the environment and how we build problem instances. The instance counts that used to sit in §4.1 are in Table 1. The new Figure 2 puts the PIU distributions and the communication structures side by side, and §3.3 gives a SARA example in which the same fact appears once as a PIU and once inside a source excerpt. In the appendix, Figure 6 (Appendix D) shows the three evaluation conditions and the two gaps, and Figure 7 (Appendix F.2) shows how PIUs are built and distributed.
  • §4 Evaluation. We replaced the original heatmap (Figure 2) with Figure 3, which compares the three conditions directly across all models and domains.
  • §5 Training. The new Figure 5 shows a FinLongDocQA conversation before and after training, with each private fact mapped to the participant who holds it. The supporting diagnostics (Tables 7–9) are now in Appendix A.
  • §6 Conclusion. We added a short conclusion, and the colored takeaways are now bold text.

We also added an experiment roadmap (Figure A1, Appendix A) showing every study and where to find it. We believe it will help in seeing the paper as a whole. Table 10 in Appendix B lists what varies and what stays fixed in each comparison, and §3.3 points to both.

2. What the two gaps measure (ChTm, pRgn)

The communication gap (22.9 pp) and the retrieval gap (19.9 pp) each compare two specified conditions while the model, problem instance, PIU set, and evaluator stay fixed. The reviewers are right that each gap bundles several demands. The communication gap includes the move to a team protocol and a final-decision process, and the retrieval gap includes source length, irrelevant content, and extraction. The revision now states this explicitly in Appendix D, and the two studies in the next section examine individual components.

3. New controlled mechanism studies (§4.3; Appendix G.7)

Section 4.3 keeps the cross-domain analysis of the interaction between PIU distribution and communication structure and adds two controlled studies.

The first uses 59 finance instances (§4.3.1). Round robin's advantage over Coordinator-selected speaker is larger under All-but-one PIU concentration, and we wanted to know whether this comes from who speaks or from what the coordinator asks. Keeping speaker selection, order, source material, and budgets fixed, we broadened or removed the coordinator's query. PIU sharing went up, and the interaction dropped from +15.5 pp to +5.9 pp when the query was broadened and to −5.4 pp when it was removed. Part of the difference between protocols therefore comes from the content of the query.

The second uses 44 SARA instances (§4.3.2), where every participant receives its assigned PIUs directly. Giving source material to participants who hold no PIU still lowers success sharply. We froze all messages and removed only those participants' messages from the final decision maker's input. Success rose by 47.7 pp when they had received source material and did not change significantly when they had not.

4. Training evidence (§5; Appendix I)

Figure 4 now reports the mean ± SD over three runs for both joint coordinator/participant training and finance shared-policy training (Appendices I.2.3 and I.3.6). The detailed behavioral diagnostics still use the original representative runs.

We also trained one Llama-3.1-8B-Instruct policy on the finance tasks. It improves on its own baseline on the held-out test by 18.83 pp (Appendix I.3.7).

For CUAD-to-MAUD transfer, we traced information flow in paired episodes and found that participation behavior transfers more clearly than decision quality does (Appendix I.2.4).

5. Anonymized website and playground

We set up a project website: https://teamgym-anon.github.io. In line with ICLR policy, it is fully anonymized and does not track visitors. It includes a playground where you can run TeamGym episodes yourself with your own API key.

We hope the revised paper is easier to follow and are happy to answer any further questions.

AuthorsOpenReview ↗

Update: transfer of the trained policy across communication structures

Following Reviewer ChTm's Q2, we tested whether the learned coordination transfers to communication structures not seen during training. We share the result here since it bears on the generality of the training results discussed in several reviews.

The finance policy is trained under Current-speaker selection, where it controls three decisions: writing messages, selecting the next speaker, and deciding when to stop. We evaluated the representative run (Run A: seed 17, update 70) without further training under Round robin and Simultaneous responses. Both structures keep the message-writing role but remove next-speaker selection and stopping, so if the training gains came only from these two decisions, they should disappear. We use the 40 held-out four-participant FinLongDocQA instances and the GPT-5.5 judge; all rows come from the same evaluation setup.

StructureBaseTrainedGain [95% CI]
Current-speaker selection (trained)28.538.7+10.2 [−1.9, 22.4]
Round robin (not trained)20.942.0+21.2 [11.3, 31.6]
Simultaneous responses (not trained)8.830.6+21.9 [11.1, 32.5]

Under both untrained structures, the trained policy improves over the base model by about 21 points, with intervals well above zero (paired source-group bootstrap). The learned coordination is therefore not tied to the decisions specific to the training structure. The first row re-runs the training structure in this setup as a consistency check and is close to the Run A scores in Appendix I.3 (25.53 and 41.13).

We thank Reviewer ChTm for the suggestion, which helped strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.