acceptodds
Under review as a conference paper at ICLR 2027

510K: When-to-Cooperate under Latent Team Relationships and the Other-Regarding Value of Information

Abstract

Agents acting with others must first decide whether and with whom to cooperate, often by inferring relationships from observed behaviour. We study this when-to-cooperate problem in a setting where the relationship is learnable in principle yet unused in practice. We introduce 510K, a four-player card game in which team membership is hidden at deal time and the reward itself depends on the unknown relationship. We separate four questions that benchmarks conflate — availability, decodability, utilization, and value — and report a consistent pattern. (1) The relationship is decodable but unused. 67% of decisions are logically determined by public play and a history probe reaches AUC 0.72/1.00, yet 1M-step PPO never conditions on the team: actions change in only 0.5–3.6% of decisions when the team bits are zeroed, and adding more evidence, explicit revelation, stronger win incentives, or an undiscounted objective changes nothing. (2) Information value is other-regarding. Informing the agent is inert; informing partners moves outcomes — team-aware partners raise the agent's first-place rate by +0.066 (p=0.007, replicated on a fresh deal set), adding team-awareness to both teams lowers the agent's reward relative to the ignorant condition, and training against aware bots yields behavioural adaptation without reciprocity. Small language models reproduce the same gap, and a full-information search (first-place 0.43–0.48 vs. the best policy's 0.21–0.36) shows large headroom. We release the environment, the evaluation protocol, and belief-labelled trajectories as an instrument for when-to-cooperate decisions.

Primary area:
reinforcement learning
Keywords:
benchmark, multi-agent reinforcement learning, card games, partial observability, hidden teams, cooperation, testbed

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DnsrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces 510K, a four-player card game in which team membership depends on the initial deal and can be hidden from players. The authors separate four questions: whether relationship information is available, whether it can be decoded, whether a policy uses it, and whether it improves outcomes. They study these questions using belief probes, interventions on team-information inputs, and scripted players with different levels of team-awareness. Under the tested PPO configuration, removing explicit team information rarely changes the selected action. Making teammates team-aware improves the learner’s individual first-place rate. The paper also includes ablations of observations, reward settlement, discounting, and training populations, along with exploratory LLM experiments and a privileged-information search reference.

Critical strengths

Separating decodability from policy utilization is useful. The probes and behavioral interventions address different questions that outcome metrics alone cannot answer. This gives the benchmark a clear purpose: studying whether agents act on information about their relationships.

Holding the learner fixed while changing other players’ team-awareness helps identify where performance improvements come from. The same-deal comparisons reduce variation from card luck, and checking the effect on a second deal set strengthens the evidence for the teammate-awareness result.

Critical weaknesses

I am not yet convinced that the behavioral measures establish the claimed utilization bottleneck. Low action-flip rates are difficult to interpret without knowing how often the tested decisions offer meaningful team-dependent alternatives. Many states may call for the same action regardless of team identity. The raw deferral measure also depends on hand strength, game phase, and leader identity, as the authors acknowledge. Finally, the history probe has access to temporal evidence that is not directly comparable to the canonical feedforward policy’s input. The results show limited sensitivity on the measured channels, but the broader conclusion that recoverable relationship information goes unused needs stronger support.

The outcome results support a narrower interpretation than some of the paper’s claims suggest. In Table 5, teammate awareness increases individual first-place rate from 0.212 to 0.278, while finish-based team-win changes from 0.442 to 0.430. The table does not establish whether the reward increase from 78.2 to 80.8 is statistically reliable. The clearest benefit is therefore in individual placement. I also need clarification on Table 7, where first-place rates are 0.206 for DYNAMIC and 0.362 for OBVIOUS. These figures appear difficult to reconcile with statements that explicit revelation does not change outcomes.

Questions for authors
  1. What concrete class of coordination problems is 510K intended to model, and which properties of the environment are essential to that connection? What does the reported utilization failure teach us about those problems beyond the limitations of the tested learner?

  2. How do the DYNAMIC and OBVIOUS first-place rates in Table 7 relate to the claim that explicit revelation leaves outcomes unchanged? Which comparison isolates revelation while holding the other conditions fixed?

  3. What fraction of decisions in the counterfactual test have multiple legal actions and a meaningful opportunity for team-dependent behavior? Does the low action-flip rate persist within that subset?

  4. Does teammate awareness produce a statistically reliable improvement in terminal reward or finish-based team-win, beyond the reported improvement in individual first place? What exact changes to the scripted bots’ behavior implement team-awareness?

1 comment

We thank the reviewer for the detailed and constructive review. The distinction between "sensitivity on the measured channels" and "the broader conclusion" is well taken; we agree with the narrower interpretation and have run the analyses requested.

Q3: meaningful opportunities. We repeated the team-indicator intervention and recorded the decision context. On the default OBVIOUS policy (5 seeds × 100 deals; 8,253 decisions with team indicators, pooled):

54% (4,490) have >1 legal action; 46% are forced passes. Removing the indicators changes 0.0% of forced decisions and 0.04% of choice decisions (2/4,490 pooled; per-seed mean 0.04%, range 0.0–0.2%); partner-leads 0.04%; opponent-leads 0.0%; pass↔play 0.0%. For RevealEnv p=1, choice-state change is 5.8% pooled (per-seed mean 5.2%, range 0.3–12.9%), partner-leads 3.9%. Even the full masked action distribution barely changes: mean KL(true||zero) is 9.4×10⁻⁵ in choice states (OBVIOUS) and 0.043 for RevealEnv p=1. A second deal set (seeds 2000–2099) reproduces the pattern: flip 0.0–0.1%, outcome reward difference 0.01 (p=0.37). Consistently, the raw deferral asymmetry is sample-sensitive (100 vs 300 episodes can change its sign), so we use it only as a behavioral channel, not as proof. Thus the low rate is not a denominator artifact; it persists where the team relation could change the action. Q2: Table 4 / revelation. We agree the DYNAMIC–OBVIOUS comparison is not a reveal manipulation: they are separately trained checkpoints, and OBVIOUS also makes the rule bots team-aware, so the other players change too. The revised manuscript relabels it as a cross-checkpoint capability comparison and removes it from the causal claim. The comparisons that hold other conditions fixed are: (i) reveal dial during DYNAMIC training: p=0 vs 1 first 0.212 vs 0.200, slope p=0.83; team-win 0.442 vs 0.460, slope p=0.47; (ii) within-policy counterfactual test on OBVIOUS: same policy, same 100 deals, indicators true vs removed vs shuffled: first 0.362/0.362/0.362, team-win 0.482/0.482/0.482, reward 69.88/69.88/69.88; the intervention changed the chosen action in two of 8,253 decisions, and terminal outcomes were identical. For the RevealEnv p=1 checkpoint, test-time zeroing gives first 0.200 vs 0.186 (p=0.21), team-win 0.460 vs 0.432 (p=0.23), reward 82.20 vs 80.36 (p=0.19): no reliable change.

Q4: teammate awareness and statistics. You are right that the original 100-deals/seed evaluation was underpowered for reward. We re-ran the same fixed-policy decomposition with 500 deals/seed (same 5 seeds). Teammate-aware vs ignorant on the primary deals: first +0.061 (p<0.0001; all five seeds positive), reward +2.39 (95% CI 1.35–3.44; p=0.0031), team-win −0.010 (p=0.071). On the fresh deal set: first +0.045 (p=0.0009), reward +6.78 (CI 3.38–10.17; p=0.0052), team-win +0.012 (p=0.234). So terminal reward is statistically reliable; finish-based team-win is not. The clean opponent-only contrast (all-aware vs teammate-aware) lowers reward by 13.47 and 13.08 (p<0.0001).

Bot implementation: teammate-aware bots use one extra rule: when their partner is the current trick leader and passing is legal, they pass; otherwise they play the rule bot's lowest legal combination. Opponents are unchanged. All-aware applies the same rule to every seat using its own team. This is a minimal one-rule intervention, which is why it isolates partner responsiveness; it is not a rich cooperative policy. When the learner is trained under teammate-aware bots, the same contrast gives first +0.064 (p=0.0112), reward +5.29 (p=0.0106), team-win +0.012 (p=0.3585).

Q1: class of problems / what it teaches. 510K instantiates latent-partner ad-hoc teamwork under coupled individual and team objectives: team membership is random at deal time (2v2 76.5%, 1v3 23.5%), hidden, decodable from the agent's observation and public history, and payoff-relevant through the team settlement. It isolates "who is with me" from "how to coordinate" (known teams in Hanabi/Overcooked) and from adversarial hidden-role games (Avalon/DeepRole). The essential properties are: stochastic hidden membership; public evidence sufficient for inference; reward indexed by the latent team; counterfactual interventions and ground-truth labels. The utilization failure teaches that availability/decodability are not sufficient; outcome metrics can mask non-use; standard PPO can achieve high reward via generic card play without conditioning on a payoff-relevant latent. This motivates auxiliary belief objectives, memory, and population training. We scope this to the tested learner; the revision adds a scope paragraph.

History probe. We agree the history probe is not the feedforward policy's input; it is an upper-bound diagnostic. The revised manuscript relabels it as "decodability by an idealized history probe" and not as "what the policy sees".

Reviewer_QP6XOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses a notable gap in multi-agent reinforcement learning (MARL): the lack of compact, tractable benchmark environments where cooperative team structures are latent, controllable, and quantitatively measurable at the decision level. To study when and how agents infer and act upon hidden relationships, the authors introduce 510K, a four-player trick-taking/shedding card game characterized by latent team dynamics.

The primary contributions claimed by the authors are: 1) Benchmark Environment: A controlled multi-agent environment (510K) designed to evaluate cooperative decision-making under latent relationships; 2) Empirical Findings: Two central observations regarding current MARL algorithms: (a) existing methods neither effectively condition on latent relationship signals nor benefit from implicit representations, and (b) the value of relationship information is inherently "other-regarding" (dependent on teammates' actions); 3) Open Challenge: Demonstrating that learning robust cooperation under latent team partitions remains an open, unsolved challenge for standard MARL baselines.

Critical strengths
  1. Clear and Impactful Research Motivation: Formulating cooperative game dynamics where social and team affiliations are latent variables to be inferred—rather than given as ground-truth inputs—is a compelling and fundamentally important direction for multi-agent reinforcement learning and social intelligence.

  2. Extensive Empirical Evaluation: The paper conducts a broad range of experiments across multiple baseline formulations to evaluate whether agents utilize latent team structure, providing an empirical foundation for studying this benchmark.

Critical weaknesses
  1. Severe Presentation Deficiencies and Incoherent Terminology: The manuscript suffers from significant readability and drafting issues consistent with unedited or poorly integrated LLM-generated text. Key concepts, domain terms, and metrics (e.g., "aware bots", uncontextualized AUC values, "closed-form hypergeometry", "1-ply search", "obv-v2", "canonical policy in DYNAMIC") are introduced abruptly without necessary context, definitions, or formal grounding for a general ML audience. Furthermore, the structural narrative is disjointed, featuring fragmented single-sentence paragraphs and unresolved transitions.

  2. Incomplete Literature Survey and Erroneous Citations: Crucial prior works addressing hidden roles and imperfect-information multi-agent games (such as DeepRole and related hidden-role benchmarks) are either inadequately discussed or omitted. Additionally, there are citation anomalies (e.g., Section 2.5, Line 150 citing "Prior work (Anonymous, 2027)"), reflecting insufficient proofreading and academic diligence prior to submission.

Questions for authors

***Clarifications Materially Impacting the Assessment:

  1. Evaluation Metrics and Grounding: -- In the abstract and introduction, several metrics and terms appear without contextual definitions (e.g., "AUC 0.72 / 1.00", "aware bots"). Could the authors formally define how AUC is computed in this context, what baseline it benchmarks against, and how AUC performance correlates with cumulative game reward? -- What does the phrase "23.5%, closed-form hypergeometry" formally signify regarding the deal distribution, and how is it derived?

  2. Structural Generalizability: The 510K environment is restricted to a four-player setting (resulting in a fixed 2v2 split). How does the proposed formulation and latent inference mechanism scale or generalize to settings with N>4 players, asymmetric team sizes, or dynamic/shifting alliances?

  3. Clarification of Textual Ambiguities: -- Lines 60–61: Could you explain what is meant by "the layers" and why "the answer pattern is asymmetric"? -- Line 71: What is the precise operational definition of "when the agents' teammates act on the hidden team"? -- Line 256: What is the "canonical policy in DYNAMIC", and how does it formally connect to the subsequent baseline analysis? -- Line 298: Please clarify the term "zero-sum rule" in the context of this game's scoring function.

*** Suggestions for Improvement (Future Work & Revisions):

  1. Citation Corrections & Related Benchmarks: Please fix the erroneous citation placeholder in Section 2.5 (Line 150) and contextualize 510K directly against established hidden-role benchmarks (e.g., DeepRole / Avalon, Secret Hitler, Hanabi).
  2. Readability and Flow: Merge fragmented single-sentence paragraphs into cohesive thematic discussions, and ensure all abbreviations (e.g., SMAC) are introduced upon first mention.
1 comment

We thank the reviewer for the careful reading and for recognizing the paper's motivation and empirical breadth. We are encouraged that the reviewer found the research motivation clear and impactful and the empirical evaluation extensive; the revision preserves these strengths while fixing the presentation. We also appreciate the specific, actionable list of presentation and citation issues; it has directly shaped our revision.

What we heard. The main concern is presentation rather than the empirical contribution: terminology and abbreviations are introduced without context, the prose is fragmented, and the citations contain an anonymous placeholder. We agree the submitted draft did not meet the standard we intend. We have rewritten and manually edited the manuscript. The revised version includes:

a terminology table (510K; SINGLE/STATIC/DYNAMIC/OBVIOUS; obs-v1/v2; first-place; team-win; reward; asym; counterfactual sensitivity; AUC; probe; determined; KL); all abbreviations expanded at first use (SMAC, PPO, AUC, KL, AEC); a rewritten abstract and introduction, including a third-person 510K example that makes the team-partition latent variable explicit, plus a four-questions table; a Q1–Q4 roadmap; "layers" replaced by Q1–Q4; "asymmetric" explained as availability/decodability high, utilization low, value other-regarding; clarification that "when the agent's teammates act on the hidden team" means the scripted bots on the agent's team use the simulator's ground-truth assignment to defer, while the agent's observation is unchanged; "canonical policy" replaced by "default policy", defined as MaskablePPO MLP 256×256, 1M steps, 5 seeds, obs-v2, rule bots; "zero-sum rule" renamed "zero-sum settlement" (winners +score difference, losers −score difference), explicitly a reward ablation, not a game rule. Q1: AUC and 23.5%. AUC is the area under the ROC curve of a linear probe trained on non-team-bit features to predict each opponent's teammate indicator, macro-averaged over the three labels. Chance is 0.5; the probe is trained on 75% of decisions and tested on 25%; a majority-class predictor gets ~0.63 accuracy. The probe is a decodability diagnostic, not a policy or reward predictor; a probe can be highly accurate while the policy ignores it, which is the central finding. The "0.72/1.00" are DYNAMIC/OBVIOUS history probes; the single-observation probes are 0.63/0.90.

The 23.5% is an exact hypergeometric calculation, not a simulation: with 52 cards and two red aces, P(both red aces in the same 13-card hand) = 4·C(50,11)/C(52,13) ≈ 0.235, so 2v2 = 76.5%, 1v3/3v1 = 23.5%. "Hypergeometry" was a typo for "hypergeometric"; it is fixed.

Q2: generality. The current benchmark is four-player 510K; within it, team sizes are already asymmetric (1v3/3v1 in 23.5% of deals), so the split is not fixed 2v2. The formulation (latent deal-time partition + probe + interventions) extends to N players by defining a random partition at deal time and indexing the reward by it; the main costs are the combinatorial action space and credit assignment. We have not validated N>4 or mid-game shifting alliances; the revision states this as a limitation and adds a short generalization paragraph, rather than claiming it.

Q3: citations and related work. We will add DeepRole (Serrino et al., NeurIPS 2019), Avalon/The Resistance, Secret Hitler, Hanabi, and LLM hidden-role work; remove the "Anonymous, 2027" placeholder and rewrite Section 2.5; check every citation. We will also expand SMAC at first mention. We apologize for the citation anomaly; it should not have been in the submission.

Substance. So that the revision is not only cosmetic, we also ran new analyses that strengthen the empirical case: with 500 deals/seed, teammate-aware partners raise first-place by +0.061 and terminal reward by +2.39 (p=0.003; replicated on a fresh deal set); and a conditional bit intervention shows that the policy changes the chosen action in two of 4,490 choice decisions, with a mean KL of 9.4×10⁻⁵. These make the utilization claim more precise and better supported.

We hope the revised manuscript addresses the reviewer's concerns and merits reconsideration. We would be grateful for any remaining specific issues.

Reviewer_gaTYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors ask whether agents can figure out who their teammates are, and use that knowledge to cooperate. It introduces a four-player card game with hidden teams. The tested agents barely change their decisions when team identities are revealed, while making their scripted teammates team-aware improves their placement.

Critical strengths
  1. The controls are clear and make sense. Revealing teams, changing partners' behavior, and compare outcomes on the same card deals help understand the treatment effect.
  2. The conceptual difference between "knowing your teammates" vs "cooperating with them" is interesting. The former seems implicit in the latter, but this may not be the case.
  3. Seeded experiments and variance declaration is useful.
Critical weaknesses
  1. The learned policy doesn't have fully game history and receives a fixed training budget. If it fails, this doesn't establish that a better-trained agent would benefit from team information.
Questions for authors

Does a policy with memory and longer training learn to use team identity, and does revealing the team help then?

1 comment

We thank the reviewer for the constructive assessment and for isolating the key scope question.

Scope. We agree that a negative result at one architecture and budget does not establish that no better-trained agent can use team identity. Our claim is deliberately scoped to the default learner (MaskablePPO, MLP 256×256, 1M steps, 5 seeds); we will state this explicitly in the abstract/conclusion and add a "what this does and does not establish" paragraph.

Memory and longer training. We ran three additional checks. (i) Longer training: we continued the default DYNAMIC and OBVIOUS policies from 1M to 2M (5 seeds). On OBVIOUS, removing the team indicators changes 0.0–0.3% of chosen actions, and first/team-win are 0.356/0.484 (1M: 0.362/0.482). On DYNAMIC, the 1000-episode asymmetry is +0.008 (range +0.003 to +0.014; 1M: +0.009), first-place is 0.206 vs 0.206, team-win 0.412 vs 0.450, and reward +3.19 (p=0.116): no reliable improvement. (ii) History access: we trained a feedforward MLP with the same 21 public-history features used by the history probe (revealed red aces, follow/defer/overtake rates; 1M, 3 seeds). It still ignores the explicit bits: OBVIOUS bit-flip is 0.1% (range 0.0–0.2%); DYNAMIC asymmetry is −0.006. It numerically improves reward (90.9 vs 79.3, +12.7, p=0.062) through scoring play, not team conditioning; first-place is 0.183. (iii) We also tried a RecurrentPPO (LSTM) baseline, but it collapsed to a near-deterministic policy; we do not use it as evidence. The history-augmented MLP is our memory check. Thus neither longer training nor history access produces meaningful use of the explicit team indicators; the raw asymmetry is sample-sensitive and does not provide reliable evidence of team use. The revised manuscript reports these results and scopes the claim to "no meaningful use of the explicit team indicators; longer training can change raw pass patterns but does not produce reliable team-conditioned play." Learning curves also plateau by ~200–300k steps; 1M is 3–5× past the plateau, so the null is unlikely to be a simple budget artifact.

Does revealing the team help then? We separate two questions. (a) Does the policy use the indicators at test time? On the default OBVIOUS policy, holding the policy and the 100 deals fixed, removing the team indicators changes the chosen action in two of 8,253 decisions; in the outcome ablation, first-place, team-win, and reward are identical (0.362/0.482/69.88 in all conditions). Even in the 54% of decisions with >1 legal action, the change rate is 0.04% (0.0–0.2% per seed). (b) Does training with indicators help? The reveal dial p=0→1 is flat (first 0.212→0.200, slope p=0.83). So revealing the team does not help this learner; the failure is utilization, not availability.

Why this is still informative. The history probe reaches AUC 0.72 (DYNAMIC) / 1.00 (OBVIOUS) without team indicators, so the relation is recoverable from public history; the policy simply does not use it. The revised manuscript reports these results and the scope caveat. A fully powered recurrent study is beyond the rebuttal period; we would welcome guidance on the most informative memory architecture to add.

Comments on the paper

1 comment

We thank all three reviewers for their careful reading. The reviews converge on three issues: (i) the scope of the utilization claim, (ii) the causal interpretation of "explicit revelation", and (iii) presentation/clarity. We have run new analyses and revised the manuscript accordingly.

New analyses.

Conditional bit intervention. On the default OBVIOUS policy (5 seeds × 100 deals), 54% of decisions with team indicators have more than one legal action. Removing the indicators changes the chosen action in two of 8,253 decisions (two of 4,490 choice decisions), and outcomes are identical (first 0.362, team-win 0.482, reward 69.88). The low change rate therefore persists where a choice exists and is not an artifact of forced passes.

Within-policy counterfactual test. Holding the policy and deals fixed, true/removed/shuffled indicators give identical outcomes for the OBVIOUS policy. For RevealEnv p=1, test-time removal gives first −0.014 (p=0.21), team-win −0.028 (p=0.23), reward −1.84 (p=0.19): no reliable change.

Teammate-awareness statistics. With 500 deals/seed, teammate-aware vs ignorant gives first +0.061 (p<0.0001), reward +2.39 (95% CI 1.35–3.44, p=0.0031), team-win −0.010 (p=0.071); the fresh deal set replicates first +0.045 (p=0.0009) and reward +6.78 (p=0.0052), team-win +0.012 (p=0.234). So both first-place and terminal reward are reliable; team-win is not. The clean opponent-only contrast (all-aware vs teammate-aware) lowers reward by 13.47 (p<0.0001) on the primary deals and 13.08 (p<0.0001) on the fresh deals, which supports the other-regarding claim.

Scope. The revised manuscript replaces "never conditions" with "no meaningful use of the explicit team indicators in the default learner", and adds a training-budget/memory subsection. A 2M-step continuation (5 seeds) leaves the OBVIOUS change rate at 0.0–0.3% and the DYNAMIC 1000-episode asymmetry at +0.008 (1M: +0.009); reward changes by +3.19 (p=0.116), with no first-place/team-win improvement. A history-augmented MLP (3 seeds) with the probe's 21 public-history features still has a 0.1% change rate and −0.006 asymmetry, though it numerically improves reward (90.9 vs 79.3, p=0.062) through scoring play. We report both and do not interpret the raw measure as reliable team use.

Revisions. The revised manuscript (1) removes the OBVIOUS–DYNAMIC comparison from the causal reveal claim—OBVIOUS also makes the rule bots team-aware—and uses the reveal dial plus within-policy counterfactual test instead; (2) updates Table 3 (who-knows decomposition) with 500-deal p-values/CIs and the reward/team-win distinction; (3) adds a terminology table, expands all abbreviations, and rewrites the abstract/introduction in third person, including an explicit team-partition example; fixes the anonymous citation placeholder; (4) adds DeepRole/Avalon/Secret Hitler/Hanabi context; (5) clarifies AUC, the 23.5% hypergeometric calculation, "default policy", and the zero-sum settlement; (6) states the 4-player scope and the N>4/shifting-alliance limitation; (7) removes the collapsed LSTM row from the training-budget table and reports it only as an exploratory baseline that is not used as evidence; and (8) pools the Table 8/10 counts over the 5 seeds and aligns all reported counts (2/4,490 choice decisions, 0.04%).

Per-reviewer responses are posted separately. We are grateful for the detailed feedback, which has already improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.