acceptodds
Under review as a conference paper at ICLR 2027

Entity-Wise Factorization Enables Generalist Agents in Hanabi

Abstract

Self-play lets multi-agent reinforcement learning agents cooperate well with copies of themselves, but the arbitrary conventions they invent along the way rarely transfer to a new partner, human or otherwise. This is exemplified in Hanabi, a cooperative card game for 2 to 5 players widely used to probe theory of mind. Until now, -player game has been considered separately, with multiple works focusing only on 2- or 3-player games. In this work, we propose a generalist Hanabi agent that can play and be trained on any -player game jointly. To enable this, we require a common state representation agnostic to the number of players. Inspired by UPDeT (Hu et al., 2021b), a successful example of factorizing an observation into per-entity tokens so that one set of parameters can handle a variable number of entities, we propose FHA (factorized Hanabi agent), a simplified and fast factorization architecture. We then perform a comprehensive analysis of self-play, other-play and off-belief learning trained with our proposed architecture, jointly and independently across players. In doing so, we observe that factorization and joint training can improve cross-play performance of self-play agents, because the agent learns a general policy that it applies regardless of the number of players. Thus, by avoiding the overspecialization issues encountered by self-play agents, self-play can be a viable training method for cross-play agents. Finally, the learned policy is transferable, as FHA is able to train on a subset of settings and zero-shot transfer at deployment to unseen settings.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning (MARL), Cooperative game, Zero-shot coordination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1BWdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes the Factorized Hanabi Agent (FHA), whose observation and action spaces are factorized by teammate, making the model independent of the number of players. A single FHA model can therefore be jointly trained across 2–5-player games and transferred zero-shot to unseen player counts.

Critical strengths
  1. Joint training across player counts reduces overfitting to setting-specific conventions and improves coordination with independently trained partners.
  2. The paper evaluates self-play, cross-play, and zero-shot transfer, and compares FHA with UPDeT, Other-Play, and Off-Belief Learning.
Critical weaknesses
  1. Entity-wise factorization and permutation-aware architectures have already been extensively studied. The proposed architecture does not appear to introduce a fundamentally new mechanism and is evaluated only on Hanabi, limiting the broader significance of the contribution. Moreover, combining FHA with Off-Belief Learning appears somewhat modular or incremental, without a sufficiently clear motivation or a strong claim regarding their compatibility.
  2. The core methodological design closely resembles techniques established in previous literature, particularly entity-wise decomposition, parameter sharing, and permutation-equivariant processing. The paper does not clearly distinguish its method from prior work or demonstrate a substantial conceptual advance beyond adapting these ideas to Hanabi.
  3. The results in Figures 2 and 3 do not show consistent or meaningful improvements over the baselines. In several settings, FHA appears comparable to—or weaker than—existing methods.
Questions for authors

Please refer to weakness.

1 comment

Thank you for your feedback, comments and review. We have addressed common questions such as the failure modes of OBL and the inter-algorithmic cross-play in a general comment, and focus on specific comments here.

The proposed architecture does not appear to introduce a fundamentally new mechanism [...]. The core methodological design closely resembles techniques established in previous literature

We agree that we do not introduce fundamentally new mechanisms, and our methodological approach takes inspiration from many of the existing works in the literature. However, we want to emphasize that proposing a new factorized architecture is not the main focus of our work. The main message we want to convey is that, under certain circumstances, one can use simple self-play training and still perform ad-hoc cooperation with unseen partners. This observation is of interest to the community because 1) ad-hoc cooperation is a fundamental desired property of multi-agent learning, 2) self-play is one of the simplest multi-agent learning algorithms there is. Simply by training on multiple settings jointly, one can use a very simple algorithm to achieve a crucial property for cooperative learning.

In contrast, many of the works on entity-wise decomposition (Wang et al., 2020; Iqbal et al., 2021), parameter sharing (De Witt et al., 2020), and permutation-equivariant processing (Hao et al., 2023; Park et al., 2025) focus on learning efficiency and team performance. UPDeT (Hu et al., 2021b) performs transfer experiments on variable player count, but keeps self-play as an evaluation procedure.

Thus, in our work, factorization is but a means to achieve multi-setting training, unlocking ad-hoc cooperation in the process.

Results in Figures 2 and 3 do not show consistent or meaningful improvements over the baselines

IQL-LSTM does not show any cross-play performance, which is to be expected. Moreover, while FHA trained in a single setting shows some cross-play performance thanks to a restricted factorized observation-space, it is not on par with FHA trained in multi-player settings. This shows the effect training on multiple player-settings has on self-play agents. We agree that UPDeT also displays good cross-play performance, but this does not contradict our results: UPDeT was trained in self-play, on 2-5P games simultaneously as well. One main disadvantage of UPDeT lies in its training speed. Compared to FHA, it is 8-to-13 times slower in walltime (see Appendix A). Thus, our conclusion remains the same, and is showcased by 2 different network architectures.

Once again, thank you for your comments, they have helped us clarify and improve our work. We hope to have properly addressed your concerns, and would be glad to further discuss and clarify our work otherwise.

Reviewer_GwpyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FHA, a factorized architecture that trains a single set of parameters across all Hanabi player counts from 2 to 5. FHA splits observations into uniform per-teammate tokens processed by shared recurrent networks and aggregated with a fixed symmetric rule, yielding two variants: FHA-H with mean pooling and FHA-T with cross-player attention. Experiments show that multitask FHA matches specialist self-play, transfers zero-shot to unseen player counts, and achieves substantially better cross-play, which the authors attribute to the regularizing effect of training across diverse player counts. The paper further combines FHA with zero-shot coordination methods and finds that Off-Belief Learning improves cross-play only at the first hierarchy level because FHA's recurrence processes private information.

Critical strengths
  1. The architecture is simple and fast. Splitting observations into uniform per-teammate tokens and separating temporal recurrence from cross-player mixing works well, and the 11 to 18 times speedup over UPDeT makes FHA practical.
  2. This work gives useful insight into multi-task training and cross-play, convincingly showing that training one model across all player counts prevents over-specialization, and that heuristic aggregation acts as a regularizer pushing independent seeds toward a shared strategy.
  3. The analysis of zero-shot coordination integration is honest and thorough. The authors report negative results and correctly trace OBL's failure to FHA's per-player LSTMs violating its public-information assumption, exposing a real tension between entity-wise factorization and current zero-shot coordination methods.
Critical weaknesses
  1. Cross-play is only measured between seeds of the same FHA configuration, so it is unclear whether the shared strategy actually coordinates with different architectures such as IQL-LSTM or UPDeT, and the claim of meaningful cross-play is therefore somewhat limited.
  2. The central claim about entity-wise factorization for variable team sizes would be much stronger with an evaluation on a larger or more continuous environment with variable agent counts, such as SMAC or MPE.
  3. The paper identifies why OBL fails at deeper hierarchy levels and leaves the public variant of FHA entirely to future work, which makes the contribution feel incomplete, especially since a public LSTM ablation would be straightforward to run and would clarify the trade-off between self-play and zero-shot coordination.
Questions for authors
  1. How does FHA perform when cross-played with a completely different architecture, such as an IQL-LSTM specialist or UPDeT? The current cross-play results only cover seeds of the same FHA configuration, so it would be helpful to know whether the shared strategy is robust enough to coordinate with fundamentally different agents.
  2. Did you try a public variant of FHA by replacing the per-player LSTM with a public LSTM? If so, how much does it hurt self-play and the multi-task regularizing effect, and does it finally enable OBL at level 2? This seems straightforward to run and would make the analysis much more complete.
  3. Figure 5 only reports FHA-T combined with OBL. Since FHA-H has better cross-play and acts as a stronger regularizer, how does FHA-H perform with OBL, and does the absence of attention mixing reduce the interference with OBL's belief modeling?
1 comment

We have addressed common questions such as the failure modes of OBL (W3/Q2) and the inter-algorithmic cross-play (W1/Q1) in a general comment, and focus on specific comments here.

W2. The central claim about entity-wise factorization for variable team sizes would be much stronger with an evaluation on a larger or more continuous environment with variable agent counts, such as SMAC or MPE.

Hanabi is specifically designed to evaluate cross-play generalization, incorporating key challenges common to real-world multi-agent scenarios: partial observability, communication constraints, theory-of-mind reasoning, and long-term planning. These properties make it well-suited for studying adaptability to new collaborators and settings. Notably, strong performance in Hanabi has been shown to correlate with improved human-AI coordination, with prior work demonstrating that strategies learned in the game transfer effectively to human-AI collaboration (Hu et al., 2020; Hu et al., 2023).

In contrast, SMAC and MPE are mainly for centralized-training-decentralized-execution (CTDE). We take inspiration from these works for variable agent counts (e.g., by using factorized architectures, and parameter weight sharing across players during training) however they are less suited to measure ad-hoc cooperation. For example, MPE is fully observable, while SMAC contains necessary information about allies within sight range.

Thus, while we do agree that other environments next to Hanabi might be suitable as a cross-play generalization testbed, many of existing works (Foerster et al. 2019; Hu et al. 2020; Cui et al. 2022; Hu et al. 2021) focus on Hanabi because it captures the core challenges of pairing with unseen partners. These works have successfully leveraged Hanabi to establish foundational multi-agent reinforcement learning (MARL) concepts that inform broader cooperative AI research.

Q3. Figure 5 only reports FHA-T combined with OBL. Since FHA-H has better cross-play and acts as a stronger regularizer, how does FHA-H perform with OBL, and does the absence of attention mixing reduce the interference with OBL's belief modeling?

At the moment of the initial submission we had only FHA-T data. In the current revision we have curves both for FHA-H (Section 5.4) and FHA-T (Appendix H, Fig. 14), for public and regular (private) variants.

Once again, thank you for your comments, they have helped us clarify and improve our work. We hope to have properly addressed your concerns, and would be glad to further discuss and clarify our work otherwise.

Reviewer_evUdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the zero-shot coordination problem in the cooperative game Hanabi—specifically, how to enable an AI agent to adapt to varying team sizes and effectively cooperate with unfamiliar teammates without prior joint training.

To address this, the authors propose FHA (Factorized Hanabi Agent). Its core idea is to factorize the global state (which encompasses all teammates) into separate individual-teammate representations, processing each through a shared neural network, and jointly training the model across games with different player counts.

Experimental results demonstrate that this approach enables a single model to support multiple team scales while mitigating the traditional self-play issue where agents fail to coordinate with partners outside their training pool.

Critical strengths
  • Jointly training across varying team sizes significantly enhances an agent's ability to coordinate with independently trained, unfamiliar teammates. This demonstrates that the design of the training environment directly shapes the emergence of generalizable cooperative strategies
  • A single parameter set effectively covers 2-to-5-player Hanabi configurations and generalizes to unseen player counts, substantially reducing the need to train specialized models for each team size.
  • While higher-capacity models achieve superior self-play performance, simpler architectures exhibit better compatibility when paired with unfamiliar partners. This suggests that introducing structural constraints can help prevent over-specialization and promote better cooperative generalization.
Critical weaknesses
  • The baseline LSTM only handles fixed player counts, mixing up architectural changes with multi-player training exposure. Authors can add a padded and masked multi-player LSTM baseline to clearly isolate how much gain stems specifically from entity factorization

  • Evaluating zero-shot coordination using models of identical architecture differing only in random seeds is insufficient. To convincingly demonstrate generalizable coordination, the authors should evaluate against an independent pool of heterogeneous partners (e.g., varying architectures, OP/OBL policies, and player count regimes) without using this pool for model selection or tuning

  • A faster epoch doesn't guarantee lower overall training costs. Authors can provide full training details—including environment steps, sampling ratios, model sizes, and hardware—and include learning curves for both sample and wall-clock efficiency

Questions for authors

See Critical Weaknesses

1 comment

Thank you for your feedback, comments and review.

W1. Authors can add a padded and masked multi-player LSTM baseline to clearly isolate how much gain stems specifically from entity factorization

We thank the reviewer for proposing this variant. We have trained a padded LSTM variant of IQL, such that the same fixed-size network can be used across different settings, allowing for multi-setting training. Padding is done only in the part of the observation that carries the partners' hands, all the other parts exist in all settings. We then train this padded LSTM over 2-5P games simultaneously, over 2000 epochs, with 5 seeds. We have updated the results and learning curves in the paper (see Appendix I and Figure 15), and show a summary table here:

Maximum over training epoch cross-play

Training set2P3P4P5P
Padded LSTM-2, 2P4.47 ± 5.722.19 ± 1.901.29 ± 0.751.82 ± 0.36
Padded LSTM-2, 5P0.28 ± 0.240.10 ± 0.180.02 ± 0.030.04 ± 0.04
Padded LSTM-2, 2-5P6.11 ± 4.672.32 ± 2.121.61 ± 1.771.67 ± 0.30

We see that naively using a common observation-space, padded to fit the largest number of possible agents, does not suffice to enable cross-play, even when trained in a multi-player fashion. This suggests that factorization is essential to the effect.

W2. Evaluating zero-shot coordination using models of identical architecture differing only in random seeds is insufficient. To convincingly demonstrate generalizable coordination, the authors should evaluate against an independent pool of heterogeneous partners (e.g., varying architectures, OP/OBL policies, and player count regimes) without using this pool for model selection or tuning

This question intersects with reviewer's Gwpy W1/Q1 and therefore we answer to it in the global thread.

W3. A faster epoch doesn't guarantee lower overall training costs. Authors can provide full training details—including environment steps, sampling ratios, model sizes, and hardware—and include learning curves for both sample and wall-clock efficiency

We have added additional training details in Appendix A. For UPDeT, we show the difference in performance over different transformer layers. Moreover, we provide the wall-clock time per epoch for UPDeT and FHA. UPDeT’s training time is between 8 to 13 times longer than FHA in 2-5P training. However, FHA’s training time doubles between 2P and 5P agents. Despite this, FHA is significantly faster to train than UPDeT.

Once again, thank you for your comments, they have helped us clarify and improve our work. We hope to have properly addressed your concerns, and would be glad to further discuss and clarify our work otherwise.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common response

We thank reviewers for the time they dedicated to the reviewing process, as well as their feedback and reviews. We are happy that they all acknowledge the main findings of our work as a critical strength: by training across multiple player-settings simultaneously, agents trained through self-play can generalize to unseen partners.

We have performed multiple improvements that were asked by several reviewers, that will be of interest to all. We address them here, before addressing individual concerns to each reviewer separately.

Identifying OBL failures

Our previous training pipeline used the policy of OBL level n-1 as the starting point for OBL level n. Changing this to a randomly initialized policy led to policy improvement at level 2 compared to level 1. Moreover, we have modified the state representation such that observations only contain public information. Here are the results, with a control experiment using the private observation variant used initially:

Maximum cross-play

Algorithm2P3P4P5P
FHA-H (lvl 1)13.86 ± 0.4714.83 ± 0.2914.79 ± 0.1514.10 ± 0.18
FHA-H (lvl 2)13.88 ± 1.5514.86 ± 0.5314.61 ± 0.4113.64 ± 0.22
Public FHA-H (lvl 1)11.57 ± 1.3313.06 ± 0.9412.89 ± 0.5612.49 ± 0.18
Public FHA-H (lvl 2)15.27 ± 1.8115.72 ± 0.7914.71 ± 0.5813.84 ± 0.20

Last cross-play

Algo2P3P4P5P
FHA-H (lvl 1)13.86 ± 0.4714.83 ± 0.2914.79 ± 0.1514.10 ± 0.18
FHA-H (lvl 2)9.66 ± 0.2413.29 ± 0.2214.01 ± 0.3013.64 ± 0.22
Public FHA-H (lvl 1)9.71 ± 1.6711.19 ± 1.4012.43 ± 0.4212.18 ± 0.27
Public FHA-H (lvl 2)13.06 ± 0.8313.12 ± 0.3413.90 ± 0.3413.28 ± 0.31

We have updated the results and learning curves for FHA-H in the paper (see Section 5.4 and Figure 5) and for FHA-T in appendix H. Randomly initializing level n+1 policies allow for initial learning, using the level n belief. However, after this initial learning, we observe a decrease in cross-play performance for the non-public variant. Thus, the algorithmic improvements enabled by OBL also apply to FHA, allowing for multi-setting training.

While the main message we convey is cross-play of self-play agents, these experiments show that the FHA architecture, and the multi-setting training paradigm are compatible with other algorithmic improvements to enable cross-play.

Inter-algorithm cross-play evaluations

We are currently performing inter-algorithm cross-play evaluations between LSTM, Updet, FHA-H/T trained with self-play, LSTM, FHA-H/T trained with OBL (level 1 and level 2). We will post the full comparison matrix as soon as it is ready, but preliminary results indicate that self-play agents trained in multi-player settings are able to collaborate with other cross-play algorithms.

Dear reviewers, we have finalized the inter-algorithmic cross-play evaluations. We compare cross-play performance of agents trained in self-play (IQL, FHA-H/T, FHA-H/T-2/3/4/5P, UPDeT, UPDeT-2/5P) and agents trained using OBL (LSTM, FHA, Public FHA). For each setting of nn players, we evaluate all pairs of i,n−ii, n-i agents of one algorithm and another algorithm, respectively. For the same algorithm, different seeds are used to avoid any form of self-play. We provide a short overview of the results here, and have updated the paper (see Section 5.5) for a more comprehensive report. On average across player numbers and algorithms,

ModelMean inter-cross-play
Publ-FHA-H OBL-210.0
Publ-FHA-T OBL-29.9
FHA-H OBL-29.7
UPDeT9.6
FHA-T OBL-28.8
FHA-H 2p8.4
FHA-H8.2
FHA-T7.6
UPDeT 2p7.6
LSTM OBL-27.4
FHA-T 3p7.1
FHA-H 3p6.8
FHA-H 4p6.5
UPDeT 5p6.5
FHA-T 2p6.4
FHA-T 4p6.2
FHA-T 5p3.6
IQL LSTM2.4
FHA-H 5p1.6

As expected, IQL-LSTM is unable to perform cross-play. However, this is not the case for the other methods. The self-play algorithms obtain reasonable cross-play scores, even when paired with other algorithms. Moreover, in general the multi-setting algorithms (FHA-H, FHA-T, UPDeT) each outperform their single-setting counterparts, with the exception of FHA-H-2P which has similar performance to FHA-H. Finally the self-play algorithms can perform cross-play with OBL-trained agents. For OBL, the FHA-trained variant outperforms LSTM-OBL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.