acceptodds
Under review as a conference paper at ICLR 2027

JaxAHT: A JAX-Based Library for Ad Hoc Teamwork

Abstract

As autonomous agents are increasingly deployed, they must be able to coordinate with previously unseen agents, such as humans. This challenge has been formalized as Ad Hoc Teamwork (AHT), and has attracted a substantial body of research. However, progress has been hindered by the prohibitive computational cost of the AHT research lifecycle, lack of standardized benchmark implementations, and the absence of a diverse, validated evaluation teammate suite. This work introduces **JaxAHT**, the first open-source, JAX-based library designed to accelerate and standardize the AHT research lifecycle. Leveraging JAX's hardware acceleration and massive parallelization capabilities, JaxAHT provides a unified framework for teammate generation, ego agent training, and evaluation against unseen teammates, achieving approximately 95x wall-clock speedup over existing PyTorch counterparts. Along with the library, we contribute a diverse suite of evaluation teammates across Level-Based Foraging, Overcooked, and Hanabi. Using JaxAHT, we conduct a large-scale, compute-controlled benchmark study of AHT algorithms, finding that no algorithm consistently performs best, that agent modeling primarily offers benefits in role-based scenarios, and that best response returns are underestimated by reinforcement learning from scratch.

Primary area:
datasets and benchmarks
Keywords:
ad hoc teamwork, software library, benchmark, evaluation, multi-agent reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wgMuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces JaxAHT, a JAX-based framework that unifies teammate generation, ego-agent training, unified AHT training, and evaluation with unseen teammates for Ad Hoc Teamwork (AHT). It integrates several existing algorithm classes, provides evaluation teammates and empirical best-response (BR) references across LBF, Overcooked, and Hanabi, and releases a new LBF human gameplay dataset. Runtime comparisons, cross-play and behavioral trajectory analyses, and systematic hyperparameter tuning and algorithm comparisons on representative tasks assess the framework’s efficiency, evaluation resources, and existing methods. The main contributions are unified research infrastructure, standardized evaluation resources, and a substantial empirical study.

Critical strengths
  1. The motivation is clear, and the problem has substantial community relevance. AHT research requires coordinating teammate construction, ego-agent learning, and generalization evaluation. Fragmented implementations and substantial computational costs limit experimental scale and comparability. JaxAHT integrates these stages into a unified, accelerated framework, lowering barriers to experimentation and reproduction. Its value as infrastructure and a benchmark lies in enabling reusable research workflows and providing a common basis for systematic algorithm comparisons.
  2. The overall design is coherent and connects teammate construction with evaluation. The framework supports combinations of teammate-generation and ego-learning methods, and constructs evaluation suites from heuristic, RL-based, and human-data-derived policies. Empirical BR references aim to account for differences in teammate competence, while cross-play examines whether different teammates require different response strategies. These choices address central requirements of AHT evaluation. In particular, the cross-play results provide useful evidence that the suite contains distinct coordination demands.
  3. The evaluation is broad and systematically organized. The paper examines runtime efficiency, teammate quality and diversity, and algorithm performance across multiple representative tasks. The comparisons use separate validation teammates, extensive hyperparameter searches, controlled interaction budgets, multiple random seeds, and confidence intervals, reducing potential confounding from insufficient tuning and inconsistent experimental settings. The release of code, policies, data, and tuned configurations further strengthens reproducibility and reuse.
Critical weaknesses
  1. Limited methodological novelty. The engineering contribution is substantial, but most constituent elements—including the core algorithms, environments, JAX acceleration mechanisms, cross-play analysis, and BR-normalized evaluation—build on prior work. The paper argues that the unified framework enables studies that were previously difficult because of incompatible implementations, and Sec. 5.3 provides examples of such comparisons. However, the incremental scientific capability enabled specifically by JaxAHT is not isolated very clearly. It would strengthen the paper to show more concretely which findings depend on the unified infrastructure or new evaluation suite, rather than mainly on larger-scale, carefully controlled reimplementation and experimentation.
  2. The 95× speedup claim is insufficiently qualified. The single-seed comparisons in Table 4 show approximately 5× acceleration for LIAM and 25× for BRDiv. The 95× result is obtained for training 300 LIAM seeds on LBF 7×7, relative to an estimated runtime for sequential PyTorch execution. This is strong evidence of a substantial throughput advantage in a highly parallel multi-seed setting, but it does not establish 95× as a general speedup across algorithms, tasks, or workloads. Since efficiency is a central contribution, the abstract and conclusion should state the conditions behind this headline number more explicitly.
  3. Behavioral-diversity validation is not fully independent. The behavioral analysis trains an LSTM classifier to predict teammate–BR pair identity from joint observation–action trajectories and then applies t-SNE to the learned latent representations. Because the representation is explicitly trained for identity discrimination, the resulting cluster separation reflects separability under a supervised objective rather than providing an independent validation of behavioral diversity. In addition, the inputs contain both teammate and BR behavior, with each teammate paired with its own BR, so the analysis cannot clearly attribute the observed separation to the teammate alone. The results therefore show that teammate–BR interaction trajectories are distinguishable, but provide weaker evidence that the evaluation teammates themselves exhibit distinct behavioral conventions.
Questions for authors
  1. Distinctive contribution: What is the most distinctive research contribution relative to existing platforms and evaluation resources? Beyond unified implementation and acceleration, which previously difficult research investigations are enabled by the new teammate suite or evaluation pipeline?
  2. Updating and fixing BR references: The footnote in Section 5.4 states that BR references were revised upward using the best-seen returns for each teammate. At what stage of evaluation were these updates performed? After observing all benchmark agents’ performance, were all methods’ normalized scores recomputed using one shared, fixed set of final references?
  3. Data split for behavioral embeddings: Do the t-SNE visualizations in Figure 3 and Appendix Figure 9 use the classifier’s training trajectories or the held-out test trajectories described in Appendix C? What is the corresponding held-out classification accuracy?
8 comments

We thank the reviewer for the detailed and constructive review.

Distinctive contribution (W1, Q1). Beyond enabling new methods, JaxAHT has enabled research on new AHT paradigms, as shown by two works cited in the paper. These are ROTATE [1], which reformulates AHT as open-ended learning that interleaves teammate generation and ego-agent training, and ICRL4AHT [2], which studies AHT through in-context reinforcement learning, building on JaxAHT’s teammate generation. Both paradigms require tightly coupling teammate generation with ego-agent learning and evaluation, which is straightforward when all three share one codebase and policy interface. We have added this discussion to the revision (Sec. 6).

95× speedup (W2). We agree, and have revised the abstract, introduction, and conclusion to report the measured single-seed speedups (up to 25×; Table 4 reports about 5× for LIAM and 25× for BRDiv).

BR return estimates (Q2). Yes: the BR return estimates were revised after all benchmark agents were evaluated, and all methods’ normalized scores were then recomputed using one shared, fixed set of final estimates. Sec. 5.4 now states this explicitly. We now say “BR return estimates” (previously “upper bounds”) to emphasize that they are not the true BR returns, and have added a sensitivity analysis of method rankings under the original vs. final estimates (App. D.3, Fig. 13). Teammate generation and unified method rankings are unchanged, while ego agent rankings display minor pairwise swaps in LBF 7×7 and Mini Hanabi. All swapped pairs have overlapping 95% CIs under both estimates, and the top-ranked method is unchanged on every task.

Behavioral embeddings (W3, Q3). The classifier is trained on trajectories of every teammate paired with every BR; the t-SNE plots show only held-out trajectories of each teammate with its own BR, unseen during training. Held-out pair-classification accuracy is far above chance and close to training accuracy (Coordination Ring: 56.4% heldout vs. 59.4% train, chance 1/324 = 0.31%; LBF 7×7: 43.8% heldout vs. 44.0% train, chance 1/529 = 0.19%), so the separation is not an artifact of overfitting the supervised objective. App. C now reports the data split and both accuracies.

On attributing the separation to the evaluation teammate alone: the goal of this experiment is to test whether the evaluation set as a whole challenges the AHT agent, not whether teammates differ when paired with a fixed reference teammate. Following Rahman et al. (2024) [3], the space of teammate policies can be partitioned by their BR conventions. A strong AHT agent must be able to play many BR conventions, so a good evaluation set should require diverse response strategies. We therefore embed each teammate with its own BR, which characterizes the teammate’s convention. Note also that a teammate’s behavior often depends on its partner’s: a competent yet lazy teammate may stop contributing if its partner can complete the task alone, yet take on the entire task if its partner is unskilled. A teammate’s behavior must therefore be studied with respect to some partner.

We agree that the t-SNE figures alone do not establish that the teammates require diverse BR conventions; instead, they are intended as complementary evidence. Following prior work [4, 5, 6], our primary evidence of diversity is the cross-play (XP) matrices (App. B), which pair all teammates with all BRs. If teammates require distinct BR conventions, a BR trained for one teammate should perform worse with others, yielding high diagonals and lower off-diagonals. On all nine tasks, teammates score higher on average with their own BR than with other teammates' BRs, with a mean per-teammate gap of 0.19–0.77 normalized return (see table below, which is now Table 5 in App. C). High returns across the suite thus require an AHT agent to model distinct response strategies, although some off-diagonal blocks retain high returns, suggesting partial overlap within agent types (e.g., PPO-generated teammates). The XP matrices alone, however, do not show how much teammate behaviors differ: low off-diagonal returns could arise from brittle policies that fail with slightly out-of-distribution partners while behaving similarly to one another. The t-SNE results rule this out, as teammate–BR pairs occupy largely non-overlapping regions.

TaskOwn BR (diagonal)Other teammates' BRs (off-diagonal)
LBF 7×70.940.62
LBF 12×120.890.70
Asymmetric Advantages0.920.41
Coordination Ring0.970.20
Counter Circuit0.840.27
Cramped Room0.990.21
Forced Coordination0.930.26
Hanabi0.680.21
Mini Hanabi0.940.70

Note that the table above presents returns normalized under the final BR return estimates.

Sec. 5.2 now presents the XP matrices as the primary evidence of diversity.

Citations

[1] Wang et al. ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork. NeurIPS 2026.
[2] Jing et al. Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork. ICML 2026.
[3] Rahman et al. Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents. AAAI 2024.
[4] Agapiou et al. Melting Pot 2.0. arXiv 2022.
[5] Rahman et al. Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity. TMLR 2023.
[6] Nekoei et al. Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi. CoLLAs 2023.

Thank you to the authors for their detailed response and revisions. Most of my concerns have been addressed, with the exception of W1 regarding the paper's scientific novelty.

I acknowledge that JaxAHT is a well-engineered and substantial infrastructure contribution, and I believe it will be a valuable resource for the AHT research community. However, I remain unconvinced by the response to W1. While the framework clearly facilitates further research, I am still uncertain about the extent to which this work itself provides new scientific insights beyond its engineering contributions.

I appreciate the authors' efforts in addressing my concerns and recognize the practical value of this work. I have therefore decided to increase my Soundness score to 3. However, given my remaining reservations about the significance of the paper's scientific contribution in the context of ICLR, I will maintain my overall rating of 2 (Weak Rejection).

Reviewer_ejj1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces JaxAHT, the first open-source,JAX-based library designed to accelerate and standardize the AHT research life cycle.

Critical strengths

The paper addresses a real bottleneck in AHT research: the lack of a unified, accelerated, and multi-environment framework covering teammate generation, ego-agent training, cooperative MARL, and standardized evaluation. The JAX implementation, open-source evaluation teammate suite, LBF human gameplay dataset, and released hyperparameters/BR policies substantially lower the barrier for compute-controlled AHT studies at academic scale.

Critical weaknesses
  1. The novelty is primarily engineering/infrastructure rather than algorithmic, and the paper does not fully isolate what drives the reported gains.
  2. Presentation, terminology, and claim precision need substantial cleanup. There are several typographical and terminology issues:“reliable” should likely be “rliable” (Agarwal et al., 2021);
Questions for authors
  1. The paper should explicitly state that JaxAHT is an infrastructure/benchmark contribution, not a new AHT algorithm, and then justify why the infrastructure itself is a research contribution. Add a feature matrix that lists, for each prior library, which AHT lifecycle stages and environments are supported. Define “full AHT research lifecycle” operationally. This will make the “first library” claim precise and defensible.
  2. The finding in Sec. 5.4 that AHT agents can outperform PPO-derived BR estimates is important, but the paper currently calls the empirical BR reference a “per-teammate upper bound,” which is inconsistent with Fig. 12. Please rename it as an “empirical BR reference/estimate,” quantify the gap between original and best-seen BR bounds, and propagate this uncertainty into the BR-normalized returns. Add a sensitivity analysis comparing algorithm rankings under original BR bounds vs. best-seen BR bounds. Discuss the circularity that AHT agents trained against diverse populations may improve BR estimates, and clarify how future work should report raw and normalized returns jointly.
1 comment

We thank the reviewer for their time and the constructive feedback.

Clarifying contribution type (W1, Q1). We agree that JaxAHT is an infrastructure and benchmark contribution, and the paper is already framed this way: the abstract and conclusion introduce JaxAHT as a library, and the three contributions in Sec. 1 are the library, the evaluation teammate suite, and the benchmark study. Table 1 (Sec. 2) already compares JaxAHT with prior AHT libraries across lifecycle stages (teammate generation, ego agent training, and evaluation teammates), i.e., the requested feature matrix.

The lifecycle is defined in Sec. 1 as generating teammate populations, training ego agents, and evaluating them against held-out teammates (each stage detailed in Sec. 3). We have restated this definition in the Table 1 caption and tightened the “first library” claim in Sec. 1 to match the abstract (“first JAX-based library”). On why the infrastructure is a research contribution, see our response to Reviewer wgMu (W1, Q1).

Could the reviewer clarify which reported gains W1 refers to? We would be glad to address this.

BR return estimates (Q2). We have renamed “per-teammate upper bound” to “BR return estimate” throughout, to emphasize that these values are not the ground-truth BR returns, and added a sensitivity analysis of algorithm rankings under the original vs. best-seen (final) estimates (App. D.3, Fig. 13). Teammate generation and unified method rankings are unchanged, while ego agent rankings display minor pairwise swaps in LBF 7×7 and Mini Hanabi. All swapped pairs have overlapping 95% CIs under both estimates, and the top-ranked method is unchanged on every task.

Sec. 5.4 now adds two recommendations for benchmark users: (1) normalize against the updated BR return estimates from this paper, which we will release as a versioned file with the library, and state the version used; (2) report raw, unnormalized per-teammate returns so normalized results can be contextualized and sanity-checked.

Presentation (W2). We thank the reviewer for catching these issues and have fixed them in the revision.

Reviewer_7REuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

JaxAHT brings teammate generation, ego-agent training, and evaluation into a shared JAX implementation. It provides evaluation partners for LBF, Overcooked, and Hanabi, plus LBF human-play data. The benchmark compares existing AHT methods with separate validation partners and controlled interaction budgets.

Critical strengths
  1. Reusable infrastructure. Policies and populations can be passed between algorithms within one codebase. This addresses a practical obstacle in AHT, where evaluating one component often requires implementing several others (Sections 3–4).

  2. A credible comparison protocol. The study includes teammate-generation costs, tunes on disjoint validation partners, and evaluates five seeds. This makes the finding that no method wins across all tasks informative for choosing an AHT method (Section 5.3; Appendix D).

  3. An important evaluation finding. Some AHT policies outperform PPO responses trained specifically for a fixed teammate. This shows why a learned best response should not be treated as a reliable performance ceiling (Section 5.4).

Critical weaknesses
  1. The 95× speedup needs a narrower headline. Table 4 obtains this figure by comparing 300 parallel JAX seeds with an extrapolated sequential PyTorch runtime. The measured single-seed gains are about 5× and 25×. These are substantial, but the 95× figure should not stand for a general implementation-level speedup.

  2. The effect of updating normalization references is unclear. Section 5.4 updates the BR references using benchmark returns. This is disclosed, but changes the weight of affected teammates in the aggregate score. Figure 12 shows substantial changes for some partners; ranking sensitivity and a fixed reference version would make future comparisons more reliable. These empirical references are not guaranteed upper bounds.

  3. The behavioral embedding supports a narrower diversity claim. The classifier predicts teammate–BR pair identity from joint trajectories, and each plotted teammate uses its own BR. Cluster separation establishes distinct pair behavior, not teammate behavior independently of the response policy. The cross-play matrices remain useful evidence of different coordination demands (Section 5.2 and Appendix C).

Questions for authors
  1. Is a measured comparison with concurrently scheduled PyTorch runs available for the multi-seed setting? Otherwise, please keep the sequential-reference qualification alongside the headline speedup.

  2. Do the main comparisons change under the original versus updated BR references, and which fixed reference version should future benchmark users adopt?

  3. Can the already collected cross-play trajectories test whether teammate separation persists with a common response policy? Otherwise, please limit the embedding conclusion to pair-level behavior.

2 comments

We thank the reviewer for the thoughtful review.

95× speedup (W1, Q1). We agree, and have revised the abstract, introduction, and conclusion to report the measured single-seed speedups (up to 25×; Table 4 reports about 5× for LIAM and 25× for BRDiv). A measured comparison with concurrently scheduled PyTorch runs is not available, as the original PyTorch implementations do not support concurrent multi-seed training. We have therefore removed the extrapolated 95× claim from the abstract, introduction, and conclusion.

BR return estimates (W2, Q2). All methods’ normalized scores use one shared, fixed set of final BR return estimates, computed after all benchmark agents were evaluated. We have added a sensitivity analysis of the main comparisons under the original vs. final estimates (App. D.3, Fig. 13). We now say “BR return estimates” (previously “upper bounds”) since they are not the ground-truth BR returns, and hence not guaranteed upper bounds. Teammate generation and unified method rankings are unchanged, while ego agent rankings display minor pairwise swaps in LBF 7×7 and Mini Hanabi. All swapped pairs have overlapping 95% CIs under both estimates, and the top-ranked method is unchanged on every task. For future users, Sec. 5.4 now recommends (1) normalizing against the updated BR return estimates from this paper, which we will release as a versioned file with the library, stating the version used, and (2) reporting raw per-teammate returns so results can be re-normalized if better BRs are found.

Behavioral embedding (W3, Q3). On pair-level vs. teammate-level behavior: the goal of this experiment is to test whether the evaluation set as a whole challenges the AHT agent, not whether teammates differ when paired with a fixed reference policy. Following Rahman et al. (2024) [1], the space of teammate policies can be partitioned by their BR conventions. A strong AHT agent must be able to play many BR conventions, so a good evaluation set should cover diverse ones. We therefore embed each teammate with its own BR, which characterizes the teammate’s convention. Note also that a teammate’s behavior often depends on its partner’s: a competent yet lazy teammate may stop contributing if its partner can complete the task alone, yet take on the entire task if its partner is unskilled. A teammate’s behavior must therefore be studied with respect to some partner.

Regarding Q3, following prior work [2, 3, 4], the cross-play (XP) matrices (Sec. 5.2, App. B) are intended as the primary evidence of teammate diversity. They already provide the common-partner test the reviewer suggests, pairing all teammates with all BRs. On all nine tasks, teammates score higher on average with their own BR than with other teammates' BRs, with a mean per-teammate gap of 0.19–0.77 normalized return (table below). High returns across the suite thus require an AHT agent to model distinct response strategies, although some off-diagonal blocks retain high returns, suggesting partial overlap within agent types (e.g., PPO-generated teammates). Low off-diagonal returns could, however, arise from brittle policies that fail with slightly out-of-distribution partners while behaving similarly to one another. The embedding visualization is intended as secondary evidence and tests this possibility. Sec. 5.2 now makes this framing explicit, and App. C, Table 5 summarizes the SP - XP return gap (with mean normalized returns under the final BR return estimates).

TaskOwn BR (diagonal)Other teammates' BRs (off-diagonal)
LBF 7×70.940.62
LBF 12×120.890.70
Asymmetric Advantages0.920.41
Coordination Ring0.970.20
Counter Circuit0.840.27
Cramped Room0.990.21
Forced Coordination0.930.26
Hanabi0.680.21
Mini Hanabi0.940.70

Citations

[1] Rahman et al. Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents. AAAI 2024.
[2] Agapiou et al. Melting Pot 2.0. arXiv 2022.
[3] Rahman et al. Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity. TMLR 2023.
[4] Nekoei et al. Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi. CoLLAs 2023.

Thank you for the clarifications and additional analysis. Most of my concerns are addressed, although the embedding results still warrant a narrower interpretation. I remain supportive and will maintain my rating of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.