acceptodds
Withdrawn from ICLR 2027

Z2SP: Turning Zero-Shot Coordination into Self-Play via Partner-Specific Test-Time Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot coordination (ZSC) requires independently trained agents to coordinate with unseen partners in cooperative multi-agent reinforcement learning (MARL). Most existing methods learn policies that generalize broadly before deployment. Once a particular partner appears, however, the optimization focus of ZSC shifts from distribution-level generalization to online partner-specific policy estimation and adaptation. When the partner's self-play return exceeds that of the initial ZSC pairing, the current partner policy provides a concrete adaptation target: moving the ego policy toward it can steer coordination toward a self-play-like mode, narrowing the ZSC-to-SP gap and improving ZSC performance. We present Z2SP, which constructs a finite-sample approximation of the inaccessible partner policy over encountered observations from online partner-view observation–action samples and directly adapts the ego actor toward this target. This unifies partner-policy estimation and ego-policy adaptation without a separate partner model. All adaptation uses samples collected only after the initial zero-shot encounter. For persistent adaptation, Z2SP retains its adaptation state across episodes, reuses accumulated samples through delayed history replay, and employs closed-loop meta-initialization to mitigate early adaptation cost. Across five OvercookedV2 layouts with unseen partners from four MARL families, Z2SP recovers 91.8% of the initial ZSC-to-SP gap after 40 episodes of persistent adaptation and averages 98.0% recovery over the final 20 episodes.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, zero-shot coordination, test-time adaptation, self-play

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_GNesOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Z2SP addresses the coordination gap that arises when policies that perform well in self-play interact with unfamiliar partners. Its central idea is to adapt the ego policy toward the current partner's policy, seeking to recover the coordination behavior underlying that partner's self-play performance. After each joint transition, the method uses the partner's observation and executed action to update the ego actor through supervised imitation, while the ego continues to act on its own observations. Persistent adaptation retains the actor, optimizer state, and replay buffer across episodes; delayed history replay reuses past samples, and offline meta-initialization aims to reduce early adaptation costs. Experiments use five fully observable OvercookedV2 layouts and a common set of 20 unseen, self-play-trained partners from four algorithm families excluded from meta-training. Across 100 adaptation episodes with alternating roles, performance progressively approaches partner self-play. Additional experiments examine adaptation after partner switches, the importance of role coverage, auxiliary components, and online computational cost.

Critical strengths
  1. Direct partner imitation is a simple way to obtain supervision without estimating reward-based policy gradients during deployment. Pairing partner actions with partner-view observations makes the update coherent under compatible observation encodings. The experiments demonstrate that this approach can substantially improve coordination with fixed, self-play-trained partners.

  2. The common held-out partner set is a useful control. Comparing against the same partners and separating meta-training and test algorithm families provides more meaningful evidence than testing compatibility only among policies produced by the same training procedure. Multiple adaptation seeds and partner-level results help assess consistency.

  3. The role ablation exposes an important boundary of the approach. The appendix also reports transient regressions and heterogeneous adaptation rates, making the evidence more informative than an endpoint average alone.

Critical weaknesses
  1. The central persistent-adaptation result lacks a matched adaptive comparison. The baseline comparison covers the short-term setting, while the main long-horizon experiment measures Z2SP's improvement over its own initialization. TALENTS' online partner inference [1] and CooT's cross-episode context [2] provide relevant alternatives despite frozen network parameters. Comparisons under the same partners, information access, interaction budget, and role schedule are needed to establish Z2SP's relative value and interaction efficiency. Checkpoint returns also leave the cumulative reward earned during adaptation unresolved.

  2. The methodological increment is modest, and the auxiliary gains are not adequately attributed. Conditional imitation learning already adapts a shared policy's strategy component through likelihood updates on observed partner actions [3]. Z2SP's full-actor, self-play target is distinct, but the supervised update offers limited novelty. The core-only ablation supports the basic mechanism; however, DHR adds optimizer updates, and meta-initialization adds offline supervision. Controls against ordinary replay with matched update/sample budgets and supervised initialization using the same offline examples are needed to establish the value of these designs. This attribution is particularly important given the modest methodological advance.

  3. Near-self-play recovery depends strongly on role coverage. Training on partner-view observations does not necessarily teach the behavior required from the ego's role. The fixed-role results on More Fun and Fun Symmetries remain far below alternating-role performance, including under same-role evaluation (Table S3). Although explicitly acknowledged, this is a substantive restriction: the headline result depends on observing the partner in both responsibilities on these layouts. Full observability and compatible encodings alone do not supply that coverage.

  4. Partner diversity is insufficiently characterized. The common held-out pool is a strength, but its selection procedure is under-specified, and different training algorithms do not necessarily produce different conventions or response requirements. Behavioral and response-diversity analysis, as motivated by ZSC-Eval [4], would clarify what the evaluation actually covers. Since all partners are self-play-trained, the results establish self-play recovery on this pool but provide limited evidence about when imitation remains effective across heterogeneous partner behaviors.

[1] Li et al. Adaptively Coordinating with Novel Partners via Learned Latent Strategies. NeurIPS, 2025. Paper.

[2] Wang et al. CooT: Learning to Coordinate In-Context with Coordination Transformers. ICML, 2026. Paper.

[3] Shih, Ermon, and Sadigh. Conditional Imitation Learning for Multi-Agent Games. HRI, 2022. Paper.

[4] Wang et al. ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination. NeurIPS Datasets and Benchmarks, 2024. Paper.

Questions for authors
  1. On two representative layouts, how does Z2SP compare with a persistent partner-inference or context-adaptation baseline under the same partners, observations, 100-episode budget, and role schedule? Please report cumulative interaction reward alongside frozen-checkpoint returns.

  2. Do DHR and meta-initialization retain their gains against ordinary replay with matched update/sample budgets and supervised initialization using the same offline examples? Please report the additional training budget and complete meta-training settings.

  3. Can imitation errors measured separately on ego-view and partner-view observations explain the fixed-role deficits and identify when partner-action supervision transfers successfully to the ego's role?

  4. How were the five test policies per family generated and selected, and which distinct conventions do they represent? Does the result extend to a small independently specified partner family with compatible observations, actions, and role coverage?

Reviewer_B2TsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Z2SP adapts an agent to an unseen partner by imitating the partner’s observed actions. It combines online behavioral cloning, delayed replay, and meta-learned initialization. Experiments cover five OvercookedV2 layouts and 20 held-out partners. With alternating roles, the method reports reaching 98% of partner self-play return over the final 20 checkpoints of a 100-episode adaptation run.

Critical strengths

The method is simple and avoids a separate partner model. Its improvements across held-out partner families provide a useful proof of concept for partner-specific imitation. However, this evidence supports only a narrow setting and does not outweigh the weaknesses below.

Critical weaknesses

Validation is narrow and strongly dependent on role switching. All experiments use five layouts of one fully observable, two-agent gridworld. More importantly, the headline results rely on alternating roles during adaptation. On More Fun Coordination and Fun Symmetries, fixed-role adaptation evaluated in the same role achieves only 43.8 and 57.6, compared with 178.7 and 194.6 under alternating-role adaptation (Tables S3 and 2). This substantially limits the practical applicability of the approach.

The comparisons do not establish a compelling advantage. The main persistent-adaptation experiment lacks competing methods evaluated with the same 40–100-episode interaction budget. In Table 1, Z2SP after one adaptation episode scores 79.0, below TALENTS at 98.7 and PPO-SA at 92.5. Approaching partner self-play after extensive interaction does not establish superiority over alternative adaptation methods. Furthermore, checkpoint evaluations do not show the reward actually earned during adaptation.

Questions for authors

Clarification: Do the compared methods receive equivalent partner observations, persistent interaction history, and total training budgets, including the additional meta-training used by Z2SP?

Evidence needed to assess the current claims: Under matched interaction and optimization budgets, how does Z2SP compare with ordinary replay-based behavioral cloning and competitive online adaptation methods? Please distinguish checkpoint performance from cumulative reward earned during adaptation.

Reviewer_KGJUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a framework for test-time adaptation in zero-shot coordination within fully observable environments. Instead of relying solely on pre-deployment generalization, it captures online partner-view observation-action samples to build a finite-sample estimate of the partner's behavior. The core mechanism then directly updates the ego agent's policy using a cross-entropy loss to match this estimated partner policy, effectively steering the joint coordination toward a self-play-like mode to narrow the initial performance gap. The empirical validation is conducted primarily within the OvercookedV2 benchmark, testing the approach across five distinct grid layouts against 20 unseen partners generated from four different MARL algorithmic families.

Critical strengths
  • 1. The paper is well-written and structured. It clearly articulates the online partner-specific adaptation, making the motivation and core methodology easy to follow.

  • 2. The empirical results on the OvercookedV2 benchmark are good. Evaluated across five diverse layouts and 20 unseen partners from four distinct MARL families (IQL, VDN, MAAC, HAPPO), Z2SP consistently drives coordination performance close to the partner's self-play level, reaching an average of 98.0% of self-play return over the final 20 episodes.

  • 3. The introduction of Delayed History Replay and closed-loop meta-initialization are smart, practical additions that directly address the pitfalls of online learning. The detailed ablations prove that DHR effectively reuses accumulated samples to accelerate mid-stage adaptation, while the meta-initialization successfully mitigates the early performance drops that typically plague test-time updates.

Critical weaknesses
  • 1. Framing this work under "Zero-Shot Coordination" is conceptually inaccurate. Because Z2SP continually updates the ego policy's parameters during test-time inference based on live interactions, it is fundamentally performing online adaptation or ad-hoc teamwork. True zero-shot coordination implies successfully interacting with an unseen partner without relying on in-episode parameter updates.

  • 2. The objective function is entirely driven by pulling the ego policy toward the estimated partner policy. Consequently, the adaptation mechanism does not meaningfully retain the unique capabilities or conventions of the base ego policy over the long term. In the limit, the ego agent simply becomes a clone of the partner. This suggests that even a randomly initialized policy could achieve good coordination given enough steps, raising questions about whether the method is actually leveraging pre-trained capabilities or just performing online imitation learning from scratch.

  • 3. The method's core methodology is constructing the finite-sample estimate, this requires the ego agent to perfectly pair the partner's executed actions with the exact environment observations the partner saw. This mandates fully observable environments. Consequently, the approach will not easily scale to partially observable domains like Hanabi, where the ego agent cannot access the hidden state information that drove the partner's decision-making.

  • 4. The framework's ability to accurately estimate the partner policy relies on encountering a wide diversity of partner behaviors across the state space. The experimental protocol heavily relies on alternating the ego agent's role across episodes to expose it to the full scope of the partner's strategy. However, in many complex games and real-world ad-hoc teamwork scenarios, exhaustive exposure to different states or strategies is simply not feasible. Agents are often assigned fixed operational roles and cannot swap responsibilities or force specific trajectories purely to gather behavioral data. The paper's own fixed-role ablation highlights this vulnerability: performance drops significantly on asymmetrical tasks like More Fun Coordination when the ego agent is locked into a single role, proving the method struggles when state and strategy exposure is naturally restricted.

  • 5. Because Z2SP requires continuous gradient-based updates (via the Adam optimizer) during test-time interaction, it imposes a much higher computational burden than standard forward-pass inference.

  • 6. The framework's core motivation assumes that matching the partner's "self-play" mode is the ideal outcome. However, if the unseen partner is sub-optimal, noisy, or a human novice, overwriting the ego policy to mimic the partner could actively degrade task performance. A truly robust ZSC agent should ideally compensate for a flawed partner rather than unconditionally adapting to mirror their mistakes.

Questions for authors
  • 1. Why is the work framed as ZSC rather than online adaptation or ad-hoc teamwork, given that it relies heavily on continuous in-episode parameter updates during test-time?

  • 2. Since the objective function explicitly minimizes the cross-entropy against the partner's actions, to what extent does the adapted ego policy retain its original capabilities? Have you evaluated whether a randomly initialized policy would eventually achieve similar coordination performance purely through this online imitation?

  • 3. How can the finite-sample estimation scale to partially observable environments (e.g., Hanabi), where the ego agent cannot directly observe the exact hidden state information driving the partner's actions?

  • 4, Given the performance regressions in the fixed-role ablations for asymmetric tasks (like More Fun Coordination), how would the method handle real-world ad-hoc teamwork domains where agents are locked into fixed operational roles and cannot swap responsibilities to gather behavioral data?

  • 5. How does the ego agent behave when paired with a highly sub-optimal, noisy, or human novice partner? Does unconditionally mimicking the partner's "self-play" mode in these cases actively degrade overall task performance compared to a more robust, compensatory strategy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.