acceptodds
Under review as a conference paper at ICLR 2027

Partner as Task: Belief-conditioned Successor Features for Zero-shot Coordination

16 views

Abstract

Zero-shot coordination (ZSC) requires cooperation with partners unseen during training. We view each partner as a task: the reward remains fixed, but the partner changes the ego-agent's effective dynamics and hence how it should act. Learning these dynamics is difficult when the partner policy is unobserved, and reward is sparse. We introduce belief-conditioned successor features (BSFs), which summarize discounted future features conditioned on a belief over the partner's latent type. The belief is inferred from interaction history and augments the state. We show that the value of a belief-independent policy is linear in the belief, connecting belief inference to task conditioning in standard successor features. We approximate BSFs in MA-JEPA, an on-policy self-play (SP) algorithm that combines a learned history encoder with a temporal difference latent prediction objective. This objective provides reward-free supervision for long-term predictive representations. Across Overcooked and SMAC, MA-JEPA outperforms self-play, population-based, and agent-modeling baselines, with the largest cross-play gains over the strongest baseline under sparse rewards () and RGB observations ().

Primary area:
reinforcement learning
Keywords:
Zero-shot coordination, Multi-agent, Reinforcement Learning.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1RsnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

For zero-shot coordination, this paper introduces belief-conditioned successor features (BFS), which summarize discounted future features conditioned on a belief over the partner's latent type. This paper approximates BSFs in MA-JEPA, an on-policy self-play (SP) algorithm that combines a learned history encoder with a temporal difference latent prediction objective. This objective provides reward-free supervision for long-term predictive representations. Experiments evaluate the proposed algorithm with promising results.

Critical strengths

The evaluation results seem promising.

Critical weaknesses
  1. The theory assumes an exact belief, but MA-JEPA learns an approximate embedding z_t with no guarantee of posterior sufficiency. Proposition 5.4 only covers belief-independent policies, while the actual policy conditions on z_t . Thus the theoretical results do not directly justify the trained agent’s behavior.

  2. Novelty seems limited. The distinction from explicit partner modelling (E3T, CBPR, etc) is clear, but the relationship to other latent-predictive / successor-style objectives (TD-JEPA, BYOL-γ, forward-backward representations, intention-conditioned value functions) is discussed only briefly. A more precise statement of what is new versus what is an adaptation of TD-JEPA to the multi-agent case would strengthen the novelty claim. TD-JEPA already combines a JEPA-style predictor with a temporal-difference target to recover successor features. As the coordination treated as a task in this paper, TD-JEPA should work in this coordination case.

  3. Limited comparison to recent ZSC methods in the main text: TALENTS and PACE.

Questions for authors

How does MA-JEPA compare to other latent-predictive / successor-style objectives (TD-JEPA, BYOL-γ, forward-backward representations, intention-conditioned value functions)?

5 comments
AuthorsOpenReview ↗

Response to reviewer's concerns (1/2)

We thank the reviewer for the careful and constructive review, and for finding our evaluation results promising.


W1. Belief and theory

Thank you for raising this important point. We acknowledge that the theory assumes an exact belief, and Proposition 5.4 covers belief-independent policies. We would like to clarify what each mathematical result establishes for the trained agent.

Proposition 5.3. As Section 5.2 states, Proposition 5.3 applies to any fixed policy π(ai∣s,b)\pi(a_i\mid s,b). Therefore, the Markov property, the Bellman recursion in Eq. (8), and its unique fixed point hold for belief-dependent policies. This Bellman recursion gives the TD target in Eq. (10). It does not give the same guarantee for the learned embedding or the jointly trained networks.

Proposition 5.4. Only the linearity in Proposition 5.4 needs a belief-independent policy. We do not apply this identity to the belief-dependent policy of MA-JEPA. As Appendix A.1 states, it motivates our design rather than guaranteeing the behavior of the trained agent. Specifically, it shows that the belief takes the role of the task vector ww in standard SFs.

Learned representation. MA-JEPA does not aim to identify the partner's policy, actions, or memory. The goal is to learn how the partner changes the ego-agent's dynamics, so that the ego-agent can act well in the current situation. Two different partners can require the same ego action when they lead to the same future from that situation. Therefore, a representation can support this prediction without recovering the full posterior. However, the objective does not guarantee that ztz_t keeps sufficient information. Our posterior probe in Section 6.5 shows that ztz_t keeps partner information, but it does not show prediction sufficiency (Reviewer mwRh Q2).

Revision. We added the posterior probe to Section 6.5 and updated Appendix A.1 accordingly.


W2 & Q1. Prior representation learning baselines

Thank you for highlighting this connection. We agree that TD-JEPA is the closest algorithmic predecessor of MA-JEPA, and we have stated it in Appendix A.2. It studies latent successor prediction in a single-agent setting. MA-JEPA addresses a problem that does not occur there: the effective transition depends on an unobserved partner policy, so the process is not Markov in the state (Proposition 4.1). Therefore, we condition the successor recursion on a belief over the partner and analyze its fixed point on the belief-augmented process (Proposition 5.3). In practice, a history embedding approximates this belief. On-policy collection also keeps each history, action, and bootstrap target consistent with the rollout policy and the current partner. As a result, it avoids replay data that may mix behavior policies or partner dynamics.

We also agree that a comparison with representation learning baselines is important. We compare MA-JEPA with representative objectives, including Laplacian, contrastive, BYOL, and BYOL-γ\gamma, with the same encoder and RL networks (Figure 5). None of them approaches MA-JEPA beyond the Easy layout. In the original manuscript, this comparison was in Appendix F.1. We moved it to the main text.

Our ablations in Figure 7 also give two controls. Since TD-JEPA is an offline single-agent method, its closest on-policy adaptation is MA-JEPA w/o belief encoder. It keeps the latent TD objective but removes the partner-conditioned history. MA-JEPA w/o BSF predictor keeps the same history encoder, PPO objective, actor, and critic, but removes the BSF prediction. Therefore, it isolates the benefit of BSF beyond memory alone. Both controls stay far below MA-JEPA in average XPSPXP_{\mathrm{SP}} (Reviewer 9MgJ W1).

For FB [2] and ICVF [3], the main difference is what defines the task. Both methods condition the prediction on an ego-side variable, namely an ego objective or an ego intention, within fixed environment dynamics. In our setting, the reward stays fixed, and the hidden partner changes the effective dynamics TθT_\theta of the ego-agent. Therefore, two histories can require different predictions even when they end at the same state and ego action (Proposition 4.1).

Revision. We moved the comparison with representation learning baselines from Appendix F.1 to Section 6.2.

AuthorsOpenReview ↗

Response to reviewer's concerns (2/2)

W3. Comparison with TALENTS and PACE

Thank you for this suggestion. We would like to explain why we chose these main baselines. SP, FCP, and E3T mainly differ in how the ego-agent meets diverse partners during training: self-play, a population of partners, or a mixture of the ego policy with a random policy. CBPR adds the inference of the partner's meta-task. Our research question is: beyond partner exposure, can the ego-agent coordinate better if it learns how the partner changes dynamics? For this reason, our main comparison focuses on these exposure-based and agent-modeling baselines.

We agree that TALENTS and PACE are relevant recent methods.

  • TALENTS is particularly relevant because it combines a learned latent strategy space with strategy inference during interaction. Therefore, this gives a direct comparison between explicit strategy inference and our approach, which learns a history embedding through future-feature prediction and PPO. Its original setup uses a learned strategy library from offline trajectories, so we also report its adapted results separately.
  • PACE addresses a related adaptation problem through context-aware exploration and peer identification. Its original setting uses context across multiple episodes. Our evaluation instead measures adaptation within a single episode, without prior interaction with the test partner. This protocol difference motivates reporting its adapted results separately. We nevertheless include both PACE variants in the empirical comparison.

The table reports the average XPSPXP_{\mathrm{SP}} over five layouts.

MethodShaped rewardSparse reward
TALENTS160.372.8
PACE without online adaptation123.968.1
PACE with online adaptation153.880.8
MA-JEPA161.9136.7

Under this protocol, MA-JEPA and TALENTS obtain similar scores with shaped rewards. MA-JEPA has a larger advantage with sparse rewards. Table 3 in Appendix A.3 gives the per-layout means and 95% confidence intervals (Reviewer 9MgJ W4).

Revision. We added a footnote in Section 6 that points to this comparison. We also replaced the five-layout averages in Table 3 (Appendix A.3) with per-layout average performance and 95%95\% confidence intervals.


Closing. We thank the reviewer again for the insightful and constructive comments. We hope that these responses answer the questions of the reviewer, and we will be glad to answer more questions!

Dear Reviewer 1Rsn,

As the reviewer–author discussion phase is approaching its deadline (November 18, AoE), I would kindly ask you to review the authors' responses and provide an update on whether your concerns and questions have been adequately addressed.

As there are differing opinions among the reviewers on this paper, your updated assessment is particularly important in helping us reach a well-informed decision.

Thank you for your time and valuable contribution.

Reviewer_9MgJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies zero-shot coordination by viewing each partner as a task: the reward remains fixed, but the ego-agent’s effective transition dynamics depend on the partner. It introduces belief-conditioned successor features and proposes MA-JEPA, an on-policy self-play algorithm that combines a learned history encoder with a temporal-difference latent prediction objective. Experiments cover shaped- and sparse-reward settings and RGB observations in Overcooked, together with N-agent ad hoc teamwork in SMAC using the SMAX implementation. The reported improvements are most pronounced in the sparse-reward setting.

Critical strengths
  1. The paper studies an interesting problem: learning representations that help coordination with previously unseen partners.
  2. Predicting partner-dependent long-term outcomes provides a reasonable motivation for the auxiliary objective.
  3. The evaluation includes sparse-reward and RGB observations, which are relevant settings for this problem.
Critical weaknesses
  1. Figure 6 and Appendix F.1 mainly report training returns. Without cross-play results for the ablations, it remains unclear whether the gains reflect better partner adaptation or simply stronger self-play policies.

  2. The results largely follow standard belief-MDP and successor-feature arguments. Eq. (5) assumes state-Markov partners, while Proposition 5.4 excludes belief-dependent policies, limiting their applicability to the implemented algorithm.

  3. Tables 1 and 9 combine self-play with cross-play, making improvements in unseen-partner compatibility difficult to isolate. Pure cross-play scores and their confidence intervals should be reported separately.

  4. The single-episode adaptations of PACE and TALENTS are underspecified, Table 2 lacks confidence intervals, and key Transformer hyperparameters are missing. These omissions make the comparisons harder to assess and reproduce.

Questions for authors
  1. Do the Figure 6 ablations retain their cross-play advantage at comparable self-play performance?

  2. Which theoretical conclusions remain applicable when partners have memory and the ego policy depends on the inferred belief?

  3. Does MA-JEPA's advantage persist when self-play entries are excluded from the aggregate scores?

  4. What baseline adaptations and Transformer settings were used, and how variable are the Table 2 results across seeds?

5 comments
AuthorsOpenReview ↗

Response to reviewer's concerns (1/3)

We sincerely thank the reviewer for the careful and constructive review. We have colored the revised part of the manuscript in blue.


W1 and Q1. Cross-play of the ablations

Thank you for highlighting this distinction. As the reviewer points out, training returns alone cannot separate better partner adaptation from better task learning. Figure 7 and Figure 5 (Figure 6 and Appendix F.1 in the original manuscript) show the learning efficiency under sparse rewards. When we remove a component, the agent learns the task less often and more slowly. Therefore, these experiments measure the learning benefit of each component.

To also assess the performance with unseen partners, we evaluate the Figure 7 ablations with the XPSPXP_{\mathrm{SP}} protocol. All methods face the same held-out SPevalSP_{\mathrm{eval}} pool. As Appendix E.2 states, we train this pool with reward shaping and evaluate with sparse rewards only. We report mean XPSPXP_{\mathrm{SP}} scores with 95% confidence intervals over eight seeds. The last column is the average over the three layouts.

MethodCramped roomAsymmetric advantagesForced coordinationAverage
SP96.9 ± 9.8103.4 ± 14.20.0 ± 0.066.8
w/o belief encoder92.4 ± 11.6108.7 ± 17.90.0 ± 0.067.0
w/o BSF predictor112.8 ± 10.2121.5 ± 15.40.6 ± 0.978.3
w/ MSE116.9 ± 9.2134.2 ± 16.81.1 ± 1.484.1
MA-JEPA135.3 ± 8.3223.8 ± 20.586.0 ± 8.3148.4

MA-JEPA achieves the highest average XP performance in all three layouts. Cramped room gives a comparison at similar self-play performance. In this layout, w/o BSF predictor and w/ MSE reach final training returns similar to MA-JEPA. However, MA-JEPA still has a higher XPSPXP_{\mathrm{SP}}. This gap shows better partner adaptation, not only a stronger self-play policy. In Forced coordination, the ablations fail to learn within this budget, while MA-JEPA learns the task and achieves non-zero cross-play returns. There, the gap mainly shows better task learning.


W2 & Q2. Scope of the theory

Thank you for raising this point. We would like to clarify the scope and purpose of our analysis. We use standard belief-MDP and SF arguments to identify the information state and prediction target for hidden-partner coordination. Proposition 5.3 allows belief-dependent ego policies. Proposition 5.4 provides an additional linearity property for a narrower policy class. This policy restriction applies only to the linearity property, not to the Bellman recursion or its fixed-point result.

Partners with memory. As the reviewer points out, Eq. (5) writes the partner policy as π−i(a−i∣s;θ)\pi_{-i}(a_{-i}\mid s;\theta). The marginalization over a−ia_{-i} stays the same, and the input of the partner policy changes from ss to (s,mt)(s,m_t). This suggests that a belief over (θ,mt)(\theta,m_t) can extend Proposition 5.3 to this case. All partners in our experiments use recurrent policies or history encoders, so our XP results already include such partners (Reviewer mwRh W1).

Belief-dependent ego policies. As Section 5.2 states, Proposition 5.3 applies to any fixed policy π(ai∣s,b)\pi(a_i\mid s,b). Therefore, the Markov property, the Bellman recursion in Eq. (8), and its unique fixed point hold for belief-dependent policies. Only the linearity in Proposition 5.4 needs a belief-independent policy.

Learned embedding. The exact-belief guarantees do not automatically apply to ztz_t. Our posterior probe in Section 6.5 shows that ztz_t keeps partner information, but it does not show prediction sufficiency (Reviewer mwRh Q2).

Revision. We added a footnote to Eq. (5) on partners with memory and the posterior probe to Section 6.5.

AuthorsOpenReview ↗

Response to reviewer's concerns (2/3)

W3 & Q3. Cross-play only in SMAC task

Thank you for this suggestion. We agree that pure XP performance makes the comparison easier to assess. In summary, we confirm that the advantage of MA-JEPA persists on most maps after we remove the self-play entries.

For Table 10 (Table 9 in the original manuscript), the off-diagonal entries of Figure 19 show pairwise XP results. The table below averages these entries and excludes all same-algorithm pairings. Values are average win rates with 95% confidence intervals.

MapVDNQMIXIPPOMAPPOPOAMMA-JEPA
3m84.2 ± 7.277.0 ± 8.570.4 ± 9.079.6 ± 7.581.2 ± 8.090.4 ± 5.7
8m55.6 ± 10.250.8 ± 10.146.2 ± 10.159.6 ± 9.748.8 ± 10.860.6 ± 9.7
2s3z94.6 ± 3.979.2 ± 5.885.2 ± 5.097.6 ± 2.084.8 ± 6.398.6 ± 1.3
5m_vs_6m0.0 ± 0.00.6 ± 1.01.2 ± 2.02.2 ± 2.40.0 ± 0.01.6 ± 1.9
10m_vs_11m14.4 ± 6.414.6 ± 6.68.2 ± 3.520.4 ± 6.40.4 ± 0.818.8 ± 5.5
27m_vs_30m0.0 ± 0.00.0 ± 0.04.6 ± 3.18.8 ± 4.10.6 ± 0.612.8 ± 5.0

MA-JEPA retains the highest mean on four maps and ranks second on the remaining two. MAPPO has higher means on 5m_vs_6m and 10m_vs_11m. All methods achieve at most 2.2% on 5m_vs_6m, indicating that this map remains challenging for inter-algorithm coordination.

Additionally, for Table 2 (Table 1 in the original manuscript), XP only evaluations are below.

MapSPMAPPOPOAMMA-JEPA
3m0.70 ± 0.050.83 ± 0.050.85 ± 0.050.89 ± 0.04
8m0.49 ± 0.060.52 ± 0.060.37 ± 0.060.58 ± 0.06
2s3z0.89 ± 0.020.96 ± 0.020.95 ± 0.020.99 ± 0.01
5m_vs_6m0.01 ± 0.020.01 ± 0.010.02 ± 0.010.02 ± 0.02
10m_vs_11m0.07 ± 0.040.07 ± 0.040.00 ± 0.000.17 ± 0.05
27m_vs_30m0.04 ± 0.040.10 ± 0.050.00 ± 0.000.14 ± 0.05

MA-JEPA has the highest mean on five maps and ties with POAM on 5m_vs_6m, where all methods remain near zero. Overall, its advantage persists in most comparisons after self-play entries are removed.

Revision. We added the XP-only version of Table 10 to Appendix F.9 as Table 11.

AuthorsOpenReview ↗

Response to reviewer's concerns (3/3)

W4 & Q4. Reproducibility

Thank you for highlighting these details. We agree that the baseline procedures, encoder settings, and seed variability should be stated explicitly.

Baseline implementations. We use the fixed-share TALENTS variant and follow the original PACE procedure. Each evaluation consists of one episode with an unseen partner, without prior interaction with that partner. We retain the original inference and update rules, with no reset of the adaptation state within the episode. For PACE, the variant without online adaptation disables only the online context update. The variant with online adaptation retains this update throughout the episode. Therefore, our single-episode protocol limits the interaction budget while preserving the original update mechanism.

Transformer. The MA-JEPA history encoder uses two causal Transformer blocks with four attention heads per block. Its hidden dimension is 64, its FFN hidden dimension is 256, and its output dimension is 64. The context length is 16 timesteps, and dropout is disabled. Each input token concatenates the observation features with a learned embedding of the previous action. The encoder uses learned positional embeddings and projects the final token output to the belief embedding. We reset its context buffer at episode boundaries.

Seed variability. Table 3 (Table 2 in original manuscript) summarized performance by averaging over five layouts. To show the variability across training seeds for each layout, we provide the layout-specific means and 95% confidence intervals below.

Shaped reward

MethodCramped roomCoordination ringAsymmetric advantagesCounter circuitForced coordination
FCP128.9 ± 6.5139.0 ± 7.7319.8 ± 6.097.5 ± 15.888.5 ± 13.4
CBPR138.7 ± 10.2138.8 ± 15.1315.2 ± 1.091.6 ± 15.685.8 ± 20.8
TALENTS134.4 ± 13.1144.1 ± 13.2330.4 ± 10.896.8 ± 13.895.9 ± 14.2
PACE (w/o online adaptation)107.8 ± 15.3122.9 ± 19.7285.3 ± 10.261.4 ± 13.642.1 ± 12.2
PACE (w/ online adaptation)134.2 ± 8.2142.3 ± 11.7312.6 ± 14.385.7 ± 12.794.2 ± 7.1
MA-JEPA131.9 ± 10.6149.7 ± 23.9316.0 ± 4.1109.8 ± 20.5101.6 ± 8.3

Sparse reward

MethodCramped roomCoordination ringAsymmetric advantagesCounter circuitForced coordination
FCP100.3 ± 8.5119.5 ± 13.2167.2 ± 15.80.0 ± 0.00.0 ± 0.0
CBPR69.1 ± 10.295.6 ± 12.482.3 ± 8.20.0 ± 0.00.0 ± 0.0
TALENTS129.2 ± 14.1120.4 ± 13.7113.4 ± 19.40.6 ± 0.70.4 ± 0.5
PACE (w/o online adaptation)107.5 ± 17.0100.2 ± 15.8132.8 ± 21.20.0 ± 0.00.0 ± 0.0
PACE (w/ online adaptation)112.4 ± 9.8119.6 ± 11.6163.0 ± 17.35.0 ± 4.24.0 ± 3.5
MA-JEPA135.3 ± 8.3146.6 ± 13.5223.8 ± 20.592.0 ± 4.186.0 ± 8.3

Under shaped rewards, MA-JEPA and TALENTS have similar average scores, with MA-JEPA achieving the highest mean on three layouts. Under sparse rewards, MA-JEPA has the highest mean on all five layouts. The largest gaps occur in Counter Circuit and Forced Coordination, where it achieves 92.092.0 and 86.086.0, compared with 5.05.0 and 4.04.0 for the strongest baseline on each layout.

Revision. We replaced the five-layout averages in Table 3 (Appendix A.3) with per-layout average performance and 95% CI. We also added the number of blocks, the layer sizes, and the context length of the history encoder to Table 4.


Closing. We hope that these responses answer the questions of the reviewer, and we will be glad to answer more questions!

Reviewer_mwRhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses zero-shot coordination (ZSC) in multi-agent RL by introducing belief-conditioned successor features (BSFs), an adaptation of Successor Features prediction. The motivation is that the agent learns how other agents affect its long-term outcomes using an encoder of the interaction history to learn an approximate belief representation. The authors further prove the linearity of the value function in the belief for a belief-independent policy. The paper introduces MA-JEPA, an on-policy self-play algorithm that approximates BSFs, and demonstrates strong performance against other ZSC algorithms, particularly in sparse rewards settings.

Critical strengths
  1. The paper is well written and follows a solid structure. The theoretical formulation is rigorous and developed under explicit assumptions, and the ablations demonstrate the contribution of different components.
  2. The results show substantial improvement mainly in MARL environments with sparse rewards. The authors do a good job pinpointing limitations, e.g., the encoder is not guaranteed to recover the posterior, Proposition 5.4 does not cover their trained policies, and a shared embedding may not disentangle individual teammate types.
Critical weaknesses

My main concern is regarding the assumption that the partners’ policies are memoryless given the state and fixed partner type: π−i(a−i∣s;θ)\pi_{-i}(a_{-i}\mid s;\theta), which leads to defining the transition kernel in equation 5. However, the algorithm MA-JEPA uses partners with policies that condition on the latent variable zz. Thus, even with fixed networks, the same physical state can yield different partner action distributions as the partner’s history embedding changes. Consequently, a belief over partner type alone does not necessarily achieve the Markov property, so Proposition 5.3 does not directly cover the implemented self-play setting.

Questions for authors
  1. The analysis assumes full observability of sts_t, whereas some of the environments are partially observable, e.g., SMAC. Could the authors clarify the scope of Proposition 5.3 in these settings? A sufficient belief may also need to represent uncertainty about the physical state in such settings.
  2. A discussed limitation is the lack of a guarantee that the prediction objective learns a sufficient belief representation. As the paper suggests in Appendix A.1, could the authors evaluate MA-JEPA in a small controlled experiment comparing the learned embedding with an exact posterior? This would help assess how much information about the partner posterior the embedding recovers.
4 comments
AuthorsOpenReview ↗

Response to reviewer's concerns (1/2)

We thank the reviewer for the careful and constructive review, and for the positive assessment of the writing, the theory, and the stated limitations. We address the weaknesses and the two questions below.


W1. Partners with memory

Thank you for providing a really insightful observation. As the reviewer points out, Eq. (5) marginalizes over the partner's action, and it writes the partner policy as π−i(a−i∣s;θ)\pi_{-i}(a_{-i}\mid s;\theta). This policy conditions only on the state and the partner type. Therefore, as stated, Proposition 5.3 does not directly cover the implemented self-play setting, where the partner also conditions on its own belief embedding.

We would like to clarify our motivation. MA-JEPA does not aim to identify the partner's policy, actions, or memory. The goal is to learn how the partner changes the ego-agent's dynamics, so that the ego-agent can act well in the current situation. Two different partners can require the same ego action when they lead to the same future from that situation. From the ego-agent's perspective, a partner shows its character only through its actions, and these actions shape the effective dynamics. This holds whether or not the partner has memory.

Under this view, partner memory is one more input that shapes the partner's action. The marginalization over a−ia_{-i} in Eq. (5) stays the same. The input of the partner policy changes from ss to (s,mt)(s,m_t), so the effective dynamics also depend on mtm_t. This suggests that the construction can extend to partners with memory. The hidden variable would then include the partner memory mtm_t, with belief bt=p(θ,mt∣τt)b_t=p(\theta,m_t\mid\tau_t), and the belief update would propagate mtm_t through the fixed update rule of the partner. We expect the belief-augmented process to remain Markov, so that the argument in Appendix B.2 can carry over.

Empirical support. In all our experiments, we consider recurrent policies (e.g., LSTM and GRU) for the partners to have memory. The self-play partner of MA-JEPA conditions on its own history encoder. Therefore, the XP results in Section 6 already evaluate MA-JEPA with partners whose actions depend on their history. In these settings, MA-JEPA achieves the best (or comparable) average XP performance. Note that these results give empirical support for the view above, although they do not replace a formal guarantee.

Revision. We thank the reviewer again for this constructive observation! As a revision, we added a footnote to Eq. (5) in Section 4.

AuthorsOpenReview ↗

Response to reviewer's concerns (2/2)

Q1. Partial observability

Thank you for this important question. We acknowledge the importance of 'partial observability', which is key challenge in multi-agent system. Our problem setup and theoretical analysis assume full observability of the physical state, and Proposition 5.3 establishes an exact-belief guarantee within this scope.

This question is closely related to W1. Partner memory and partial observability both add hidden variables that change over time. Under partial observability, the hidden variable would also include the physical state, i.e., (st,θ,mt)(s_t,\theta,m_t), and the belief would be a posterior over it given the observation history, as in POMDPs. This suggests a hidden-Markov view of the partner-induced task, in which a belief over all hidden variables can restore the Markov property. In Appendix A.1, we already note a related case, where a partner strategy that can switch within an episode is modeled as a latent state. We believe the right next direction is the joint treatment of time-varying partner states and partial observability. The theory of approximate information states [1] can also help relate the learned ztz_t to such a belief.

Empirically, MA-JEPA also performs well under partial observability. The SMAC experiments (Section 6.6, Appendices F.8–F.9) evaluate N-agent ad hoc teamwork with local observations, and Appendix F.6 reports training on Overcooked v2 with 5×55\times5 local observations. These results give empirical support, but the theoretical guarantee still requires an exact sufficient belief.


Q2. Learned embedding versus exact posterior

Thank you for this valuable suggestion. As the reviewer suggests, we additionally run a controlled experiment that compares the learned belief embedding ztz_t with an exact posterior. This experiment measures how much information about the partner posterior the embedding recovers.

Setup. We use eight frozen partners with a uniform prior, and we collect 50 episodes. We compute the posterior reference by replaying each candidate partner and testing all its actions against the observed transitions. We sum the probabilities of all actions that produce the observed next state. This computation does not use the actual partner action.

We train a linear softmax probe against the posterior reference with soft-label cross-entropy. All encoders remain frozen. We compare MA-JEPA with two controls. The random history encoder has the same architecture as the learned encoder, and it receives the trained observation features and the previous ego actions. The observation-only control uses ϕ(ot)\phi(o_t) without history.

Results. The table reports the KL divergence from the posterior reference to the probe prediction at t=200t=200. We report mean ±95%\pm 95 \% CI across four seeds. The last column is the KL divergence from the posterior reference to the uniform prior, i.e., the value of a prediction with no information about the partner. Note that lower is better in this table.

LayoutMA-JEPARandom history encoderObservation encoder, no historyUniform prior
Coordination ring1.14 ± 0.381.75 ± 0.091.74 ± 0.212.04
Cramped room1.24 ± 0.211.73 ± 0.361.70 ± 0.362.02
Asymmetric advantages0.78 ± 0.241.17 ± 0.041.11 ± 0.031.11

MA-JEPA has the lowest mean probe KL on all three layouts. Relative to the uniform prior, MA-JEPA reduces the KL by 30∼4430\sim44%, while the two controls reduce it by at most 1616%. Therefore, the linear probe recovers the posterior more closely from the learned belief embedding than from either control.

Revision. We added this experiment to Section 6.5 and updated Appendix A.1 accordingly.


Closing. We hope these responses answer the reviewer’s questions, and we would be glad to answer any further questions!

Dear Reviewer mwRh,

As the reviewer–author discussion phase is approaching its deadline (November 18, AoE), I would kindly ask you to review the authors' responses and provide an update on whether your concerns and questions have been adequately addressed.

Your updated assessment would be greatly appreciated and will help us reach a well-informed decision.

Thank you for your time and valuable contribution.

Thanks to the authors for the clarifications and added experiment. I am keeping my rating at 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global comment

We express our gratitude to AC for managing the review process and all three reviewers for their insightful feedback. We are pleased to present the updates we have made in response to valuable suggestions, as detailed below. In the revised manuscript, we color all changes in blue.

  • Posterior information in the learned embedding. We additionally run a controlled probe to analyze posterior-probe KL of the proposed solution (mwRh Q2, 1Rsn W1).
  • Cross-play only results. We provide XP results in Table 11 of Appendix F.9. (9MgJ W1 & Q1, 9MgJ W3 & Q3).
  • Baseline procedures and reproducibility. We provide per-layout performance of recent ZSC baselines in Table 3 of Appendix A.3, and report the hyperparameter of the belief encoder (9MgJ W4 & Q4, 1Rsn W3).
  • Theoretical scope. We clarify that Proposition 5.3 allows fixed belief-dependent policies on the exact belief MDP (Section 5.2), and that Proposition 5.4 adds linearity only for belief-independent policies (mwRh W1, 9MgJ W2 & Q2, 1Rsn W1).
  • Partner memory and partial observability. We added a footnote to Eq. (5), which states that a belief over the partner type and the partner memory may extend Proposition 5.3 to partners with memory (mwRh W1 & Q1, 9MgJ W2 & Q2).
  • Relation to prior work. We acknowledge TD-JEPA as the closest algorithmic predecessor in Appendix A.2. Figure 5 now shows the representation-learning comparison in the main text (1Rsn W2 & Q1).

Our perspective MA-JEPA does not aim to identify the partner's policy or intentions. It learns how the partner changes the ego-agent's dynamics, so that the ego-agent can act well in the current situation. Two different partners can require the same ego action when they lead to the same future. BSFs formalize this view as a prediction target for action evaluation, without explicit inference of partner intentions or nested beliefs. The responses distinguish this exact-belief formulation from the learned representation that we assess in our experiments.


Relation to TD-JEPA. We would like to add one clarification about TD-JEPA. This baseline is an offline single-agent method. In our setting, its closest on-policy adaptation is MA-JEPA w/o belief encoder, which keeps the latent TD objective and removes the partner-conditioned history. Its average XPSPXP_{\mathrm{SP}} is 67.067.0, compared with 148.4148.4 for MA-JEPA (Figure 7, 9MgJ W1). Therefore, a direct transfer of TD-JEPA-style prediction is not sufficient without belief conditioning.


We have left individual comments to address each reviewer’s questions and concerns and to clarify statements that were unclear. Please review each individual comment for more details!

Below are the references we use in all our responses.

References

[1] J. Subramanian, et al. Approximate information state for approximate planning and reinforcement learning in partially observed systems. JMLR 2022.

[2] A. Touati, et al. Learning one representation to optimize all rewards. NeurIPS 2021.

[3] D. Ghosh, et al. Reinforcement learning from passive data via latent intentions. ICML 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.