- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper considers the problem of zero-shot coordination in (spatially-induced) partially observable environments. Specifically, the authors focus on a class of methods that learn a VAE-based generative model that captures diverse play styles from gameplay data and then trains a cooperator policy against latent conditioned policies sampled from the generative model.
The authors introduce the Predicting Intention of Partner (PIP) framework that consists of two main components. They generative model is a joint-view VAE that conditions on the local observations of both players, to produce the latent. They also add explicit partner modelling objectives to boost policy learning in the (spatially-induced) partially observable setting.
Critical strengths
-
The paper is easy to read and follow at a high level. The studied problem setting is quite clear and the proposed contributions are clearly stated.
-
The presented experiment results are positive, with PIP outperforming the proposed baselines. The human subject experiments results are particularly encouraging.
Critical weaknesses
The paper has multiple critical weaknesses as described below, and hence I provide a Reject rating.
Problem Setting is not well motivated
-
[Line 102] “To the best of our knowledge, our work is the first to address the failure modes that dynamic spatial partial observability induces in both partner generation and policy learning in ZSC”
-
Why should we explicitly focus on (spatially induced) partial observability? What insights do we hope to gain from this perspective? More importantly, why are the general belief modelling approaches (LBS [4], OBL [5]) and partner modelling approaches ([1],[2],[3]) from prior work not applicable?
The proposed method introduces multiple novel design choices when simpler alternatives are available.
The paper also lacks the necessary ablations to justify these design choices. Additionally, some of these design choices appear very specific to grid-world-like environments, raising concerns about the scalability of the proposed approach.
-
In Eq. (2), how is the union over observations computed? Does it scale to high-dimensional observations? Why not simply concatenate the local observations as is commonly done in the literature (e.g., MAPPO)?
-
If the challenge is that the partner disappears from the field of view, how would distilling the posterior from the teacher help? The teacher, with the full POV, can trivially locate the partner, whereas the student’s local observation simply does not contain this information, making the task impossible.
-
What is the motivation to maintain a student and a teacher VAE? Why can’t the teacher VAE be used to train the cooperator policy?
-
Missing Ablations:
-
[Line 239] Obs masking to remove “direction orientation shortcut” — what does this mean? What happens if you omit the mask?
-
Action prediction target — can you ablate against the simple target of predicting the next action?
-
Location belief
- Can you ablate against the simple case of predicting the next location?
- How is this information accessible during training? Are you assuming access to the underlying state s during training? If yes, could you pass s to the VAE as input?
-
Why should the PPO update not affect the hidden belief state?
- What happens if it does?
- How important is u_t?
- L_aux is not defined
-
Experimental Details are not clear for reproducibility
-
The paper introduces a large number of HPs, but says very little about how they are tuned. Which HPs are copied from the baselines, and which ones are tuned? Do you provide the baselines with the same HP tuning budget?
-
It is unclear whether the agent is optimised using IPPO or MAPPO
-
To my understanding, the Melting Pot env does not use a common reward, but the authors say they maintain the original rewards. This is confusing.
-
It is unclear what the error bars in Tables 1-3 represent. Which part(s) of the entire pipeline are trained using independent random seeds?
-
What do the Fixed sigma and Het sigma baselines in Figures 3–4 correspond to? Is one of them Gamma?
Baselines
- There is a large body of prior work on prediction of partners’ observations and actions ([1],[2],[3]). How does the proposed method compare to this?
References
-
Georgios Papoudakis, Filippos Christianos, and Stefano V. Albrecht. Agent Modelling under Partial Observability for Deep Reinforcement Learning. NeurIPS, 2021.
-
Arrasy Rahman, Ignacio Carlucho, Niklas Höpner, and Stefano V. Albrecht. A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning. Journal of Machine Learning Research, 24(298):1–74, 2023.
-
Dung Nguyen, Hung Le, Kien Do, Svetha Venkatesh, and Truyen Tran. Social Motivation for Modelling Other Agents under Partial Observability in Decentralised Training. IJCAI, pp. 4082–4090, 2023.
-
Hengyuan Hu, Adam Lerer, Noam Brown, and Jakob Foerster. Learned Belief Search: Efficiently Improving Policies in Partially Observable Settings. arXiv:2106.09086, 2021.
-
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, and Jakob Foerster. Off-Belief Learning. ICML, PMLR 139:4369–4379, 2021.
Questions for authors
Please answer:
- Question 2 on problem setting
- Questions 1-4 on methods
- Question 1 on reproducibility
- Question 1 on baselines
19 comments
[PS1, PS2] Why viewpoint-induced partial observability, and how it relates to LBS, OBL and agent modelling
Why focus specifically on spatially induced PO in ZSC?
We acknowledge that spatial partial observability has been studied in MARL, including the works cited by the reviewer [1-3], and that PO has also been explored in ZSC, including Hanabi, OvercookedV2, and ERS. Unlike Hanabi, where information is hidden by game rules, spatial PO causes the partner to dynamically leave and re-enter the ego agent's field of view depending on their relative positions and movements. However, it is important to distinguish the coordination strategy compatibility problems primarily addressed in existing PO-ZSC research from the problems that dynamic spatial PO introduces into the ZSC learning process itself. In generative partner-based ZSC, diverse partner policies are generated to train a Cooperator capable of coordinating with unseen partners. Therefore, our motivation is not simply that such observation loss requires hidden-state inference, but that it can affect both Partner Generation and Policy Learning.
How does this differ from existing belief- and agent-modelling approaches?
LIAM [1], GPL [2], and Social Motivation [3] primarily model other agents under partial observability to improve decision-making, while LBS [4] and OBL [5] address belief-based decision-making and coordination. These approaches may be applicable or complementary to Policy Learning under PO, but do not directly investigate how limited observations affect the generation of diverse training partners in generative ZSC. Specifically, dynamic spatial PO introduces two distinct challenges:
(1) Partner Generation: Spatial PO can make distinct partner behaviors appear similar in local trajectories. When important behavioral segments occur outside the FOV, the VAE encoder may fail to distinguish behavioral styles, leading to overlapping latent representations and potentially limiting training partner diversity.
(2) Policy Learning: When the partner leaves the FOV, the cooperator loses direct access to its location and actions. In ZSC, the cooperator must coordinate with partners whose behaviors were not encountered during training, requiring it to infer hidden states and behavioral tendencies from its limited observation history.
Although these two problems are related, addressing one does not automatically resolve the other. PIP mitigates partner representation ambiguity through the Joint-view VAE and infers the location and actions of unobserved partners through Partner-state Belief.
What insights do we gain from studying Spatial PO?
For Partner Generation, Fig. 3 and Supplementary Fig. F.1 show improved partner-specific latent separation through Joint-view VAE distillation, despite the student using the same local observations as the baseline VAE. Fig. 4 further shows improvements in coverage and MMD, supporting better latent distribution quality.
For Policy Learning, Fig. 5 shows that removing the Belief message changes the policy's action distribution more strongly shortly after the partner leaves the FOV, with the effect decreasing as Time Since Last Seen (TSLS) increases. Fig. 6 further shows larger coordination reward reductions for partners with higher occlusion ratios. These findings indicate that the contribution of inferred partner information depends not only on partner visibility, but also on the timing and frequency of dynamic observation loss. Table 3 further shows that combining JV-VAE and Partner-state Belief achieves higher coordination performance than either component alone.
Therefore, our novelty claim in Lines 102–103 concerns not Spatial PO itself, but the joint analysis and mitigation of two distinct failure modes induced by dynamic spatial PO in ZSC, namely behavioral representation ambiguity during Partner Generation and hidden partner-state uncertainty during Policy Learning.
[M1] Forming the union view, its scalability, and why not concatenation
In Burrito-PO, each local observation is a map-sized grid with cells outside the agent's field of view masked, so both observations share the same coordinate frame. The union in Eq. (2) combines visible regions cell-wise, leaving cells unseen by both agents masked. Compared with direct concatenation, , our teacher input has the same dimensions and encoder memory requirements, adding only a parameter-free cell-wise operation. The union provides a unified spatial representation of both views, while identifies regions visible to the modeled agent. This additional operation is performed only during VAE training, since the teacher is not used during Cooperator PPO training or deployment. We will clarify these details in Appendix C.
[M2, M3] Motivation for teacher-student distillation and partner generation
We apologize that Sec. 4.1 did not sufficiently clarify the purpose of distillation. First, the teacher does not observe the full global state, but uses the union of both agents' local observations (Eq. (2)). More importantly, the student is not trained to reconstruct an unobserved partner's location. Since the purpose of the VAE is to generate diverse training partners, both encoders infer a posterior over the modeled agent's behavioral latent , rather than its hidden location. The teacher uses richer joint-view evidence to distinguish behavioral modes, while the student learns from local observations through posterior distillation. Although the student cannot recover unobservable information exactly, the forward KL in Eq. (4), considering only the distillation term and an unrestricted student distribution, has the optimum
Thus, the student learns the teacher posterior averaged over information unavailable in its local trajectory. Figures 3–4 and Supplementary Fig. F.1 show improved latent separation and distribution quality, while Table 3 shows a reward improvement from 163.3 to 181.6.
After distillation, the student decoder generates training partners using only local observations. The teacher decoder could also be used if it operates on local observations. However, we use the student decoder because it is trained on local-view trajectories with posterior distillation, whereas the teacher VAE learns its posterior from richer joint-view evidence. Since we have not directly compared the two decoders, we do not claim that the teacher decoder would necessarily perform worse.
[M4.1] Obs masking
The mask in Line 239 is applied to the agent-orientation channel only in the belief-module input. Since orientation is encoded at each agent's position, this channel directly provides a visible partner's facing direction, which could let the belief module shortcut Behavioral Belief prediction rather than rely on the observation history. The actor retains this channel for action selection, while cells outside the field of view are already masked across all channels at the environment level for both the actor and the belief module. Thus, omitting the mask would not expose any out-of-FOV information or change the spatial PO setting, but the mask is necessary to block this orientation shortcut.
[M4.2, M4.3.1] Ablation of simpler prediction targets
We conducted the requested ablations by replacing either the Behavioral Belief or Location Belief prediction target while keeping all other settings identical to PIP (Table R9). Replacing the reaction and tendency targets with next-step action prediction reduces reward from to . For Location Belief, predicting the next location () with a one-hot target yields , while using the uncertainty-aware soft target(Eq. (6)) improves performance to . These results support the benefits of multi-step Behavioral Belief targets and uncertainty-aware, multi-offset Location Belief prediction.
Table R9. Ablation of alternative Behavioral Belief and Location Belief prediction targets on Burrito-PO Open. All other settings are identical to PIP. Results are mean reward SE.
| Alternative prediction target | Reward ↑ |
|---|---|
| Next action | 162.1 ± 6.6 |
| Next location (hard, ) | 84.0 ± 5.2 |
| Next location (soft, ) | 219.9 ± 7.0 |
| PIP (original targets) | 249.0 ± 12.2 |
[M4.3.2] How is this information accessible during training? Could you pass to the VAE?
Our setting is motivated by embodied agents, such as robots, that can access their own local observations and record their own positions and actions, but may not have access to the global state . Under this assumption, we consider each agent's logged positions and actions available as training-time supervision for the belief heads. Following the same motivation, we deliberately use the union of both agents' local observations rather than the global state for Joint-view VAE. We will clarify these information-access assumptions in Secs. 3 and 4.1.
Table Reference Convention. Throughout our responses, numbered tables and figures (e.g., Tables 1–3, Fig. 2) refer to the main manuscript, letter-prefixed tables and figures (e.g., Table F.2) refer to the supplementary material, and Tables R9–R12 are provided directly in this rebuttal.
Clarification on Representation-Based Agent Modeling and Partner Generation
Before addressing the individual questions, we clarify the distinction between representation-based agent-modeling methods [1-3] and partner-generation methods such as GAMMA, GOAT, and PIP, particularly regarding their feature and optimization pathways.
In representation-based agent modeling, representations inferred from the controlled agent's interaction history directly condition its policy or action-value function. LIAM [1] employs an encoder-decoder architecture rather than a VAE, using reconstruction of other agents' observations and actions to learn a representation for policy conditioning, as illustrated by
zt=fϕ(htego),πC(atego∣htego,zt).
GPL [2] uses a variational belief posterior in its action-value computation, while Social Motivation [3] similarly uses learned agent representations for decision-making.
In contrast, GAMMA and GOAT use VAE latents to generate training partners rather than directly condition the Cooperator. GAMMA samples a latent from the prior and passes it to a pretrained VAE decoder to generate a partner policy. The Cooperator then learns through environment interactions and rewards obtained with these generated partners, following
z∼p(z),μz(atp∣htp)=pθ(atp∣z,htp).
J(πC)=Ez∼p(z)[V(πC,μz)].
GOAT follows this pathway with adversarial latent selection. The pretrained partner generator remains fixed during Cooperator PPO training. Notably, GAMMA itself distinguishes direct latent conditioning as a separate, orthogonal approach in Appendix J.
PIP follows the same partner-generation pathway, using Joint-view VAE to distill a richer-view teacher posterior into a local-view student whose decoder generates training partners. Neither the VAE posteriors nor the sampled latent directly enters the Cooperator policy. Thus, PIP's distillation improves Partner Generation rather than directly inferring hidden partner states for Cooperator decision-making.