- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether synchronized first-person recordings of teammates can help a visual model infer the wider team situation from a single player’s view. It introduces X-Ego-CS, a dataset of synchronized Counter-Strike 2 gameplay, and Cross-Ego Contrastive Learning (CECL), which encourages similar representations for teammates’ observations at the same time. Using a benchmark of 33 prediction tasks, the authors find that CECL improves predictions of teammates’ and opponents’ current and future locations, while reducing performance on some self-related tasks, particularly the observer’s speed. They call this pattern the “allocentric shift.” The paper evaluates the result across several visual encoders and conducts controlled experiments to examine the contribution of teammate correspondence beyond generic adaptation and shared contextual cues. A separate two-agent ViZDoom experiment also provides preliminary evidence that adding a similar alignment objective to reinforcement learning can improve performance at an earlier training budget.
Critical strengths
-
The dataset and benchmark are valuable contributions. They provide a useful resource for studying perception in team environments and evaluating what a single player’s observations reveal about teammates, opponents, and the broader game situation.
-
The proposed method is simple, intuitive, and effective. Using synchronized teammate observations as a training signal is a natural idea, and the approach offers a straightforward way to adapt existing visual models to capture more information about the team.
-
The central predictive findings are supported by convincing experimental evidence. The results show that the proposed training method makes information about other players easier to extract from a single player’s visual representation, and the controlled experiments strengthen the evidence for the value of teammate correspondence.
Critical weaknesses
-
The paper does not demonstrate improved gameplay or better coordination in Counter-Strike 2, its main experimental setting. Although the predictive results are encouraging, it remains unclear whether these improvements translate into better decisions and stronger team performance in the game. Since coordination is a central motivation for the paper, a more direct connection between improved perception and coordinated behavior would strengthen its practical contribution.
-
The applicability of the method beyond settings with distinct first-person observations remains unclear. For example, if all agents in an Overcooked setting can observe the full map, it is less clear what additional benefit would come from aligning their observations. In other applications, obtaining the required observations may be difficult. In autonomous driving, for instance, collecting other vehicles’ views and accurately synchronizing them could present substantial practical challenges. Although the additional views are required only during training, their availability is an important requirement beyond access to a single agent’s raw first-person observations. The paper would benefit from a clearer discussion of which coordination settings are suitable for the method and what data requirements may limit its applicability.
Questions for authors
-
How would you extend the proposed method and framework to other coordination settings, particularly those with shared observations or limited access to other agents’ views? How transferable do you expect the approach to be, and what are the main challenges you anticipate? How might these challenges be addressed?
-
Could the framework be extended to collaboration between LLM-based agents? For example, could a pretrained LLM be fine-tuned using synchronized observations or interactions from teammates to improve its ability to collaborate? What would constitute an appropriate synchronization signal and training objective in such a setting?
2 comments
Thank you for the clarifications. The discussion of suitable settings and training-data requirements addresses my questions about applicability. I appreciate the clearer distinction between the CS2 representation-learning findings and the preliminary control results in ViZDoom. Demonstrating improved coordination in CS2 remains an important next step. Nevertheless, the dataset, benchmark, and predictive findings already constitute a valuable contribution. Overall, the response supports my original positive assessment, and I will maintain my rating of clear acceptance.
We thank the reviewer for the positive assessment and the accurate summary of our contributions. We address the remaining concerns below, and the revisions are marked in teal in the revised PDF.
[W1] Coordinated gameplay in CS2
We agree that coordination motivates this work, and we see testing whether better perception yields better coordinated play in CS2 as a natural next step. CS2 replays are recordings and cannot execute an agent's actions. In this paper, we approach the problem as representation learning for video understanding. We therefore measure what a decentralized actor can condition on, the team state its encoder makes linearly readable (Sec. 4.2). We believe this understanding itself makes a better teammate. Teamwork research, as cited in Sec. 1, links situation awareness, knowing where teammates and opponents are and what they will do, to team performance. In basketball, for example, a point guard's court vision is valued as a skill in its own right. Representation learning similarly evaluates an encoder by linear-probe accuracy, as for CLIP and DINOv2, and our probes measure team understanding in the same way.
Our ViZDoom experiment (Sec. 5.7) tests the step to control, where adding the cross-ego term to MAPPO improves sample efficiency. We acknowledge that ViZDoom is a simpler game than CS2, and the Limitations in Sec. 7 focus the CS2 results on the linear-probe decodability of the learned representations.
[W2 + Q1] Applicability and data requirements
Thank you for this question. We added an Applicability paragraph to Sec. 7. CECL suits teams whose members observe different parts of a shared situation. It needs synchronized recordings of several members during training, and deployment uses one view.
In fully observable settings such as Overcooked, one view already carries the team state, and we expect little gain. Our data agree with this interpretation, as teammates closer than 600 units are already similar before alignment, and the gain grows with distance (Fig. 6a). For limited access to other views, CECL needs recordings of only two members at a time, because any two teammates form a positive pair. It also tolerates imprecise synchronization. Each positive is a five-second window, and after alignment teammate similarity decays gradually with time offset (Fig. 6b), which suggests that clock offsets of a few seconds still yield valid pairs. The main challenge we anticipate is heterogeneity, because driving and rescue teams mix roles and sensors such as cameras and lidar. Each role could have its own encoder, with one shared projection head that maps all of them into the space where CECL compares them.
[Q2] LLM-based agents
Yes, we expect the method to transfer, most directly to multimodal LLM agents or VLM agents that observe and act from images. Many such agents build on CLIP- or SigLIP-family vision encoders, the same families CECL already adapts (Sec. 4.2), and Sec. 5.6 shows that the adapted SigLIP2 features move toward teammate concepts in the shared image-text space. The simplest extension would give the agent a CECL-adapted vision encoder. A fuller one would fine-tune the agent with CECL as an auxiliary loss. The synchronization signal would be the shared timestamp or environment step at which teammates see different parts of one scene, and the same sigmoid loss would pull together the agents' representations of their views at that step and push apart those from other steps. Evaluation would follow our paper, first probing whether one agent's representation predicts what its partner sees, then measuring cooperative task success. We updated Sec. 7 to mention this extension as future work.
Conclusion
We thank the reviewer again for the constructive review, the accept recommendation, and the guidance on improving the paper. All four points are reflected in the revision, and we would be grateful if the reviewer kept their score and supported the paper in the discussion.