acceptodds
Under review as a conference paper at ICLR 2027

The Allocentric Shift: Multi-Ego Contrastive Perceptual Alignment from Coordinated Behavior

Abstract

Efficient Human-Agent Teaming (HAT) benefits from implicit coordination. Most teaming research relies on explicit communication, but in fast, reactive environments communication is often constrained, so each member must infer where teammates are and what they are doing from its own first-person view. Taking a video-understanding perspective, we ask whether this allocentric awareness can be learned from the observations coordinated teams naturally produce. We introduce X-Ego-CS, 273 hours of synchronized first-person gameplay from 101 high-skill Counter-Strike 2 matches, with a 33-task benchmark measuring what one player's view reveals about the player, teammates, opponents, and game. Aligning teammates' synchronized representations with Cross-Ego Contrastive Learning (CECL) produces a consistent Allocentric Shift, a trade-off in which other players' locations become more predictable across encoders and horizons at a modest cost to self-centric prediction, especially of the observer's own motion. Controlled experiments on held-out matches show that the gain grows with how much team state paired views share, not with alignment itself, and that aligned teammates become similar even when far apart. Without text supervision, a vision-language encoder's concept rankings also move away from self-centric phrases, and in a two-agent ViZDoom task the objective improves the sample efficiency of a multi-agent reinforcement learning baseline. These results suggest that the natural correspondence in recorded teamwork could offer a scalable path toward agents that sense their team's situation and coordinate without explicit communication.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
contrastive learning, multi-agent perception, egocentric video, human-agent teaming, multi-agent reinforcement learning, vision language model, video understanding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_t9guOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether synchronized first-person recordings of teammates can help a visual model infer the wider team situation from a single player’s view. It introduces X-Ego-CS, a dataset of synchronized Counter-Strike 2 gameplay, and Cross-Ego Contrastive Learning (CECL), which encourages similar representations for teammates’ observations at the same time. Using a benchmark of 33 prediction tasks, the authors find that CECL improves predictions of teammates’ and opponents’ current and future locations, while reducing performance on some self-related tasks, particularly the observer’s speed. They call this pattern the “allocentric shift.” The paper evaluates the result across several visual encoders and conducts controlled experiments to examine the contribution of teammate correspondence beyond generic adaptation and shared contextual cues. A separate two-agent ViZDoom experiment also provides preliminary evidence that adding a similar alignment objective to reinforcement learning can improve performance at an earlier training budget.

Critical strengths
  1. The dataset and benchmark are valuable contributions. They provide a useful resource for studying perception in team environments and evaluating what a single player’s observations reveal about teammates, opponents, and the broader game situation.

  2. The proposed method is simple, intuitive, and effective. Using synchronized teammate observations as a training signal is a natural idea, and the approach offers a straightforward way to adapt existing visual models to capture more information about the team.

  3. The central predictive findings are supported by convincing experimental evidence. The results show that the proposed training method makes information about other players easier to extract from a single player’s visual representation, and the controlled experiments strengthen the evidence for the value of teammate correspondence.

Critical weaknesses
  1. The paper does not demonstrate improved gameplay or better coordination in Counter-Strike 2, its main experimental setting. Although the predictive results are encouraging, it remains unclear whether these improvements translate into better decisions and stronger team performance in the game. Since coordination is a central motivation for the paper, a more direct connection between improved perception and coordinated behavior would strengthen its practical contribution.

  2. The applicability of the method beyond settings with distinct first-person observations remains unclear. For example, if all agents in an Overcooked setting can observe the full map, it is less clear what additional benefit would come from aligning their observations. In other applications, obtaining the required observations may be difficult. In autonomous driving, for instance, collecting other vehicles’ views and accurately synchronizing them could present substantial practical challenges. Although the additional views are required only during training, their availability is an important requirement beyond access to a single agent’s raw first-person observations. The paper would benefit from a clearer discussion of which coordination settings are suitable for the method and what data requirements may limit its applicability.

Questions for authors
  1. How would you extend the proposed method and framework to other coordination settings, particularly those with shared observations or limited access to other agents’ views? How transferable do you expect the approach to be, and what are the main challenges you anticipate? How might these challenges be addressed?

  2. Could the framework be extended to collaboration between LLM-based agents? For example, could a pretrained LLM be fine-tuned using synchronized observations or interactions from teammates to improve its ability to collaborate? What would constitute an appropriate synchronization signal and training objective in such a setting?

2 comments

We thank the reviewer for the positive assessment and the accurate summary of our contributions. We address the remaining concerns below, and the revisions are marked in teal in the revised PDF.

[W1] Coordinated gameplay in CS2

We agree that coordination motivates this work, and we see testing whether better perception yields better coordinated play in CS2 as a natural next step. CS2 replays are recordings and cannot execute an agent's actions. In this paper, we approach the problem as representation learning for video understanding. We therefore measure what a decentralized actor can condition on, the team state its encoder makes linearly readable (Sec. 4.2). We believe this understanding itself makes a better teammate. Teamwork research, as cited in Sec. 1, links situation awareness, knowing where teammates and opponents are and what they will do, to team performance. In basketball, for example, a point guard's court vision is valued as a skill in its own right. Representation learning similarly evaluates an encoder by linear-probe accuracy, as for CLIP and DINOv2, and our probes measure team understanding in the same way.

Our ViZDoom experiment (Sec. 5.7) tests the step to control, where adding the cross-ego term to MAPPO improves sample efficiency. We acknowledge that ViZDoom is a simpler game than CS2, and the Limitations in Sec. 7 focus the CS2 results on the linear-probe decodability of the learned representations.

[W2 + Q1] Applicability and data requirements

Thank you for this question. We added an Applicability paragraph to Sec. 7. CECL suits teams whose members observe different parts of a shared situation. It needs synchronized recordings of several members during training, and deployment uses one view.

In fully observable settings such as Overcooked, one view already carries the team state, and we expect little gain. Our data agree with this interpretation, as teammates closer than 600 units are already similar before alignment, and the gain grows with distance (Fig. 6a). For limited access to other views, CECL needs recordings of only two members at a time, because any two teammates form a positive pair. It also tolerates imprecise synchronization. Each positive is a five-second window, and after alignment teammate similarity decays gradually with time offset (Fig. 6b), which suggests that clock offsets of a few seconds still yield valid pairs. The main challenge we anticipate is heterogeneity, because driving and rescue teams mix roles and sensors such as cameras and lidar. Each role could have its own encoder, with one shared projection head that maps all of them into the space where CECL compares them.

[Q2] LLM-based agents

Yes, we expect the method to transfer, most directly to multimodal LLM agents or VLM agents that observe and act from images. Many such agents build on CLIP- or SigLIP-family vision encoders, the same families CECL already adapts (Sec. 4.2), and Sec. 5.6 shows that the adapted SigLIP2 features move toward teammate concepts in the shared image-text space. The simplest extension would give the agent a CECL-adapted vision encoder. A fuller one would fine-tune the agent with CECL as an auxiliary loss. The synchronization signal would be the shared timestamp or environment step at which teammates see different parts of one scene, and the same sigmoid loss would pull together the agents' representations of their views at that step and push apart those from other steps. Evaluation would follow our paper, first probing whether one agent's representation predicts what its partner sees, then measuring cooperative task success. We updated Sec. 7 to mention this extension as future work.

Conclusion

We thank the reviewer again for the constructive review, the accept recommendation, and the guidance on improving the paper. All four points are reflected in the revision, and we would be grateful if the reviewer kept their score and supported the paper in the discussion.

Thank you for the clarifications. The discussion of suitable settings and training-data requirements addresses my questions about applicability. I appreciate the clearer distinction between the CS2 representation-learning findings and the preliminary control results in ViZDoom. Demonstrating improved coordination in CS2 remains an important next step. Nevertheless, the dataset, benchmark, and predictive findings already constitute a valuable contribution. Overall, the response supports my original positive assessment, and I will maintain my rating of clear acceptance.

Reviewer_tK2cOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper is motivated by the idea that agents operating in a team context can benefit from a shared perception across teammates, even when they have completely non-overlapping contexts. Humans in teams might have a sense of what their teammates are experiencing, so it makes sense that artificial agents could benefit from this as well. The paper proposes a contrastive method to align the perceptive representations of agents within a team, with training labels indicating whether or not sensory data belongs to a true member of the team. Experiments using a new Counter-Strike 2 dataset show that this training objective does lead to improved recognition of teammates and enemies. Extensive ablations and analysis confirm that this learning is happening. The paper suggests how this could improve downstream performance.

Critical strengths
  1. The motivation of wanting agents to have deep alignment of perception across the members of a team is thought-provoking.
  2. The contrastive objective used to train this notion of "teamness" is interesting. It is an elegant way to take advantage of a dataset of recordings of human teams to encode for a model what its teammates might be experiencing.
Critical weaknesses
  1. The paper shows that the model is able to learn to some extent which sensory instances are more likely to be those of team members, and there is extensive analysis showing that this is indeed learned, but the paper lacks grounding of why or whether this would actually translate to being a better teammate. It is not surprising that a model can learn to fit this dataset of where teammates are likely to be and what they are likely to be doing, but it is difficult to see the value. Sections 5.6 and 5.7 hint at downstream impact, but they are not well fleshed-out, clearly explained, or motivated.
  2. Experimental design overall is lacking clarity and precision, both in description and motivation.
Questions for authors

Questions:

  1. Do you have an concrete insights of how the CECL metric alters the behavior of the team? I see in Section 5.7 that it improves a score in VizDoom, but without any more context, it is hard to see if this is meaningful or noise. E.g., do agents behave differently in some meaningful way or does the objective simply serve as "some" regularizer that smooths learning?
  2. What do you think of refocusing the paper so that it focuses more on how downstream behavior of models change under CECL? Again, it is not surprising that the system can learn to generalize within the new dataset the paper introduces, but in all the ablations and analysis, the paper does not present clear or useful insights into what is happening.

Minor Clarifications:

  1. What is Mirage? It is not defined.
  2. What do you mean "Following Zhai et al. (2023)"? Did they introduce CECL first or something related?
  3. What is MAPPO?
  4. Suggestion to improve awkward text like "LLM outputs are not experimental measurements"
2 comments

We thank the reviewer for the thoughtful review. We address each concern below and mark the revisions in teal in the revised PDF.

[W1 + Q2] Novelty of the findings

We appreciate the reviewer's feedback on the value of the findings, but we have to respectfully disagree that the results are expected. The paper approaches the problem as representation learning for video understanding (abstract and Sec. 1) and asks whether perception can learn the team situation from recordings of coordinated play. CECL pairs teammates' views, which often share no visible content, and uses opponents only as negatives. One would therefore expect alignment to learn only coarse cues that all teammates share, such as side and round phase, and nothing about opponents. One might also expect alignment to require shared visible content, as in cross-view alignment for tasks such as image recognition, which pairs views of one object or scene from different cameras (Fig. 2a). Our experiments show otherwise.

  1. Opponents become more readable, with Enemy location macro-F1 rising from 5.5 to 15.5, as much as teammate macro-F1 (3.9 to 14.3, Table 2), and location AUROC rising for every encoder and forecast horizon (Secs. 5.1-5.2).
  2. Distant teammates become similar at the same moment, with AUROC against side- and phase-matched clips of other rounds rising from 0.41 to 0.70 for teammates more than 1,500 units apart (Sec. 5.5).
  3. Concepts shift without text supervision, as a vision-language encoder ranks Self phrases lower and Teammate phrases higher (Sec. 5.6).

[W1 + Q1 + Q2] Downstream behavior

We appreciate the reviewer's suggestion to refocus the paper on downstream behavior and see it as a natural next step. It would, however, answer a different question. A team's return mixes what agents perceive with how their policies learn. The focus of this paper is perception, what a single first-person view can be trained to encode about the team, and the exact game state of CS2 lets us isolate it.

We believe this understanding itself makes a better teammate. Teamwork research, cited in Sec. 1, links situation awareness, knowing where teammates and opponents are and what they will do, to team performance. In basketball, for example, a point guard's court vision is valued as a skill in its own right. Representation learning similarly evaluates an encoder by linear-probe accuracy, as for CLIP and DINOv2, and our probes measure team understanding in the same way.

Our ViZDoom experiment (Sec. 5.7) tests the step to control. CECL-MAPPO reaches a normalized return of 0.734 at 5M frames against 0.565 for MAPPO, with non-overlapping 95% intervals over three seeds (Table 6). Even at 10M frames, MAPPO stays below this level (0.685, Appendix K), and the 5M return of CECL-MAPPO is the highest of either method at either checkpoint. This indicates that the cross-ego term improves sample efficiency. We believe the cross-ego term does not act as a generic regularizer, because its only positive is the partner's same-timestep embedding, the information needed to time a joint pickup.

[W2] Clarity of the experimental design

We clarify that Sec. 5 opens by stating that each experiment answers one question, and groups the experiments into three, what the shift is (Secs. 5.1-5.2), what causes it (Secs. 5.3-5.5) and how far it reaches (Secs. 5.6-5.7). Each subsection then opens with its question. Sec. 5.3 rules out match overlap and under-trained probes, Sec. 5.4 removes cues that could reveal targets directly, and Sec. 5.5 contrasts the predictions of shared visible content and shared team situation.

We also improved the overall readability of the revised PDF.

[Q3-Q6] Minor clarifications

  • Mirage is one of the maps in the game CS2.
  • Zhai et al. (2023) introduced the pairwise sigmoid loss of SigLIP for image-text contrastive learning. CECL adopts this loss and contributes the pairing by team membership and time.
  • MAPPO is Multi-Agent Proximal Policy Optimization (Yu et al., 2022), a common baseline algorithm for multi-agent reinforcement learning.
  • We have clarified the above points in the revision.

[Ethics] Why Counter-Strike 2

We chose CS2 because, like StarCraft II, Dota 2 and CS:GO in the earlier work cited in Sec. 1, it is a fictional game that offers a controlled and reproducible research environment. The game engine re-renders all ten players' first-person views from public replays, the game logs every position and event as exact ground truth at no annotation cost. The capacity we study matters for human-agent teaming in emergency response, search and rescue, sports and games. We summarized this rationale in the Ethics statement, which also addresses privacy and the risk of extending cross-view inference to real people.

Conclusion

We hope these clarifications make the contribution clearer, and we would be grateful if the reviewer would reconsider the score and the ethics flag.

I thank the authors for the clarifications. I can now see how the experiments and results clearly establish a meaningful, interpretable, and compelling difference from the baseline, and that this difference is more surprising than I initially understood. As such, I have increased my score.

Reviewer_PLAHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a framework to align teammates’ first-person views recorded at the same moment, so that members can infer where teammates are and what they’re doing from their own view, without explicit communication. They use hours of synchronized first-person gameplay from Counter-Strike matches and align teammates’ representations with cross-ego contrastive learning, even when teammates are far apart and share no visible objects. They find that other players’ locations become easier to predict, while predicting the observer’s own motion gets impaired. They also test shortcuts, held-out matches, a language probe, and a two-agent ViZDoom task.

Critical strengths
  1. The authors test directly whether the gain comes from shortcuts, such as team side, game phase, bomb plants, or deaths, and they replicate the results on held-out matches with fully trained probes.
  2. The authors design controls that separate what drives the gain. They compare real teammate pairs against pairs of augmented views of the same clip and against pseudo-teams matched only on side and phase.
  3. The authors measure gains and costs together, so the paper shows what the representation loses about the observer, not only what it gains about others.
Critical weaknesses
  1. In Section 6, the authors show that pseudo-teams sharing only side and phase already recover about 70% of the gain, so true synchronization adds only a small part on top. Still, the conclusion says the controls tie the effect to synchronized correspondence, which I think it's too strong for such a small difference.
  2. The gains are small in absolute terms. Location prediction for other players stays low even after alignment, and at longer horizons both the baseline and the proposed method are close to zero. In Table 2, teammate R2 actually drops, and the authors don’t discuss it.
  3. The authors report variance only for AUROC. For the F1 and R2 results, they use only two probe seeds and no confidence intervals, so it might be that some of these differences are still within bounds.
  4. In the abstract, the authors describe the loss in predicting the observer’s own motion as modest, but for some encoders it’s quite large. In Table 3, for example, V-JEPA2 loses a lot of its ability to predict the observer’s own speed after alignment. So the model gains information about teammates but forgets a meaningful amount about the observer itself. In Section 6, the authors also show that with large batches the method gives no gain at all, so the result depends heavily on how the model is optimized.
Questions for authors
  1. In Section 6, you show that pseudo-teams sharing only side and phase recover most of the gain. How confident are you that the remaining difference comes from synchronization, given only nine held-out matches and three seeds?
  2. How did you choose the optimization regime, given that large batches give no gain?
  3. Why do enemy locations improve in the probes while enemy phrases fall in the language probe?
  4. In Section 5.7, you show a gain over MAPPO at 5 million training frames. Does plain MAPPO catch up with more training, for example at 10 million frames, and how does your objective compare against other auxiliary objectives, not only plain MAPPO?
1 comment

We thank the reviewer for the careful review. We address each concern below, and mark the revisions in teal in the revised PDF.

[W1 + Q1] Strength of the synchronization claim

Thank you for catching this. We agree that the prior writing in the conclusion might read too strong, and we revised the sentence (Sec. 7).

To address the confidence concern, we ran two additional experiments on the held-out matches, reported in Sec. 6 and the new Appendix L.

  1. We retrained the real-team, pseudo-team and single-view arms with five encoder seeds each. Real teams exceed pseudo-teams by 0.011 [0.008, 0.013] (95% paired bootstrap over seeds and test matches), in every seed and in all nine test matches. We fixed this test and its success criterion before training.
  2. To test whether the increment comes from synchronization, we paired teammates from the same round observed 10-20 seconds apart, which keeps the round, the team and the players and changes only the timing. This setting falls below pseudo-teams (0.709 versus 0.723), whereas synchronized groups of the same sizes reach 0.734. This indicates that the increment depends on whether the teammates' views share the same timing.

[W2 + W3] Size and variance of the F1 and R² results

We agree that the gain from CECL is harder to see in the absolute scores. We wish to clarify that this comes from the difficulty of the task. For location tasks, the labels are sparse (players occupy about 3 of 23 map regions), and the low performance of the baseline confirms the difficulty. This is exactly why we also report ranking metrics to better demonstrate the gain (Secs. 3 and 5.2).

Similarly, at 20 seconds ahead, the F1 metric alone does not reveal the gain, because it relies on a fixed threshold. For the same reason, we also report AUROC, which shows the improvement for every encoder and target (Fig. 4).

Regarding teammate R², this metric measures teammate speed. Its decline is consistent with our finding that CECL does not improve every aspect of the representation, as it trades away fine motion detail, measured by speed (Sec. 5.1).

Regarding variance, the F1 and R² variance supports the same conclusion. We have updated Secs. 5.1-5.2 in the revision to include it. Every F1 gain (0.010 to 0.252) exceeds the probe-repeat SD of both arms (at most 0.025), 122 of 123 same-seed differences are positive (Appendix F.1), and for speed R², both CECL seeds fall below both baseline seeds on every map.

Regarding seed numbers, we also clarify that location F1 uses at least three seeds per comparison (246 runs, Appendix F.1), and the two probe seeds in Table 2 apply to its per-map columns as cross-map evidence.

[W4 + Q2] Self-motion cost and optimization regime

We agree that the word "modest" understates the motion cost, as the reviewer notes for V-JEPA2 (Table 3), and we removed it from the abstract. We also wish to further clarify that the cost is largest for motion, and this trade-off is one of our main findings (Sec. 1).

On the optimization regime, the training configuration (batch size, number of updates and gradient clipping) is the one used to train the original benchmark encoders, and it was fixed before the held-out study began. The held-out study reused this configuration unchanged for all three arms (Tables 14-15), and no setting was tuned on the held-out matches. We further clarify this in Sec. 6.

[Q3] Enemy locations versus enemy phrases

The enemy location tasks and the language probe measure two different things. The location probes predict where opponents are on the map, whereas the language probe ranks fixed text phrases by their similarity to the adapted visual features. CECL never trains the text encoder, yet the visual shift alone moves Self phrases down in all 129 template-map pairs and Teammate phrases up in 121 (Table 5). Many Enemy phrases describe opponents in the observer's own view, such as "enemy in frame" (Appendix J.2). This first-person content is consistent with the Enemy group declining along with the Self group, by much less (5.0 ranks, against 20.7 for Self). Sec. 5.6 now explains this.

[Q4] MAPPO at 10M frames and other objectives

Yes. In ViZDoom, as stated in Table 6 and Appendix K, both methods reach the same level by 10M frames. The cross-ego term improves sample efficiency, because CECL-MAPPO at 5M frames (0.734) already exceeds MAPPO at 10M (0.685). We did not test a second objective in ViZDoom.

Conclusion

We hope our clarification, the new experiments and the revision of the paper address the reviewer's concerns, and we would be grateful if the reviewer would consider raising their score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion

Dear AC and reviewers,

As the discussion period closes, we briefly summarize:

  • Reviewer t9gu kept the score of 4 (clear acceptance) and found that the new Applicability paragraph (Sec. 7) is helpful for clarifying the questions on suitable settings and data requirements.
  • Reviewer tK2c raised the score to 3 (weak acceptance) and found that the experiments are "meaningful, interpretable, and compelling difference from the baseline".
  • For Reviewer PLAH, we added two experiments on the held-out matches (Sec. 6 and Appendix L). Over five seeds, synchronized teammates exceed side- and phase-matched pseudo-teams in every seed and every test match, and teammates paired 10-20 seconds apart fall below pseudo-teams. The revised pdf also reports probe variance for F1 and R^2 and quantifies the self-motion cost (Secs. 5.1-5.2).

We thank the AC and all reviewers for their time and the constructive discussion!

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.