acceptodds
Under review as a conference paper at ICLR 2027

COMRAD: A Benchmark for Embodied Multi-Agent Reinforcement Learning

Abstract

Benchmarks are central to the development and evaluation of multi-agent reinforcement learning (MARL) algorithms. As the cooperative MARL community has grown, two categories of evaluation environments have proven indispensable: low-dimensional feature-vector benchmarks that isolate algorithmic behavior in compact state spaces, and two-dimensional pixel-based benchmarks that rely on overhead visual observations. Embodied cooperation, however, requires high-dimensional 3D egocentric perception, partial observability arising from line-of-sight occlusion, and the ability to close the gap between recognizing a coordination pattern and executing it from a first-person view. While various 3D environments have been explored for vision-based RL, no existing platform simultaneously provides a standardized, purely cooperative multi-agent benchmark with 3D first-person observations and high-throughput simulation. To address this gap, we introduce , a operative ulti-Agent einforcement Lerning benchmark suite in oom, featuring a diverse set of challenging scenarios spanning role asymmetry, temporal synchronization, and spatial navigation. To introduce within-scenario variability, we develop , a procedural map generator that produces diverse layout configurations for every scenario. We integrate COMRAD with Sample Factory, a high-throughput asynchronous RL framework, and implement seven MARL baselines on top of it, reaching frames per second during training. Our experiments show that COMRAD poses significant challenges for current CTDE methods, and a capability-linked analysis of where each algorithm family fails, establishing visual cooperative MARL as an important open frontier.

Primary area:
datasets and benchmarks
Keywords:
Reinforcement Learning, MARL, Benchmark, Egocentric, 3D, Doom, Vision, Perception, Cooperation, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_C2K6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+2)
Read review
Paper summary

This paper introduces COMRAD, a ViZDoom based benchmark for embodied cooperative multi-agent RL where each agent only sees through its own first-person view, so teammates can be out of sight behind walls. COMRAD provides 13 cooperative scenarios in which coordination must emerge solely from local, visual and egocentric observations. They also introduce DoomGen, a procedural generated library that generates various seedable maps in batches with separate difficulty levers for task mechanics and space. The training stack is built on Sample Factory and evaluated on 8 baseline configurations across 7 algorithm families under a shared 100M-step budget training on shaped rewards but evaluating on a separate task score. A clear pattern emerges is that no single algorithmic family dominates across all scenarios. The final part digs into the two very Difficult scenarios where every baseline scores near zero - using scripted partners, oracle variants and human pairs, they argue that the bottleneck is the agents co-adapting between roles rather than perception or reward design.

Critical strengths
  1. It fills a real gap as a cooperative MARL testbed with first-person vision, procedural layouts (DoomGen) and ready baselines to immediately start training on especially integrating with sample factory
  2. The Very Difficult tier is carefully checked - scripted partners, oracle variants and human pairs make a good case that it’s hard because of coordination, not because the tasks are broken
Critical weaknesses
  1. The results don’t line up - IPPO’s numbers in Table 3 are ~1.5-5x lower than where its own learning curves in Fig. 28 end up on three scenarios, and Figs. 4 and 29 don’t quite match the table they’re built from. The “why methods win” story is also shaky, where each algorithm family gets different reward handling and architectures, so the gaps can’t be pinned on the algorithms themselves, and QMIX’s wins over MAPPO are within noise.
  2. The benchmark doesn’t really show it measures what it says - there’s no random baseline to compare against, and methods barely get anywhere on Foraging Commons and Lava Pit (~12% and ~9% of the ceiling at best). The claims that the two difficulty knobs work independently and that scores don’t drop on new maps aren’t backed by the evidence they show, and they miss earlier work like Megaverse, so the “no existing platform” claim in the abstract needs a bit of toning down.
Questions for authors
  1. Why are IPPO’s Table 3 scores on Smart Enemies, Armory Siege and Co-op Puzzle ~1.5-5x lower than its Fig. 28 curves? Are the curves on training or held-out maps, and can you share per-seed scores?
  2. Tables 4-5 say on-policy methods use “scenario-dependent team blending” while value-based ones get per-agent rewards - what blending did each scenario use, and how much of the on- vs off-policy gaps comes from that?
  3. Could you add a random and a frozen-partner baseline for the main scenarios(App. H only scripts partners on the two hardest), plus training-map scores next to Table 3, so that the score floor and the no-degradation claim can be verified ?
  4. The paper says the 100M budget is in logical environment steps, but at its own App. E throughput for two agents (~1,676 env steps/s) that’s ~17 hours, or ~9 hours even at 25K FPS - how does Table 1 get ~1 hour?
5 comments

Thank you for checking the numbers this carefully. Several of your points are correct, and we fixed them in the revised PDF.

W1 and Q1: Mismatched figures

Figures 4, 28, and 29 were drawn from an earlier local copy of the training logs; these old training logs were inflated by a bug in counting agents' progress. This bug was discovered and fixed, but the figures were not updated. In addition, the submitted Figure 28 showed a single seed per algorithm on training layouts, while Table 3 is the 5-seed mean on held-out layouts.

So the gap you noticed came from outdated logs containing counting bugs, it did not come from a drop between training and held-out layouts.

In the revised PDF:

  • Figures 4, 28 and 29 are regenerated from the final logs behind Table 3.
  • Figure 28 shows the mean ±\pm 95% CI over all five seeds instead of a single seed, and the caption states that the curves are on training layouts. This replaces the single-seed curves, so the spread across seeds is now visible in the figure.
  • Training- vs held-out scores (under Q3) are added to the appendix, so the curves and the table can be checked against each other.

W1: QMIX vs MAPPO are ties. You are right that QMIX's wins over MAPPO on Foraging Commons and Armory Siege are within noise. In the revised PDF, we report them as ties (both bold in the table under Q3), and we updated the claims that depended on them.

W1: Can the gaps be pinned on the algorithms? We ported every algorithm into our asynchronous training stack ourselves. During this process and when designing the scenarios, we optimized the hyperparameters for each scenario.

Some of the remaining differences are part of the algorithms: the mixer, the centralized critic, HAPPO's sequential updates. Others are implementation choices inherited from the reference codebases and are not part of the algorithm. We kept these architecture details because forcing one architecture and update scheme on all methods would mean evaluating variants that differ from the published ones. Cooperative MARL benchmarks have usually compared algorithms the same way, as complete methods: SMAC [1], EPyMARL [2] and the MAPPO study [3] are examples. Reward handling follows the same principle (see Q2).

Therefore, our rankings compare complete methods within their families. A claim that a gap comes from one specific component, such as the centralized critic or the mixer, can only be tested directly when two methods share everything else.

References:

[1] Samvelyan et al. "The StarCraft Multi-Agent Challenge." AAMAS 2019 (arXiv:1902.04043).

[2] Papoudakis et al. "Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks." NeurIPS 2021 Datasets and Benchmarks Track (arXiv:2006.07869).

[3] Yu et al. "The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games." NeurIPS 2022 Datasets and Benchmarks Track (arXiv:2103.01955).

W2: Does the benchmark measure what it claims?

Random baseline. We added a uniform random policy to Table 3 (table under Q3). It confirms that the floor is close to 0 in most scenarios, but not everywhere: random reaches 13.2% on Ammo Carrier, 6.2% on Co-op Health Gathering and 4.1% on Co-op Puzzle.

Foraging Commons and Lava Pit. We agree that the best methods reach only about 12% and 9% of the ceiling, but the scores are still above the floor. The best methods score 11.9% and 9.1% against 2.2% and 0.4% for the random policy, and they fall back to the random level when the partner is replaced (1.7% and 0.0% with a no-op partner, frozen-partner table under Q3). When designing the scenarios, we also inspected the trained agents and confirmed that they learn the intended basic behaviour (restore spawn rate and hold the bridge at least once). However, they fail to stick to their assigned role (e.g. keep the bridge up) for the full episode, which is what the ceiling requires. The revised Section 5 has explicitly noted this.

Difficulty knobs. The two axes are independent in the sense that they are configured separately, which holds by construction. You are right that the 3×33\times3 grid in Appendix D.5 does not show more than that. The differences between its cells are within noise, and the scores don't represent difficulty well. We inspected the recordings and found that this happens because even though the corridors are made smaller and longer, it still did not take longer to traverse, and thus did not cause much more difficulty for the agents.

We replaced the grid with a new one on Co-op Puzzle, with a much wider range between the easiest and the hardest cell. One trained policy (MAPPO) is evaluated zero-shot in every cell, on 5 held-out layouts with 10 episodes each. Scores are in % of the ceiling, and ±\pm is the 95% CI over the 5 layouts.

  • Spatial axis: room size and plate size.
  • Mechanical axis: the frequency of switching between 2 room types.
Spatial \ MechanicalEasyMediumHard
Easy72.0 ±\pm 5.556.4 ±\pm 2.324.4 ±\pm 2.3
Medium33.2 ±\pm 3.830.4 ±\pm 6.118.4 ±\pm 2.3
Hard10.4 ±\pm 2.97.2 ±\pm 2.98.4 ±\pm 2.3

We increase the range between difficulties to show the effect of each axis. Otherwise, different difficulty levels will just score the same.

The score falls down every column (the spatial axis) and along the Easy row (the mechanical axis). In the Medium row only the Hard cell is clearly lower. In the Hard row all three cells are near the floor and within each other's intervals. On this scenario the spatial axis has the larger effect; however, the harder spatial levels are also further from the training geometry, so part of their effect measures generalisation to new room sizes. In the revised PDF, this grid replaces the Armory Siege grid in Appendix D.5, and we describe the axes as separately configurable knobs whose effects interact.

No-degradation claim. The submission did not support "Performance does not degrade noticeably between training and held-out evaluation" (Section 5) with numbers. The table under Q3 now provides the training and held-out scores next to each other, and they support the non-degradation claim between training and evaluation. Held-out layouts are unseen layouts generated with the same generator settings as training; the difficulty grid above changes those settings, so its drop is a separate effect.

Megaverse. Thank you for the pointer. We will cite Megaverse in Related Work and in Table 1. We see Megaverse as complementary to COMRAD, because the two differ in focus, in task design, in the mechanics available for building tasks, and in the baselines and evaluation tools that come with each benchmark.

  1. Megaverse is a general-purpose simulator, and COMRAD is a cooperation benchmark. Megaverse-8 covers embodied skills such as object permanence, manipulation, memory and exploration. COMRAD builds each scenario around one mechanism from a taxonomy of cooperation (Figure 5), and we measure how much the score depends on the partner (frozen-partner table under Q3).
  2. ViZDoom provides richer mechanics and assets. Megaverse worlds are voxel grids with "random monochrome materials". Agents can walk, look, jump and use one binary interact action, and objects can only be placed "at discretized locations in space". On the other hand, COMRAD's coordination problems are built on task elements that do not appear in Megaverse-8, for example: enemies whose behaviour depends the team, weapons, ammunition, health, doors, plates, switches, lighting, etc.
  3. The two benchmarks also come with different baselines and evaluation tools. The Megaverse paper reports results for one algorithm (APPO). COMRAD releases seven baseline algorithm families implemented on one training stack, scores every method on held-out layouts, and includes scripted-partner, oracle, and human references to compare against.

Q2: Reward handling

Each agent's training reward is blended as r~i=(1−α) ri+α rˉ\tilde r_i = (1-\alpha)\, r_i + \alpha\, \bar r, where rir_i is agent ii's own shaped reward and rˉ\bar r is the team reward (the sum of both agents' rewards). α=0\alpha = 0 gives the individual reward and α=1\alpha = 1 the team reward. This is the same form as Megaverse's Team Spirit and EPyMARL's common reward option. The table below lists which reward each method receives:

MethodStealth Labyrinth, Lava MazeOther 11 scenarios
IPPO, MAPPO, HAPPOteam reward (α=1\alpha = 1, sum of both agents)individual reward (α=0\alpha = 0)
IDQNindividual rewardindividual reward
VDN, QMIX, QPLEXteam reward (sum of both agents)team reward (sum of both agents)

VDN, QMIX and QPLEX learn from the team reward because value decomposition factorises a single team value. IPPO, MAPPO and HAPPO use α=1\alpha = 1 on Stealth Labyrinth and Lava Maze because the support role (scout, observer) cannot complete the objective itself, so individual reward would barely credit it. IDQN keeps the individual reward there. On Stealth Labyrinth this makes little difference, because the scenario's reward shaping splits every team bonus and penalty equally between the two agents, so IDQN's reward is close to the team reward. On Lava Maze the observer is credited for the navigator's progress, so it gets less credit than under the team reward. This does not change any result: on Lava Maze every method scores 0.0, including the seven trained on the team reward, and on Stealth Labyrinth only QMIX scores above zero. We also leave both scenarios out of the matched IPPO and IDQN comparison below.

The individual reward is only a shaped training signal. Every scenario has one team objective, every method is scored on the same team benchmark score, and there are no rival agents; this is what "purely cooperative" means in the paper. In the two social dilemmas, Stag Hunt Arena and Foraging Commons, individual reward is also what creates the dilemma, because each agent is tempted to take the safe or selfish payoff. A shared reward would remove that temptation.

Note that the reward difference does not favour the on-policy family on Stag Hunt Arena. VDN and QMIX receive the team reward, which is the easier setting, yet they score 50.3% and 0.7%, compared with 70.8% for MAPPO trained on individual reward.

How much of the on- vs off-policy gap comes from reward? We cannot separate the two cleanly. We have one matched comparison: IPPO and IDQN receive exactly the same individual reward on 11 scenarios, and IPPO scores higher on 10 of them. So the on-policy lead does not come from the reward. For VDN, QMIX and QPLEX, the comparison with on-policy methods differs in the construction of algorithm, so part of that gap may come from the reward (they handle the reward differently). We believe this is a much broader and interesting question, and should be investigated in future work.

Q3: Random baseline, training scores, frozen partner

Table 3 normalised by the ceiling, with a random policy and training-layout scores. Scores are s~=clip(s/smax⁡,0,1)\tilde{s} = \mathrm{clip}(s / s_{\max}, 0, 1) in % of the ceiling. Each cell reports held-out / training, with training being the mean of the training-layout learning curve (Figure 28) averaged over the last 5M frames, over 5 seeds. Bold marks the best held-out method and any method whose 95% CI overlaps it.

Reviewer_YcPKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces COMRAD to address the limited evaluation of first-person visual cooperation in existing MARL benchmarks. It combines diverse cooperative tasks with high-throughput training. Experiments show that current algorithms still struggle with complex coordination under visual partial observability, highlighting open challenges in this setting.

Critical strengths
  1. The paper focuses on multi-agent collaboration under the first-person visual conditions, supplementing the insufficient evaluation of such issues by existing benchmarks and having research value.
  2. The benchmark design is relatively complete, integrating diverse collaborative tasks, efficient training, and comparisons of various algorithms, providing a useful experimental platform for subsequent research.
Critical weaknesses

The results on difficult tasks are still not easy to explain. The author thinks the main issue is collaboration, but current experiments can't clearly tell whether it's a lack of collaboration skills or just difficulty in learning individual roles. This also makes it hard for us to judge whether improvements in these tasks from later methods really come from better collaboration.

Questions for authors

After fixing the other partner, can the scout and observer learn the task? Are there any corresponding experimental results?

3 comments

Thank you for this question. Appendix H.1 had half of the experiment you ask for: it fixed the scout and the observer and trained the gunner and the navigator. We now ran the other half, fixing the gunner and the navigator and training the scout and the observer. We did not do this in Appendix H.1 because fixing the other partner was already enough to demonstrate learnability, which was the goal of this appendix.

How each partner is fixed.

  • Scripted scout (App. H.1, Stealth Labyrinth): walks from the start room to the currently active room and holds at the room threshold until the room is cleared.
  • Scripted gunner (new, Stealth Labyrinth): follows the scout and stops 96 units behind it. Whenever a turret is alive and in the gunner's line of sight, the gunner turns to it and fires.
  • Scripted observer (App. H.1, Lava Maze): computes the shortest next step toward the goal from its own telemetry and signals it through the ordinary flash, using a fixed colour convention.
  • Scripted navigator (new, Lava Maze): reacts only to the observer's flash colour and its own tile, moving exactly one tile in the direction given by the same fixed colour convention. The learned observer must therefore discover from reward alone which colour to emit at each step.

Results (held-out layouts, mean ±\pm 95% CI over 5 seeds):

ScenarioFixed partner -> learned roleIPPOIDQNFull task (best baseline)
Stealth LabyrinthScripted scout -> learned gunner0.19 ±\pm 0.060.32 ±\pm 0.080.21 ±\pm 0.01 (QMIX)
Scripted gunner -> learned scout0.17 ±\pm 0.270.01 ±\pm 0.03
Both scripted (scripted reference)0.60 ±\pm 0.04
Lava MazeScripted observer -> learned navigator0.97 ±\pm 0.120.06 ±\pm 0.060.00 ±\pm 0.01 (all)
Scripted navigator -> learned observer0.00 ±\pm 0.000.03 ±\pm 0.02
Both scripted (scripted reference)28.97 ±\pm 0.13

The "Scripted scout -> learned gunner" row differs slightly from the submitted Appendix H.1 (IPPO 0.16 to 0.19, IDQN 0.28 to 0.32) because we corrected the held-out layouts used for this one experiment; no other result is affected.

To answer your question directly:

  • Lava Maze: With a scripted navigator, the learned observer does not learn which colour to send (0.00 and 0.03). With a scripted observer, the learned navigator completes 0.97 mazes per episode under IPPO, so it learns to read the fixed signal only partly. Both roles are therefore hard on their own, and the sender is the harder one.
  • Stealth Labyrinth: With a scripted scout, the learned gunner reaches 0.19–0.32, about a third to a half of the scripted pair (0.60). With a scripted gunner, the learned scout succeeds in only 2 of 5 IPPO seeds (scores per seed: 0.00, 0.00, 0.42, 0.03, 0.39) and fails under IDQN (0.01). Thus, the scout's leading behaviour can be learned, but unreliably.

Therefore, the difficulty of the Very Difficult tier comes from two parts:

  1. Individual role learning: The leader/sender role (scout, observer) is hard to learn even next to a scripted partner. The follower/receiver role (gunner, navigator) can be learned against a scripted partner, but reaches only a fraction of the scripted reference.
  2. Co-adaptation: On top of that, both roles must co-adapt, meaning two learners must converge on a mutually consistent behaviour while the other is still changing.

In the revised PDF, we changed the paper accordingly. Section 5, Appendix H.1, and the conclusion of Appendix H no longer say that the failure is one of co-adaptation and not of learning either role. They now report both parts above, with the new rows added to the Appendix H.1 table.

Measuring collaboration vs. role learning. You also noted that it is hard to judge whether a later method improves collaboration or only role learning. We released the scripted partners with the benchmark for the two Very Difficult scenarios. A method reports three scores per scenario:

  1. the full-task score, where both roles are learned;
  2. the score of each learned role next to its scripted partner (two numbers);
  3. the scripted-pair reference.

If a method raises (2), it learns the individual roles better. If it raises (1) while (2) stays the same, the gain comes from co-adaptation.

Dear Reviewer YcPK,

Due to temporary compute constraints, we needed extra time to run the additional experiments required to thoroughly address your feedback, and fortunately managed to finish and polish everything before the deadline (this is why we did not submit the rebuttal earlier). However with the discussion period drawing to a close, could you spend some of your time to review our rebuttal and updated results?

Thank you for your time and understanding, and please let us know if any further clarification is needed.

Thank you for the detailed response and the additional experiments. The results with fixed partners help clarify the difficulty of learning each role. I will maintain my original rating of 3 (weak acceptance).

Reviewer_kd6JOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes COMRAD, a procedurally generated multi-agent DOOM environment for cooperative tasks. This paper proposes the environment, benchmarks CTDE algorithms on it, and lightly attributes the failures and successes of the algorithms to particular features of the algorithms. It also adds some curriculum learning tooling that is described in the appendix. Their appendix provides a much more thorough overview of all of these features.

Critical strengths

This paper, as far as I understand the landscape of benchmarks, does fill a gap in the literature. The benchmark seems reasonably well-designed, moving many classic cooperative tasks into a 3d setting, and I appreciate that there are all sorts of procedural controls in it that will enable interesting future studies.

Critical weaknesses

It is not obvious to me that high visual complexity is a feature that is relevant or important to a multi-agent benchmark as it is not clear what distinct multi-agent challenges this provides. I do agree that the partial observability and the ability to obscure another agents view does constitute an interesting and distinct challenge. However, I do not believe the paper really demonstrates that this is a feature that the benchmark stress-tests. How often is a teammate in view? How often is a view obscured? Does this task really require the agents to reason about partial observability? I stress this because this issue has occurred in multiple benchmarks previously where it later turned out that the partial observability did not matter much to the results.

The benchmark comes with a fair number of baselines, even if they are likely slightly under-tuned as the authors do not do per-algorithm tuning for most of them. This makes it hard to justify many of the conclusions about relative algorithmic performance.

Some non-critical weaknesses:

  • Table 3 would be easier to read if it was normalized by the maximum score.
  • Note that while I marked informativeness as a 1, this is because it's a benchmark paper and it's not unreasonable for such a paper not to teach me something new since that is not exactly the point of it.
  • In the appendix, in line 1608, it is written "The paper’s main results are then reported at the 100M-step checkpoint rather than only at the terminal step." Or, as another example "We recommend that future COMRAD scaling studies report both common-step and fixed-wall-clock comparisons, as increasing N simultaneously raises the coordination difficulty and reduces the volume of experience a system can collect within a realistic compute budget." This doesn't make sense; you're the ones running the study! Why wouldn't you just report these in this way? One more: "“the current paper instantiates this interface.” There are many lines in the appendix written in this confusing non-authorial voice.
Questions for authors
  • Is there an experiment I am missing that tests the full observability setting to make sure that there's a substantive difference between it and the partially observed setting? If it's not there, this is sort of a suggestion for future work.
  • Is the speed of 25000 accounting for frame skip and the effective number of agents? I.e. in a 4 frame-skip 2 agent scene is the throughput 25k or 25k/8? I do not know how to reconcile the 1 hour claims in table 1 and the claims in table 13.
6 comments

Weakness 1 and Q1: Does partial observability actually matter in COMRAD?


Thank you for this question, you are right that the submission did not show it directly. We ran four new experiments. In short: yes, partial observability matters in COMRAD. Moreover, we found that agents rarely see their partner, and their score depends on what the partner does.

(a) Full observability helps (Q1). COMRAD follows CTDE: centralized critics and mixers may use team-level state during training, but each actor receives only its own egocentric frame at execution. We compare this standard partially observed input with a full observability variant:

  • Partial is the agent's own 3x72x128 RGB frame.
  • Full keeps the own frame and adds the global state of the environment, read from the game engine, in the way SMAC and MPE define a global state. It measures what all hidden information costs. The global state consists of:
    • every agent's position, heading, health, ammo, alive flag and scenario variables (switches, gates, stage counters, kill counts), own agent first;
    • every non-player object in the level (position, velocity, and type), including those behind walls;
    • a 16x16 grid of the walls and doors of the current layout;
    • the normalised timestep.

We chose three scenarios in which partner information matters in different ways: Rhythm Sync (synchronisation across separated stages), Armory Siege (supply and defence roles) and Co-op Puzzle (switches and gates controlled by the partner). Full is also trained on 100M frames over the same five seeds (42, 68, 81, 97, 154). Table below reports mean ±\pm 95% CI over 5 seeds.

ScenarioPartial (%)Full (%)Full minus PartialFull each seed
Co-op Puzzle76.4 ±\pm 2.298.3 ±\pm 1.0+21.998.6, 98.2, 97.2, 99.4, 98.2
Armory Siege19.2 ±\pm 2.529.2 ±\pm 12.4+10.039.4, 15.9, 27.4, 38.7, 24.7
Rhythm Sync67.0 ±\pm 4.065.8 ±\pm 15.0-1.271.3, 83.7, 51.9, 62.0, 60.4

Full is never below Partial (even for Rhythm Sync, the difference is within the confidence interval), as expected from an input that contains Partial. In Co-op Puzzle and Armory Siege, hidden state helps with the gap to the ceiling; however, this does not help for Rhythm Sync, and the likely reason is that the stage timing this task depends on is already learnable from the own view. So partial observability is a substantive part of COMRAD's difficulty, and for some tasks learning to coordinate remains hard even with global state.

(b) How often is the teammate visible, and how often is the view blocked? COMRAD logs what each agent sees, but we did not use these in our experiments. We measured this directly for the trained MAPPO policies. MAPPO scores near the random level on Dumb Enemies, Ammo Carrier, Stealth Labyrinth and Lava Maze (Table 3), so those four rows describe the scenario under a weak policy.

  • Teammate in FOV cone: whether the partner lies inside the agent's 90°×72°90\degree \times 72\degree field-of-view (FOV) cone.
  • Teammate visible: whether it actually has rendered pixels on screen (labelled by the engine).
  • In-cone but occluded is the fraction of in cone steps where walls, doors, darkness, or other actors hide the partner completely.
  • Both see each other: Both teammates are visible.
  • The last column is the visibility rate under a uniform random policy.

Every rate is averaged over both agents' viewpoints.

ScenarioTeammate in FOV cone (%)Teammate visible (%)In-cone but occluded (%)Both see each other (%)Teammate visible, random policy (%)
Stag Hunt Arena565352646
Rhythm Sync2019502
Foraging Commons221719327
Co-op Puzzle24866111
Platform Chain472742031
Armory Siege382728921
Co-op Health Gathering171512220
Lava Pit29292021
Smart Enemies251154219
Dumb Enemies261347319
Ammo Carrier341748411
Stealth Labyrinth33317125
Lava Maze635021049

This answers your first three questions under the Weaknesses:

  • How often is a teammate in view? Not often. In 11 of 13 scenarios, the teammate is visible in 8–31% of steps. The exceptions are Stag Hunt Arena (53%) and Lava Maze (50%), where both agents share one open area. Also mutual visibility is rare everywhere except Stag Hunt Arena (26%).
  • How often is the view blocked? In 8 of 13 scenarios, the partner is in the FOV cone but occluded in 19-66% of the steps. In the other five (Stag Hunt Arena, Rhythm Sync, Co-op Health Gathering, Lava Pit, Stealth Labyrinth), occlusion is low (2-12%). In those, the partner is hidden mainly because it is outside the 90°90\degree cone, so where the agent looks is the limiting factor.
  • Do agents need to reason about partial observability? For most of an episode, each agent does not see its partner, yet its score depends on what the partner does (experiment (d)). Experiment (a) shows that providing the missing information improves performance in two of three scenarios. Experiment (c) shows that the current baselines mostly do not use the sight of their partner, so our benchmark does pose this problem.

(c) Masking the partner. We also replaced the partner's pixels (labelled by the engine) with the background colour, so the partner becomes invisible to a trained policy. Unlike (a), this removes the partner from the frame but keeps the rest of the scene. Results are in the table under Weakness 2 (Mask partner column).

  • Where the task revolves around the partner, masking breaks the policy. Ammo Carrier falls from 83.7% to 16.0%, close to the random policy (13.2%), and Dumb Enemies falls from 13.6% to 1.5%.
  • In the other scenarios, the trained policy keeps most of its score when the partner is masked. These policies already act mostly on what the partner changes in the world (switches, gates, bridges, items, enemies), because the partner is out of view most of the time (table (b)). Partner information still matters for these tasks, as experiment (a) showed.

(d) Replacing the partner. Not seeing the partner is different from not needing the partner. We evaluated each trained policy with its partner replaced by a no-op or a uniform random policy (full table in our response to Reviewer C2K6, Q3). In most scenarios with a non-zero team score, the remaining agent falls to the random level. So the agents depend on their partner's actions, and (a) shows that this extra information is useful when the agent receives it.

Weakness 2: Why does 3D egocentric vision matter for MARL?


The multi-agent challenge that egocentric vision adds is deciding where to look. A COMRAD agent sees a 90°90\degree cone, whereas a unit in the original SMAC observes every ally within a 360°360\degree sight radius. Looking at the objective therefore often means not looking at the teammate. Table (b) under Weakness 1 measures this: in five scenarios the partner is hidden mainly because it is outside the cone, not because something occludes it. Therefore, each agent has to choose between watching the task and watching its partner, and this trade-off does not exist with feature-vector observations. Prior work on egocentric embodied environments also finds that agents coordinate partly through perceiving their partner [1, 2], and that line-of-sight constraints drive qualitatively new multi-agent strategies [3, 4].

A related concern comes from the original SMAC, where Ellis et al. [5] showed that policies that see only the timestep win many maps. As explained in SMACv2, the reason is that the original SMAC has too little stochasticity: start positions and unit types are fixed, so a memorised action sequence is enough. This is our main motivation for a framework that procedurally generates layouts and evaluates on held-out layouts.

Is vision necessary at all? Before asking what partial observability costs, we check that COMRAD cannot be solved without perceiving the environment, which is the failure that Ellis et al. [5] found on the original SMAC. These controls test whether the policy needs its visual input (they are separate from the partial observability question above).

  • Zero-shot blind: the same trained policy is evaluated with its RGB frames zeroed.
  • Trained blind: an MAPPO policy (IPPO on Ammo Carrier, the best method there) is trained from scratch on zeroed frames, so it can only use the timestep (the optional measurement vector is off, as in our baselines). Because of time and budget, we ran this on four representative scenarios.

All numbers are % of the scenario ceiling on held-out layouts. Normal RGB is the 5-seed Table 3 mean. The ablation columns use 5 seeds. The method is MAPPO unless stated otherwise.

ScenarioNormal RGB (%)Mask partner (%)Zero-shot blind (%)Trained blind (%)Random (%)
Stag Hunt Arena70.874.00.0—0.6
Rhythm Sync67.072.40.11.70.0
Foraging Commons11.911.82.6—2.2
Co-op Puzzle76.476.16.2—4.1
Platform Chain10.310.21.4—1.4
Armory Siege19.218.80.00.00.0
Co-op Health Gathering26.724.46.4—6.2
Lava Pit9.18.80.0—0.4
Smart Enemies17.217.00.0—0.1
Dumb Enemies (IPPO)13.61.50.1—0.1
Ammo Carrier (IPPO)83.716.013.614.613.2
Stealth Labyrinth0.00.00.00.00.0
Lava Maze0.00.00.0—0.0

Without its frames, every policy drops to the random level. A recurrent policy trained to exploit only the timestep also stays at random level on Rhythm Sync, Armory Siege and Ammo Carrier. This is the opposite of what Ellis et al. found on the original SMAC; every evaluation layout is unseen, the agents can't just use time and have to perceive the environment.

State vs pixels. State observation replaces the RGB frame with a vector read from the game engine. We encode it the way partially observable MARL benchmarks such as SMAC do: entities relative to the agent, cut off by a sight radius (512 map units, about a third of the Armory Siege and Ammo Carrier maps with Medium difficulty; like SMAC's, this radius sees behind the agent and through walls). The vector contains:

  • the agent's heading, health, ammo and episode progress, without its absolute position;
  • the partner's offset, distance and relative heading;
  • nearest non-player objects (enemies, items, projectiles, decorations), each with offset, distance and a type code;
  • 16 depth-buffer distances across the 90°90\degree field of view, for walls and doors;
  • scenario task variables (role, section progress, kill counts, local cues).

Offsets are rotated into the agent's heading frame. A two-layer MLP (512-512, the same width as the layer after the CNN) replaces the CNN.

We chose three scenarios in which the partner matters in different ways. In Rhythm Sync the agents coordinate by timing across separate stages, so the partner is rarely in view and the frame should add little beyond the state. In Armory Siege the agents share one arena and split supply and defence roles, and experiment (a) shows that partner information helps. In Ammo Carrier the roles are fixed and asymmetric: the immobile defender depends on the runner bringing ammo.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.