- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3(+1)
- Soundness
- 3(+2)
Read review
Paper summary
This paper introduces COMRAD, a ViZDoom based benchmark for embodied cooperative multi-agent RL where each agent only sees through its own first-person view, so teammates can be out of sight behind walls. COMRAD provides 13 cooperative scenarios in which coordination must emerge solely from local, visual and egocentric observations. They also introduce DoomGen, a procedural generated library that generates various seedable maps in batches with separate difficulty levers for task mechanics and space. The training stack is built on Sample Factory and evaluated on 8 baseline configurations across 7 algorithm families under a shared 100M-step budget training on shaped rewards but evaluating on a separate task score. A clear pattern emerges is that no single algorithmic family dominates across all scenarios. The final part digs into the two very Difficult scenarios where every baseline scores near zero - using scripted partners, oracle variants and human pairs, they argue that the bottleneck is the agents co-adapting between roles rather than perception or reward design.
Critical strengths
- It fills a real gap as a cooperative MARL testbed with first-person vision, procedural layouts (DoomGen) and ready baselines to immediately start training on especially integrating with sample factory
- The Very Difficult tier is carefully checked - scripted partners, oracle variants and human pairs make a good case that it’s hard because of coordination, not because the tasks are broken
Critical weaknesses
- The results don’t line up - IPPO’s numbers in Table 3 are ~1.5-5x lower than where its own learning curves in Fig. 28 end up on three scenarios, and Figs. 4 and 29 don’t quite match the table they’re built from. The “why methods win” story is also shaky, where each algorithm family gets different reward handling and architectures, so the gaps can’t be pinned on the algorithms themselves, and QMIX’s wins over MAPPO are within noise.
- The benchmark doesn’t really show it measures what it says - there’s no random baseline to compare against, and methods barely get anywhere on Foraging Commons and Lava Pit (~12% and ~9% of the ceiling at best). The claims that the two difficulty knobs work independently and that scores don’t drop on new maps aren’t backed by the evidence they show, and they miss earlier work like Megaverse, so the “no existing platform” claim in the abstract needs a bit of toning down.
Questions for authors
- Why are IPPO’s Table 3 scores on Smart Enemies, Armory Siege and Co-op Puzzle ~1.5-5x lower than its Fig. 28 curves? Are the curves on training or held-out maps, and can you share per-seed scores?
- Tables 4-5 say on-policy methods use “scenario-dependent team blending” while value-based ones get per-agent rewards - what blending did each scenario use, and how much of the on- vs off-policy gaps comes from that?
- Could you add a random and a frozen-partner baseline for the main scenarios(App. H only scripts partners on the two hardest), plus training-map scores next to Table 3, so that the score floor and the no-degradation claim can be verified ?
- The paper says the 100M budget is in logical environment steps, but at its own App. E throughput for two agents (~1,676 env steps/s) that’s ~17 hours, or ~9 hours even at 25K FPS - how does Table 1 get ~1 hour?
5 comments
W2: Does the benchmark measure what it claims?
Random baseline. We added a uniform random policy to Table 3 (table under Q3). It confirms that the floor is close to 0 in most scenarios, but not everywhere: random reaches 13.2% on Ammo Carrier, 6.2% on Co-op Health Gathering and 4.1% on Co-op Puzzle.
Foraging Commons and Lava Pit. We agree that the best methods reach only about 12% and 9% of the ceiling, but the scores are still above the floor. The best methods score 11.9% and 9.1% against 2.2% and 0.4% for the random policy, and they fall back to the random level when the partner is replaced (1.7% and 0.0% with a no-op partner, frozen-partner table under Q3). When designing the scenarios, we also inspected the trained agents and confirmed that they learn the intended basic behaviour (restore spawn rate and hold the bridge at least once). However, they fail to stick to their assigned role (e.g. keep the bridge up) for the full episode, which is what the ceiling requires. The revised Section 5 has explicitly noted this.
Difficulty knobs. The two axes are independent in the sense that they are configured separately, which holds by construction. You are right that the grid in Appendix D.5 does not show more than that. The differences between its cells are within noise, and the scores don't represent difficulty well. We inspected the recordings and found that this happens because even though the corridors are made smaller and longer, it still did not take longer to traverse, and thus did not cause much more difficulty for the agents.
We replaced the grid with a new one on Co-op Puzzle, with a much wider range between the easiest and the hardest cell. One trained policy (MAPPO) is evaluated zero-shot in every cell, on 5 held-out layouts with 10 episodes each. Scores are in % of the ceiling, and is the 95% CI over the 5 layouts.
- Spatial axis: room size and plate size.
- Mechanical axis: the frequency of switching between 2 room types.
| Spatial \ Mechanical | Easy | Medium | Hard |
|---|---|---|---|
| Easy | 72.0 5.5 | 56.4 2.3 | 24.4 2.3 |
| Medium | 33.2 3.8 | 30.4 6.1 | 18.4 2.3 |
| Hard | 10.4 2.9 | 7.2 2.9 | 8.4 2.3 |
We increase the range between difficulties to show the effect of each axis. Otherwise, different difficulty levels will just score the same.
The score falls down every column (the spatial axis) and along the Easy row (the mechanical axis). In the Medium row only the Hard cell is clearly lower. In the Hard row all three cells are near the floor and within each other's intervals. On this scenario the spatial axis has the larger effect; however, the harder spatial levels are also further from the training geometry, so part of their effect measures generalisation to new room sizes. In the revised PDF, this grid replaces the Armory Siege grid in Appendix D.5, and we describe the axes as separately configurable knobs whose effects interact.
No-degradation claim. The submission did not support "Performance does not degrade noticeably between training and held-out evaluation" (Section 5) with numbers. The table under Q3 now provides the training and held-out scores next to each other, and they support the non-degradation claim between training and evaluation. Held-out layouts are unseen layouts generated with the same generator settings as training; the difficulty grid above changes those settings, so its drop is a separate effect.
Megaverse. Thank you for the pointer. We will cite Megaverse in Related Work and in Table 1. We see Megaverse as complementary to COMRAD, because the two differ in focus, in task design, in the mechanics available for building tasks, and in the baselines and evaluation tools that come with each benchmark.
- Megaverse is a general-purpose simulator, and COMRAD is a cooperation benchmark. Megaverse-8 covers embodied skills such as object permanence, manipulation, memory and exploration. COMRAD builds each scenario around one mechanism from a taxonomy of cooperation (Figure 5), and we measure how much the score depends on the partner (frozen-partner table under Q3).
- ViZDoom provides richer mechanics and assets. Megaverse worlds are voxel grids with "random monochrome materials". Agents can walk, look, jump and use one binary interact action, and objects can only be placed "at discretized locations in space". On the other hand, COMRAD's coordination problems are built on task elements that do not appear in Megaverse-8, for example: enemies whose behaviour depends the team, weapons, ammunition, health, doors, plates, switches, lighting, etc.
- The two benchmarks also come with different baselines and evaluation tools. The Megaverse paper reports results for one algorithm (APPO). COMRAD releases seven baseline algorithm families implemented on one training stack, scores every method on held-out layouts, and includes scripted-partner, oracle, and human references to compare against.
Q2: Reward handling
Each agent's training reward is blended as , where is agent 's own shaped reward and is the team reward (the sum of both agents' rewards). gives the individual reward and the team reward. This is the same form as Megaverse's Team Spirit and EPyMARL's common reward option. The table below lists which reward each method receives:
| Method | Stealth Labyrinth, Lava Maze | Other 11 scenarios |
|---|---|---|
| IPPO, MAPPO, HAPPO | team reward (, sum of both agents) | individual reward () |
| IDQN | individual reward | individual reward |
| VDN, QMIX, QPLEX | team reward (sum of both agents) | team reward (sum of both agents) |
VDN, QMIX and QPLEX learn from the team reward because value decomposition factorises a single team value. IPPO, MAPPO and HAPPO use on Stealth Labyrinth and Lava Maze because the support role (scout, observer) cannot complete the objective itself, so individual reward would barely credit it. IDQN keeps the individual reward there. On Stealth Labyrinth this makes little difference, because the scenario's reward shaping splits every team bonus and penalty equally between the two agents, so IDQN's reward is close to the team reward. On Lava Maze the observer is credited for the navigator's progress, so it gets less credit than under the team reward. This does not change any result: on Lava Maze every method scores 0.0, including the seven trained on the team reward, and on Stealth Labyrinth only QMIX scores above zero. We also leave both scenarios out of the matched IPPO and IDQN comparison below.
The individual reward is only a shaped training signal. Every scenario has one team objective, every method is scored on the same team benchmark score, and there are no rival agents; this is what "purely cooperative" means in the paper. In the two social dilemmas, Stag Hunt Arena and Foraging Commons, individual reward is also what creates the dilemma, because each agent is tempted to take the safe or selfish payoff. A shared reward would remove that temptation.
Note that the reward difference does not favour the on-policy family on Stag Hunt Arena. VDN and QMIX receive the team reward, which is the easier setting, yet they score 50.3% and 0.7%, compared with 70.8% for MAPPO trained on individual reward.
How much of the on- vs off-policy gap comes from reward? We cannot separate the two cleanly. We have one matched comparison: IPPO and IDQN receive exactly the same individual reward on 11 scenarios, and IPPO scores higher on 10 of them. So the on-policy lead does not come from the reward. For VDN, QMIX and QPLEX, the comparison with on-policy methods differs in the construction of algorithm, so part of that gap may come from the reward (they handle the reward differently). We believe this is a much broader and interesting question, and should be investigated in future work.
Q3: Random baseline, training scores, frozen partner
Table 3 normalised by the ceiling, with a random policy and training-layout scores. Scores are in % of the ceiling. Each cell reports held-out / training, with training being the mean of the training-layout learning curve (Figure 28) averaged over the last 5M frames, over 5 seeds. Bold marks the best held-out method and any method whose 95% CI overlaps it.
Thank you for checking the numbers this carefully. Several of your points are correct, and we fixed them in the revised PDF.
W1 and Q1: Mismatched figures
Figures 4, 28, and 29 were drawn from an earlier local copy of the training logs; these old training logs were inflated by a bug in counting agents' progress. This bug was discovered and fixed, but the figures were not updated. In addition, the submitted Figure 28 showed a single seed per algorithm on training layouts, while Table 3 is the 5-seed mean on held-out layouts.
So the gap you noticed came from outdated logs containing counting bugs, it did not come from a drop between training and held-out layouts.
In the revised PDF:
W1: QMIX vs MAPPO are ties. You are right that QMIX's wins over MAPPO on Foraging Commons and Armory Siege are within noise. In the revised PDF, we report them as ties (both bold in the table under Q3), and we updated the claims that depended on them.
W1: Can the gaps be pinned on the algorithms? We ported every algorithm into our asynchronous training stack ourselves. During this process and when designing the scenarios, we optimized the hyperparameters for each scenario.
Some of the remaining differences are part of the algorithms: the mixer, the centralized critic, HAPPO's sequential updates. Others are implementation choices inherited from the reference codebases and are not part of the algorithm. We kept these architecture details because forcing one architecture and update scheme on all methods would mean evaluating variants that differ from the published ones. Cooperative MARL benchmarks have usually compared algorithms the same way, as complete methods: SMAC [1], EPyMARL [2] and the MAPPO study [3] are examples. Reward handling follows the same principle (see Q2).
Therefore, our rankings compare complete methods within their families. A claim that a gap comes from one specific component, such as the centralized critic or the mixer, can only be tested directly when two methods share everything else.
References:
[1] Samvelyan et al. "The StarCraft Multi-Agent Challenge." AAMAS 2019 (arXiv:1902.04043).
[2] Papoudakis et al. "Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks." NeurIPS 2021 Datasets and Benchmarks Track (arXiv:2006.07869).
[3] Yu et al. "The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games." NeurIPS 2022 Datasets and Benchmarks Track (arXiv:2103.01955).