acceptodds
Withdrawn from ICLR 2027

Emergent Tool Use in the Hide-and-Seek Environment via an Adaptive Auto-Curriculum

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study the emergence of coordinated multi-agent strategies in the quadrant configuration of Openai's hide-and-seek environment. Two teams, hiders and seekers, are trained purely through self-play under a zero-sum reward with no additional rewards on specific behaviours. To drive learning we combine self-play against a pool of past opponents with an adaptive auto-curriculum, inspired by Deepmind's Adaptive Agent research. Over the course of training we observe a sequence of distinct emergent phases: pursuit and evasion, door blocking, seeker ramp use, and a hider ramp-defense strategy. We monitor the emerge of these phase transitions and, in a controlled ablation across three random seeds, find that the adaptive curriculum reaches all four phases faster than a non-adaptive task sampler and with lower seed-to-seed variance. Relative to the phase onsets originally reported for the full hide-and-seek environment, our compact single-scenario setup reaches the final phase roughly 1.3 times faster (about 3M episodes), while following the same ordering of emergent behaviours.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, emergent tool use, autocurriculum, adaptive curriculum learning, prioritized level replay, proximal policy optimization, competitive self-play

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LwWUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper conducts experiments to understand emergent tool use in a recreation of the Hide and Seek quadrant setting studied by Baker et al 2019. They utilize historical self-play and Prioritized Level Replay with domain specific modifications to train agents that reach the four distinct strategy phases identified by Baker et al in up to 30% fewer steps. They demonstrate that the system is sensitive to the number of snapshots used for historical self-play, and the learning rate.

Their curriculum operates over 5 values of p_door, which controls whether one of the two possible doors is blocked at the start of an episode. The curriculum is controlled by PLR plus a small (15%) bonus to the least-probably value of p_door. They also add a warmup period during which p_door is set to 1.0 to encourage door blocking behavior.

Baker, Bowen, et al. "Emergent tool use from multi-agent autocurricula." arXiv preprint arXiv:1909.07528 (2019).

Critical strengths

The methods used are appropriate for the setting. The related works section touches on and correctly explains most of the relevant prior work. A better understanding of open-endedness, skill discovery, and autocurricula is of significant interest to the RL community, and this paper aims to learn about these phenomena

Critical weaknesses

It is not clear if the paper presents any novel findings, and it is almost certainly true that it does not present novel findings that generalize outside of the very specific hide and seek quadrant domain studied here. This paper reproduces results from the paper "Emergent Tool Use From Multi-Agent Autocurricula" after which it is named. However, this paper studies a much more limited setting with far fewer results. The main novel claims seem to be 1) incorporating historical self-play, Prioritized Level Replay, and additional domain-specific curriculum components into the learning mechanism to achieve the same results as Baker et al up to 30% faster and 2) Conducting a sensitivity analysis of learning rate and the number of opponents in historical self play.

The first result simply adds existing methods (historical self play and PLR) and the new components are specifically designed to teach the "emergent" behavior that the authors desire. As a result, there is limited novelty, and the novel components seem to contradict the goal of the experiment. Furthermore, the evidence is a single run, so no measures of uncertainty or generalization are possible. The claims of efficiency are measured against reported scores of the original paper which used a different environment and are not directly comparable. In addition, the ablations highlighting the importance of the curriculum components are in an appendix rather than the main paper, despite being central to the paper's claims.

The second result is nice, but the importance of learning rate and historical self play are well known in the literature, and the exact relationship found here is unlikely to generalize to environments outside the hide and seek quadrant setting.

The paper seems to lack clear motivation and as a result, does not present novel or interesting results. It does not answer the most important question of why we should care about reproducing the results of hide and seek with different methods or somewhat faster. If this is meant as a step along the path to a result that the research community would care deeply about, that is not explained. Overall the writing is unclear, with many experimental details either being poorly explained or missing. For example:

  • What is the spine configuration? It is introduced before it is partially defined.
  • Is the reward sparse or dense? Early on the writing says that hiders win if they remain hidden, but later the reward is revealed to be a dense reward for being hidden at each step.
  • Where did this environment come from? The paper cites Baker et al early on, but does not state whether the environment is provided by their paper, by a later open-source release, or created entirely from scratch in this submission
  • Why does the agent need to observe the phase flag?
Questions for authors

There are many clarity questions which I ask above. I would also ask what the motivation behind this work is, and what the authors believe are the interesting and novel results toward that goal.

1 comment
AuthorsOpenReview ↗

Response to Reviewer LwWU's Comment

We thank the reviewer for the thorough review and the specific clarity questions.

Motivation. We should have stated this clearly. We study hide-and-seek in order to understand how agents learn to act in a world where they move in two dimensions but must reason about three-dimensional structure: walls, sealed rooms, and a ramp that gives access to height. Our long-term goal is to transfer these mechanisms to teams of drones, where one team searches or pursues while another evades, takes cover or blocks access, and where useful behaviour should emerge from competition rather than be scripted. The quadrant scenario is our first, controllable step towards that goal. We will make this motivation the starting point of the paper.

Novelty and the curriculum components. We agree that the warm-up and the floor favour the single-door configuration, and that this sits uneasily with the claim that the behaviours emerge without guidance. No reward targets any behaviour. Our ongoing work removes such hand-designed biases so that every behaviour has to arise from self-play alone.

Evidence and presentation. The behavioural results in the main text come from a single reference run. We agree that this does not allow uncertainty estimates. As noted to the other reviewers, we will remove the comparison with Baker et al., add more seeds with uncertainty measures, and move the curriculum ablation into the main paper.

Clarity questions:

Spine configuration. This is our name for the single-door configuration (door dropout = 1.0), the one that can be sealed completely. We will define it before first use. Sparse or dense reward. The reward is dense. During the play phase, hiders receive +1 at every step if no hider is seen by any seeker and −1 otherwise. Seekers receive the opposite, and there is no reward during preparation. “Win” is only an evaluation measure: hiders win an episode if no hider is seen at the end of play. We will separate the two clearly. Origin of the environment. The environment is OpenAI’s open-source release (multi-agent-emergence-environments with mujoco-worldgen), in its quadrant configuration with the default settings. These defaults include the 80-step episode with 40% preparation, the non-lockable ramp, and the preparation-phase observation. We did not build it from scratch. Our only change is which door-dropout value each episode uses. We will also correct the paper’s wording, which suggested that the non-lockable ramp was our design choice. Why observe the phase flag. One policy acts in both phases, but seekers are frozen during preparation and rewards only start in play.

Reviewer_dLvjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies emergent tool use in the quadrant variant of the prior OpenAI Hide-and-Seek environment. They use AdA-style sampling as well as additional settings to help with learning. The authors find the same four phase progression as the original work but occurring earlier. They conduct a 3-seed ablation over uniform sampling to suggest that the adaptive curriculum causally speeds up and stabilizes the emergent behavior.

Critical strengths
  • The authors reproduce the original Baker et al paper on a single workstation, and have additional details needed for reproducibility.
  • The role assignment comparisons are interesting and show clear differences between ramp fetchers and door sealers.
Critical weaknesses
  • The paper has limited novelty. The sampler is not original and only chooses over five values for a single scalar. The speedup to the reported Baker numbers is done with completely different environments, scale, and setup, so it’s not really clear that the 1.3x speedup means much here.
  • The 3 sees for the causal comparison are not sufficient for establishing proof of anything. The paper could also benefit from a description of if the uniform baseline gets the warmup and floor, which could isolate how much these hand designed choices are contributing. In general most of these results are done with a single seed or very few seeds, which limits the rigor and reliability of the reported claims.
Questions for authors
  • Is there reason to expect the results about the curriculum will hold in richer task spaces where design is harder?
  • How is this setup different from Baker et al in terms of the takeaways and design, and how should we interpret the speedup if the conditions for the comparison are different.
  • Does the baseline receive the warmup and floor?
  • Could you report results over more seeds and include confidence intervals and significance tests to support the claims about the curriculum’s role.
1 comment
AuthorsOpenReview ↗

Response to Reviewer dLvj's Comment

We thank the reviewer for the detailed feedback and for noting the reproducibility value and the role-assignment analysis.

Novelty and the sampler. We agree that the sampler is a small instance of Prioritized Level Replay over a single scalar with five values. We do not claim it as a new algorithm. Our aim is a small, controllable testbed in which the emergence of tool use can be measured on a single workstation.

Comparison with Baker et al. We agree that the 1.3× figure is not meaningful as a speed-up. Baker et al. used the full multi-scenario environment, far larger batches and a different training scale, so episode counts are not comparable. We will remove this comparison as a result. We will keep their results only as qualitative context for the order in which the behaviours appear.

Seeds and statistics. We agree that three seeds are not enough to support a causal claim, and that one seed per setting in the sensitivity study is weak. A full run takes about 20 days on our hardware, which limited the number of runs. In the revision we will report more seeds with confidence intervals and significance tests, and we will word the claims to match the evidence.

Answers to the questions:

Richer task spaces. We have no evidence that the result carries over to larger task spaces. The task space here is deliberately one-dimensional, and we will say this explicitly. Difference from Baker et al. We use OpenAI’s released environment code and architecture, but restrict training to the quadrant scenario and add a curriculum over door configurations. The main takeaway was that the same order of behaviours (chasing, door blocking, ramp use, ramp defence) can be reproduced in a faster learning. Warm-up and floor. No. The uniform baseline did not receive the warm-up or the floor. A matched baseline is our first planned experiment. More seeds. Yes. See above.

Reviewer_Z2hFOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether curriculum sampling accelerates and stabilizes emergent tool use in a simplified quadrant configuration of OpenAI’s hide-and-seek environment. It combines PPO self-play, attention/LSTM policies, a historical opponent pool, and PLR/AdA-inspired prioritization over door configurations. The reported progression consists of pursuit/evasion, door blocking, seeker ramp use, and hider ramp defense. Validation includes a long reference run, a three-seed-per-condition curriculum ablation, single-seed sensitivity sweeps over opponent-pool size and learning rate, and trajectory/spatial analyses.

Critical strengths

The simplified environment preserves an interpretable sequence of tool use and strategic responses. Door blocking, ramp-based intrusion, and ramp defense form a coherent progression without rewards explicitly targeting these behaviors. Rendered snapshots, object motion, and trajectory analyses provide qualitative evidence consistent with this account. This gives the setting value as an experimental platform for curriculum learning and provides behavioral substance beyond a win-rate curve.

Critical weaknesses

The ablation does not isolate adaptive prioritization, weakening the central methodological claim. The curriculum in Section 4.3 bundles advantage-based prioritization with a 2 M-episode single-door warm-up and a persistent sampling bias toward that configuration, while Appendix A compares the complete package against uniform sampling. Mean door-blocking onset, 1.7 M episodes, even precedes the start of adaptive sampling. The results therefore support the full recipe but do not establish whether online adaptation contributes beyond the manually specified warm-up and task bias.

Questions for authors
  1. Evidence needed for the current central claim: Does the acceleration persist when the baseline retains the same 2 M-episode warm-up, single-door sampling bias, and configuration-switching frequency, with only score-based prioritization removed?

  2. Clarification of evaluation comparability: Are the Figure 5 win rates measured during each condition’s own training games or in independent evaluation under a common door-configuration distribution? Are common fixed opponents or cross-play results available to support the comparison of final strategic competence?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Z2hF's Comment

We thank the reviewer for a careful and constructive review, and for recognising the value of the quadrant setting as an interpretable platform for studying emergent tool use.

On isolating adaptive prioritisation. We agree, and this is the most important weakness the reviewer identifies. In the submitted version, the “curriculum off” condition sampled the five door configurations uniformly from the start. It had no warm-up, no floor on the single-door configuration, and no matched switching schedule. The comparison therefore tests the full recipe against uniform sampling, not adaptive prioritisation alone. The reviewer is also right that door blocking appears in the reference run at about 1.74M episodes, which is before the 2M-episode warm-up ends. Door blocking therefore cannot be credited to the adaptive part of the sampler. The correct control is the one the reviewer describes: the same warm-up, the same floor probability, the same 20-batch hold, with only the score-based ranking replaced by uniform sampling. We will run this matched control, with more seeds, before resubmitting.

On evaluation comparability. The win rates in Figure 5 were recorded during each condition’s own training games. The adaptive condition plays more single-door games, which are easier for hiders to seal, so its win rate is not directly comparable to the uniform condition’s. We did not evaluate against common fixed opponents or run cross-play. In the revised version we will evaluate every checkpoint on the same fixed distribution of door configurations. We will also report cross-play between the final policies of each condition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.