- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper conducts experiments to understand emergent tool use in a recreation of the Hide and Seek quadrant setting studied by Baker et al 2019. They utilize historical self-play and Prioritized Level Replay with domain specific modifications to train agents that reach the four distinct strategy phases identified by Baker et al in up to 30% fewer steps. They demonstrate that the system is sensitive to the number of snapshots used for historical self-play, and the learning rate.
Their curriculum operates over 5 values of p_door, which controls whether one of the two possible doors is blocked at the start of an episode. The curriculum is controlled by PLR plus a small (15%) bonus to the least-probably value of p_door. They also add a warmup period during which p_door is set to 1.0 to encourage door blocking behavior.
Baker, Bowen, et al. "Emergent tool use from multi-agent autocurricula." arXiv preprint arXiv:1909.07528 (2019).
Critical strengths
The methods used are appropriate for the setting. The related works section touches on and correctly explains most of the relevant prior work. A better understanding of open-endedness, skill discovery, and autocurricula is of significant interest to the RL community, and this paper aims to learn about these phenomena
Critical weaknesses
It is not clear if the paper presents any novel findings, and it is almost certainly true that it does not present novel findings that generalize outside of the very specific hide and seek quadrant domain studied here. This paper reproduces results from the paper "Emergent Tool Use From Multi-Agent Autocurricula" after which it is named. However, this paper studies a much more limited setting with far fewer results. The main novel claims seem to be 1) incorporating historical self-play, Prioritized Level Replay, and additional domain-specific curriculum components into the learning mechanism to achieve the same results as Baker et al up to 30% faster and 2) Conducting a sensitivity analysis of learning rate and the number of opponents in historical self play.
The first result simply adds existing methods (historical self play and PLR) and the new components are specifically designed to teach the "emergent" behavior that the authors desire. As a result, there is limited novelty, and the novel components seem to contradict the goal of the experiment. Furthermore, the evidence is a single run, so no measures of uncertainty or generalization are possible. The claims of efficiency are measured against reported scores of the original paper which used a different environment and are not directly comparable. In addition, the ablations highlighting the importance of the curriculum components are in an appendix rather than the main paper, despite being central to the paper's claims.
The second result is nice, but the importance of learning rate and historical self play are well known in the literature, and the exact relationship found here is unlikely to generalize to environments outside the hide and seek quadrant setting.
The paper seems to lack clear motivation and as a result, does not present novel or interesting results. It does not answer the most important question of why we should care about reproducing the results of hide and seek with different methods or somewhat faster. If this is meant as a step along the path to a result that the research community would care deeply about, that is not explained. Overall the writing is unclear, with many experimental details either being poorly explained or missing. For example:
- What is the spine configuration? It is introduced before it is partially defined.
- Is the reward sparse or dense? Early on the writing says that hiders win if they remain hidden, but later the reward is revealed to be a dense reward for being hidden at each step.
- Where did this environment come from? The paper cites Baker et al early on, but does not state whether the environment is provided by their paper, by a later open-source release, or created entirely from scratch in this submission
- Why does the agent need to observe the phase flag?
Questions for authors
There are many clarity questions which I ask above. I would also ask what the motivation behind this work is, and what the authors believe are the interesting and novel results toward that goal.
Response to Reviewer LwWU's Comment
We thank the reviewer for the thorough review and the specific clarity questions.
Motivation. We should have stated this clearly. We study hide-and-seek in order to understand how agents learn to act in a world where they move in two dimensions but must reason about three-dimensional structure: walls, sealed rooms, and a ramp that gives access to height. Our long-term goal is to transfer these mechanisms to teams of drones, where one team searches or pursues while another evades, takes cover or blocks access, and where useful behaviour should emerge from competition rather than be scripted. The quadrant scenario is our first, controllable step towards that goal. We will make this motivation the starting point of the paper.
Novelty and the curriculum components. We agree that the warm-up and the floor favour the single-door configuration, and that this sits uneasily with the claim that the behaviours emerge without guidance. No reward targets any behaviour. Our ongoing work removes such hand-designed biases so that every behaviour has to arise from self-play alone.
Evidence and presentation. The behavioural results in the main text come from a single reference run. We agree that this does not allow uncertainty estimates. As noted to the other reviewers, we will remove the comparison with Baker et al., add more seeds with uncertainty measures, and move the curriculum ablation into the main paper.
Clarity questions:
Spine configuration. This is our name for the single-door configuration (door dropout = 1.0), the one that can be sealed completely. We will define it before first use. Sparse or dense reward. The reward is dense. During the play phase, hiders receive +1 at every step if no hider is seen by any seeker and −1 otherwise. Seekers receive the opposite, and there is no reward during preparation. “Win” is only an evaluation measure: hiders win an episode if no hider is seen at the end of play. We will separate the two clearly. Origin of the environment. The environment is OpenAI’s open-source release (multi-agent-emergence-environments with mujoco-worldgen), in its quadrant configuration with the default settings. These defaults include the 80-step episode with 40% preparation, the non-lockable ramp, and the preparation-phase observation. We did not build it from scratch. Our only change is which door-dropout value each episode uses. We will also correct the paper’s wording, which suggested that the non-lockable ramp was our design choice. Why observe the phase flag. One policy acts in both phases, but seekers are frozen during preparation and rewards only start in play.