- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose First-Explore+ (FE+), a successor to First-Explore (FE), an existing method for in-context meta-reinforcement learning. FE+ introduces two main modifications to FE: applying PPO to FE and allowing the timing of when to stop exploration. In addition to experiments in the same environments as FE, the authors conduct experiments in the more complex environment, XLand-MiniGrid, to evaluate the effectiveness of these modifications.
Critical strengths
The authors can be said to partially address issues raised for FE [1]. In FE , the number of exploration episodes before switching to the exploitation policy at meta-test time, is fixed, and it was pointed out that selecting requires additional validation data. FE+ addresses this issue by introducing a criterion for stopping exploration based on . This eliminates the need for post-training selection of on a validation set. The authors also experimentally show that, in some environments, using achieves better performance than using a fixed .
In addition, by incorporating PPO and other modifications, the authors experimentally show that FE+ is more efficient than FE in terms of both training time and sample efficiency.
Critical weaknesses
- The experimental evaluation is insufficient, and several questions left unresolved by the FE paper remain unclear.
As in the prior work on FE, although the experimental environments are challenging in terms of deceptive rewards, the environments themselves are relatively simple, have short episode lengths, and cover a rather limited range of task structures. From this perspective, the experimental evaluation is not sufficiently broad. Compared with the prior FE paper, XLand-MiniGrid is somewhat more complex; however, it is still simple compared with standard benchmarks such as robotic control tasks in Gymnasium, and its maximum episode length is only 50. I therefore consider the experimental evaluation insufficient to demonstrate the applicability of the proposed method to more general and complex environments.
Also, as with the FE paper, I think the experimental evaluation is limited because it lacks comparison with relatively strong and more recent meta-RL algorithms such as PEARL[1], rather than only with RL$^2$ and E-RL$^2$. Although PEARL was criticized in the VariBAD paper for not explicitly conditioning the policy on task uncertainty, it still has a practically distinct mechanism from VariBAD because it samples latent task variables from the inferred posterior.
In addition, FE+ differs from FE not only in its use of PPO and , but also in other aspects such as the network architecture. Since there is no ablation study that isolates the effect of replacing the original FE training procedure with PPO while keeping the other components fixed, the contribution of the PPO-based training itself remains unclear.
- Many parts of the method are insufficiently explained, making the paper difficult to understand without referring to the original FE paper.
For example, is not explicitly defined, and it is also unclear from this paper alone what objective is actually optimized when the original FE training procedure is replaced with PPO. These components should be defined more explicitly, preferably with mathematical formulations.
[1] Rakelly et al. Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables. ICML2019
Questions for authors
Please let me know if you have any counterarguments to the points above.
3 comments
Thank you for your detailed response and for revising the paper. I believe that the addition of VariBAD and HyperX, as well as the improvements to the explanation and mathematical formulation of the method, have addressed some of my concerns. However, my main concern regarding the scope of the experimental evaluation remains. I agree that Ray Maze and Deceptive XLand-MiniGrid pose challenges involving task variability, task disambiguation, and the use of information gathered in previous episodes. However, my concern is not simply whether these environments are challenging, but rather that the range of problem settings and types of challenges evaluated remains limited. Specifically, the effectiveness of FE+ in environments involving long episode horizons, continuous dynamics, and complex control remains unclear, as do its usefulness and limitations outside the high-exploration-cost settings that FE has primarily targeted. Although the addition of Deceptive XLand-MiniGrid broadens the scope of the evaluation compared with FE, the evaluation remains largely confined to the same limited problem settings as FE. I therefore do not believe that the narrow scope of empirical evaluation inherited from FE has been sufficiently addressed. Independently of my other concerns, I do not believe that my score should be changed as a result of this correction, even considering this point alone.
Thank you for your continued feedback. We are very happy you value the additions and improvements.
Given FE+'s PPO-based training, we see no obvious barrier to extending it to continuous action spaces. However, this remains an open question, and we make no claims that FE+ outperforms other methods in continuous-control or long-horizon settings. Truly long episode horizons remain an open area of study for meta-RL (and even RL) in general.
We do not believe that a paper must demonstrate generality across every major class of RL environment to make a meaningful contribution. One key contribution of this paper is demonstrating that the core idea behind First-Explore generalizes across substantially different algorithms. We believe that regardless of whether FE+ itself scales to long-horizon tasks, the underlying principle (e.g., the First-Explore framework) could apply broadly to environments with high exploration costs. Sharing this work with the ICLR community would enable others to build on these ideas and explore them in new settings.
Independently, we identify a novel and surprising phenomenon: FE+ outperforms standard RL even when evaluated without prior-episode context. Identifying new phenomena like this is valuable, and can lead to interesting future methods.
If our response increases your confidence that this paper would be of interest to the ICLR community, please consider raising your score. Regardless, thank you again for reviewing. We deeply appreciate your engagement, feedback and help strengthening the paper.
Thank you for your extensive review. We greatly appreciate your feedback and questions, and the time you have dedicated to reviewing. We are delighted that you support sharing the work with the ICLR community. Based on your feedback, we have updated the manuscript. We believe we have addressed all your concerns and questions.
We have added both VariBAD and HyperX as baselines in the FE+ paper, and demonstrate that First-Explore+ significantly outperforms both. We chose to use VariBAD and HyperX because they are more recent, potentially stronger baselines, but we now cite PEARL as well. We have clarified exactly how FE+ and FE differ as algorithms, substantially editing the method section to improve its clarity, as well as expanding the background. We have also added pseudocode for both algorithms to Appendix D, along with details on First-Explore+’s PPO objective. Please see our overall response for more details regarding these new baselines and presentation changes.
Regarding environment complexity, while limited in some respects, these environments are highly challenging in other respects: with high environment variability and heavy penalties for exploration. For example, in Ray Maze, there are thousands of possible maze environments (192 possible wall geometries, multiple goal placements and goal values). In contrast, VariBAD evaluates in a 5x5 grid world, where a goal is in one of 25 locations (25 possible environments), as well as control tasks with similarly small environment variability (e.g., walking either backwards or forwards). Deceptive XLand-MiniGrid requires complex memory and high dimensional environment disambiguation (1 million distinct rulesets and different room layouts per rule). The complexity of this learning problem is evident from the fact that no method besides FE or FE+ achieves meaningful return in this setting, despite training for 50B steps.
Like predecessors in the meta-RL literature, this paper focuses on pushing the limits of what agents can accomplish in the face of meta-RL specific challenges such as environment variability and memory. While we agree that showing these methods scale and work on environments that have other forms of challenge (e.g., long horizons) remains important future work, it is outside the scope of the current paper. And unfortunately, performing meaningful task variation on long horizon complex environments can be prohibitively expensive and outside the budget of academic labs. Indeed, the XLand-MiniGrid was created so that academic labs could perform XLand-like experiments.
To close, we greatly appreciate your feedback. We are delighted that you support sharing the work with the ICLR community. Please let us know if you have any further questions. Finally, we would greatly appreciate it if you would consider raising your score to firm acceptance, if you feel the improvements and clarification warrant that change.