acceptodds
Under review as a conference paper at ICLR 2027

First-Explore+: Sample-Efficient Training of In-Context Exploration

Abstract

Exploration is essential for learning, yet common exploration strategies—such as random action selection, generic exploration heuristics, or priors acquired through imitation learning—adapt slowly to prior attempts, can be highly domain specific, and often require orders of magnitude more interactions than a human to systematically explore the same space. In-context meta-reinforcement learning can learn to explore adaptively and as efficiently as a human, but can also fail entirely in environments with deceptive rewards. First-Explore addresses this by meta-learning separate exploration and exploitation policies, but its original instantiation is highly sample-inefficient, restricting experiments to simple environments and potentially obscuring phenomena that only emerge at greater scales. We introduce First-Explore+, which substantially improves the efficiency of First-Explore by adding a value baseline, incorporating better transformer and RNN architectures, and developing several extensions to the First-Explore framework. First-Explore+ matches or exceeds the original method's performance in challenging meta-learning benchmarks with deceptive rewards using orders of magnitude fewer training samples. This increased efficiency allows us to evaluate the First-Explore framework in a substantially more complex environment, XLand Mini-Grid, where First-Explore+ achieves high scores while other meta-learning baselines completely fail to explore. In this environment, we provide the first clear evidence of a new phenomenon—explicitly meta-learning cross-episode exploration can enable successful zero-context single-episode performance where an otherwise matched single-episode baseline fails entirely. These results provide preliminary support for a broader role of meta-RL: training a policy for cross-episode exploration to improve returns in standard single-episode RL. Together, these advances represent progress toward increasingly general exploration priors and potentially a key step toward a foundation model for general exploration.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, meta-reinforcement learning, exploration, hard exploration, in-context reinforcement learning, exploration–exploitation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VDGBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose First-Explore+ (FE+), a successor to First-Explore (FE), an existing method for in-context meta-reinforcement learning. FE+ introduces two main modifications to FE: applying PPO to FE and allowing the timing of when to stop exploration. In addition to experiments in the same environments as FE, the authors conduct experiments in the more complex environment, XLand-MiniGrid, to evaluate the effectiveness of these modifications.

Critical strengths

The authors can be said to partially address issues raised for FE [1]. In FE kk, the number of exploration episodes before switching to the exploitation policy at meta-test time, is fixed, and it was pointed out that selecting kk requires additional validation data. FE+ addresses this issue by introducing a criterion for stopping exploration based on U(c)U(c). This eliminates the need for post-training selection of kk on a validation set. The authors also experimentally show that, in some environments, using U(c)U(c) achieves better performance than using a fixed kk.

In addition, by incorporating PPO and other modifications, the authors experimentally show that FE+ is more efficient than FE in terms of both training time and sample efficiency.

[1] https://openreview.net/forum?id=AhjTu2aiiW

Critical weaknesses
  1. The experimental evaluation is insufficient, and several questions left unresolved by the FE paper remain unclear.

As in the prior work on FE, although the experimental environments are challenging in terms of deceptive rewards, the environments themselves are relatively simple, have short episode lengths, and cover a rather limited range of task structures. From this perspective, the experimental evaluation is not sufficiently broad. Compared with the prior FE paper, XLand-MiniGrid is somewhat more complex; however, it is still simple compared with standard benchmarks such as robotic control tasks in Gymnasium, and its maximum episode length is only 50. I therefore consider the experimental evaluation insufficient to demonstrate the applicability of the proposed method to more general and complex environments.

Also, as with the FE paper, I think the experimental evaluation is limited because it lacks comparison with relatively strong and more recent meta-RL algorithms such as PEARL[1], rather than only with RL$^2$ and E-RL$^2$. Although PEARL was criticized in the VariBAD paper for not explicitly conditioning the policy on task uncertainty, it still has a practically distinct mechanism from VariBAD because it samples latent task variables from the inferred posterior.

In addition, FE+ differs from FE not only in its use of PPO and U(c)U(c), but also in other aspects such as the network architecture. Since there is no ablation study that isolates the effect of replacing the original FE training procedure with PPO while keeping the other components fixed, the contribution of the PPO-based training itself remains unclear.

  1. Many parts of the method are insufficiently explained, making the paper difficult to understand without referring to the original FE paper.

For example, cc is not explicitly defined, and it is also unclear from this paper alone what objective is actually optimized when the original FE training procedure is replaced with PPO. These components should be defined more explicitly, preferably with mathematical formulations.

[1] Rakelly et al. Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables. ICML2019

Questions for authors

Please let me know if you have any counterarguments to the points above.

3 comments

Thank you for your extensive review. We greatly appreciate your feedback and questions, and the time you have dedicated to reviewing. We are delighted that you support sharing the work with the ICLR community. Based on your feedback, we have updated the manuscript. We believe we have addressed all your concerns and questions.

We have added both VariBAD and HyperX as baselines in the FE+ paper, and demonstrate that First-Explore+ significantly outperforms both. We chose to use VariBAD and HyperX because they are more recent, potentially stronger baselines, but we now cite PEARL as well. We have clarified exactly how FE+ and FE differ as algorithms, substantially editing the method section to improve its clarity, as well as expanding the background. We have also added pseudocode for both algorithms to Appendix D, along with details on First-Explore+’s PPO objective. Please see our overall response for more details regarding these new baselines and presentation changes.

Regarding environment complexity, while limited in some respects, these environments are highly challenging in other respects: with high environment variability and heavy penalties for exploration. For example, in Ray Maze, there are thousands of possible maze environments (192 possible wall geometries, multiple goal placements and goal values). In contrast, VariBAD evaluates in a 5x5 grid world, where a goal is in one of 25 locations (25 possible environments), as well as control tasks with similarly small environment variability (e.g., walking either backwards or forwards). Deceptive XLand-MiniGrid requires complex memory and high dimensional environment disambiguation (1 million distinct rulesets and different room layouts per rule). The complexity of this learning problem is evident from the fact that no method besides FE or FE+ achieves meaningful return in this setting, despite training for 50B steps.

Like predecessors in the meta-RL literature, this paper focuses on pushing the limits of what agents can accomplish in the face of meta-RL specific challenges such as environment variability and memory. While we agree that showing these methods scale and work on environments that have other forms of challenge (e.g., long horizons) remains important future work, it is outside the scope of the current paper. And unfortunately, performing meaningful task variation on long horizon complex environments can be prohibitively expensive and outside the budget of academic labs. Indeed, the XLand-MiniGrid was created so that academic labs could perform XLand-like experiments.

To close, we greatly appreciate your feedback. We are delighted that you support sharing the work with the ICLR community. Please let us know if you have any further questions. Finally, we would greatly appreciate it if you would consider raising your score to firm acceptance, if you feel the improvements and clarification warrant that change.

Thank you for your detailed response and for revising the paper. I believe that the addition of VariBAD and HyperX, as well as the improvements to the explanation and mathematical formulation of the method, have addressed some of my concerns. However, my main concern regarding the scope of the experimental evaluation remains. I agree that Ray Maze and Deceptive XLand-MiniGrid pose challenges involving task variability, task disambiguation, and the use of information gathered in previous episodes. However, my concern is not simply whether these environments are challenging, but rather that the range of problem settings and types of challenges evaluated remains limited. Specifically, the effectiveness of FE+ in environments involving long episode horizons, continuous dynamics, and complex control remains unclear, as do its usefulness and limitations outside the high-exploration-cost settings that FE has primarily targeted. Although the addition of Deceptive XLand-MiniGrid broadens the scope of the evaluation compared with FE, the evaluation remains largely confined to the same limited problem settings as FE. I therefore do not believe that the narrow scope of empirical evaluation inherited from FE has been sufficiently addressed. Independently of my other concerns, I do not believe that my score should be changed as a result of this correction, even considering this point alone.

Thank you for your continued feedback. We are very happy you value the additions and improvements.

Given FE+'s PPO-based training, we see no obvious barrier to extending it to continuous action spaces. However, this remains an open question, and we make no claims that FE+ outperforms other methods in continuous-control or long-horizon settings. Truly long episode horizons remain an open area of study for meta-RL (and even RL) in general.

We do not believe that a paper must demonstrate generality across every major class of RL environment to make a meaningful contribution. One key contribution of this paper is demonstrating that the core idea behind First-Explore generalizes across substantially different algorithms. We believe that regardless of whether FE+ itself scales to long-horizon tasks, the underlying principle (e.g., the First-Explore framework) could apply broadly to environments with high exploration costs. Sharing this work with the ICLR community would enable others to build on these ideas and explore them in new settings.

Independently, we identify a novel and surprising phenomenon: FE+ outperforms standard RL even when evaluated without prior-episode context. Identifying new phenomena like this is valuable, and can lead to interesting future methods.

If our response increases your confidence that this paper would be of interest to the ICLR community, please consider raising your score. Regardless, thank you again for reviewing. We deeply appreciate your engagement, feedback and help strengthening the paper.

Reviewer_ZubkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors present an extension of an in-context meta-RL algorithm “First-Explore” (First-Explore+), explicitly decomposing the meta-episdoe into exploration and exploitation.

The authors evaluate their method on toy tasks from First-Explore and another, new , 2D grid-world toy task. They showed much improved sample efficiency and higher returns compared to a few standard meta-RL baselines (including First-Explore) on these benchmarks.

Finally the authors performed a qualitative analysis over a toy problem, showing the advantage of their proposed method.

Critical strengths
  1. One of the more insightful parts of the paper is Section 4.2 - which clearly illustrates how the adaptive mode switching allows for continuous exploration or switching to exploitation based on the reward of the initial episode.
  2. The studied problem of exploration-exploitation decoupling within black-box meta-rl is intriguing and worth investigation.
Critical weaknesses
  1. [Critical] Limited Novelty Over First-Explore: I find the contribution in this paper to be too incremental on top of the FIrst-Explore work. The core disentangle formulation (explore/exploit), reward masking during exploration etc, are similar (if not identical to First-Explore).
  2. [Critical] Limited Empirical Evaluation: The experiments section only compares agains RL2, E-RL2 and First Explore, and not modern in-context meta-rl baselines, some even directly targeting exploration (e.g [1, 2], which were also used in First Explore or [3,4]). Further, all of the used environments are low-dimensional toy tasks, further limiting the contribution on top of weakness 1.
  3. [Major] Poor Presentation: I found the paper to be hard to follow throughout and think it can use an overall revision. First, the introduction is too brief and misses proper setup for the in-context meta-RL setup and the problem this paper tackles, explanation on what “deceptive rewards” mean, how First-Explore works or clear contributions list. Further, Figure 1 appears to be out-of-place, presenting results from Section 4 before any explanation on the experimental setup (e.g what “Deceptive XLand” is, or what the baselines are). Some environments (e.g Bandit and Dark Treasure Room) are used in Section 4 without any description of their state/action spaces or reward functions (a brief explanation in the main text is necessary). Finally, Section 2.2 discusses baselines, instead of formal preliminaries.

[1] Zintgraf et al., "Exploration in approximate hyper-state space for meta reinforcement learning", 2020.

[2] Zintgraf et al., "VariBAD: A very good method for Bayes-adaptive deep RL via meta-learning", 2019.

[3] Liu et al., "Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices", 2020.

[4] Grigsby et al., "AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents", 2023.

Questions for authors

No further questions on top of the weaknesses.

3 comments

Thank you for taking the time to review our work, we appreciate your characterization that “[decoupling exploration-exploitation] within black-box meta-rl is intriguing and worth investigation” and we are happy that you found Section 4.2 insightful. In light of your feedback we have made several changes. Please see the overall response for more details regarding new baselines, new ablations, and overall presentation.

You are correct that decoupling exploration-exploitation is present in First-Explore. We do not claim that First-Explore+ is novel in that respect. First-Explore+ is an extension to First-Explore that makes several distinct contributions.

First-Explore+ is an implementation of the First-Explore framework. While the First-Explore framework was presented as a general framework, prior to this paper, the only algorithm implementing it was the original First-Explore (Monte Carlo rollouts without value estimation, and cross-entropy based training). We also introduce a dynamic stopping rule and exploit-context injection to address limitations of the original algorithm and extend the framework. The newly added Appendix D details the substantial difference between the two algorithms.

Per your suggestion, we now list the contributions at the end of the introduction.

  • We introduce a new algorithm within the First-Explore framework that significantly increases both performance and sample efficiency. These results demonstrate that the framework’s benefits generalize across substantially different algorithmic implementations.
  • We extend the First-Explore framework with two new techniques: exploit-context injection and a dynamic stopping rule, which simplify the framework and improve performance.
  • We replicate First-Explore with more extensive hyperparameter sweeps, additional baselines, and a new environment we introduce---Deceptive XLand-MiniGrid---with high exploration costs.
  • We identify a new phenomenon in which training using First-Explore can produce policies whose zero-shot performance exceeds that of policies trained with standard RL.

We agree that these multiple contributions were not clearly framed before, and appreciate the feedback.

We have added both VariBAD and HyperX baselines, and find that First-Explore+ significantly outperforms the two methods in all environments evaluated on. These results are reflected in the updated manuscript (Figure 3, Table 1). We are currently training the VariBAD and HyperX XLand runs, and will add these to the manuscript later in the rebuttal period.

Regarding environment complexity, while limited in some respects, these environments are highly challenging in other respects: with high environment variability and heavy penalties for exploration. For example, in Ray Maze, there are thousands of possible maze environments (192 possible wall geometries, multiple goal placements and goal values). In contrast, VariBAD evaluates in a 5x5 grid world, where a goal is in one of 25 locations (25 possible environments), as well as control tasks with similarly small environment variability (e.g., walking either backwards or forwards). Deceptive XLand-MiniGrid requires complex memory and high dimensional environment disambiguation (1 million distinct rulesets and different room layouts per rule). The complexity of this learning problem is evident from the fact that no method besides FE or FE+ achieves meaningful return in this setting, despite training for 50B steps.

Like predecessors in the meta-RL literature, this paper focuses on pushing the limits of what agents can accomplish in the face of meta-RL specific challenges such as environment variability and memory. While we agree that showing these methods scale and work on environments that have other forms of challenge (e.g., long horizons) remains important future work, it is outside the scope of the current paper. And unfortunately, performing meaningful task variation on long horizon complex environments can be prohibitively expensive and outside the budget of academic labs. Indeed, the XLand-MiniGrid was created so that academic labs could perform XLand-like experiments.

Per your suggestion, we have revised the presentation of the paper. We have expanded the background, added further explanation of First-Explore, included VariBAD and HyperX in the related work section, detailed PPO as a formal preliminary, and removed "deceptive rewards" in favor of the more intuitive "exploration costs". We have also substantially improved the writing of the paper throughout.

We greatly appreciate your feedback. We believe this work makes multiple contributions worth sharing with the ICLR community, and in light of the many improvements described above, we ask you to please consider raising your score. Regardless, please let us know if you have any other concerns or questions.

I thank the authors for the detailed rebuttal and believe the new revision improved significantly on the original.

  1. I find my "Poor Presentation" weakness to be largely solved in the new revision.
  2. With regards to the limited empirical evaluation weakness - I appreciate the authors for including more relevant baselines. The paper can still use more challenging, common environments from the meta-learning community, which are well within the budget of the reported "4,300 GPU hours".
  3. I was not fully convinced regarding the novelty of the paper. I stil find the conceptual improvement and the empirical evaluation (see above) limited.

Given the above, I have raised my score and will not strongly oppose rejection, though still tend towards one.

Thank you for responding, for reading through the revised manuscript, and for being open-minded about raising your score. We appreciate your thoughtful engagement and feedback, and we are delighted that you find the revision a significant improvement over the original.

The evaluated environments directly test the central challenge of this work: how to explore when exploration is costly. For example, in Deceptive XLand-MiniGrid, FE+ achieves an IQM trial return of 1.251.25, versus 0.060.06 for HyperX—more than a 20×20\times improvement. FE+ also more than doubles FE's return.

We agree that evaluating FE+ in other environments is important future work. However, the four evaluated environments directly support our central claims, while broader evaluation remains outside the scope of this paper.

Regarding novelty, the paper makes several distinct contributions that would be of interest to the ICLR community. For example, FE+'s ability to outperform standard RL even without any prior-episode context is both surprising and novel. Indeed, readers helping us improve earlier drafts were so surprised by this result that they assumed we meant something else.

Finally, FE+ differs substantially from FE. FE+ retains FE's core idea of training exploration on subsequent exploitation returns, but changes the learning objective, optimization method, and how the policy is used. It replaces FE's filtered likelihood loss with bootstrapped PPO across explore–exploit pairs, adds exploit-context injection, and replaces FE's fixed exploration schedule with a critic-based switching rule. For comparison, HyperX extends VariBAD primarily by adding two exploration bonuses. FE+ also performs better than FE (and indeed all other methods) on all four environments, showing the value of these many changes.

As reviewer U7bE noted, these differences were unclear in the original submission; they are now explained in Section 3 and Appendix D, though we would welcome suggestions for making them even clearer.

We ask you to consider these contributions in their totality. If, upon reflection, you believe the paper merits acceptance, we would greatly appreciate a corresponding increase in your score. Regardless, we appreciate your engagement and believe it has helped improve the paper.

Reviewer_U7bEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper proposes a modified version of First-Explore: First-Explore+, which builds on PPO, introduces an auxiliary critic for return estimation, introduces exploit context injection, and proposes a stopping rule. They also replace First Explore's Transformer by GRU. On comparison with First-Explore, RL2, E-RL2 and GRU-PPO, the paper demonstrates benefits on the proposed modification over First-Explore in terms of sample efficiency and compute-time efficiency. First-Explore +, RL2 and E-RL2 results are obtained with sufficiently rigorous hyper-parameter tuning for GRU runs.

Critical strengths
  • The paper improves upon the First-Explore algorithm by proposing several modifications:

    • It uses GRU instead of First-Explore's original transformer.
    • Uses PPO as backbone RL algorithm.
    • exploit context injection
    • an auxiliary critic for return estimation.
  • The experimental evaluation is done on 10 seeds, which is higher than found in many papers.

Critical weaknesses
  • Presentation: The paper writeup is difficult to follow despite multiple reads. The introduction includes vague descriptions in many places (e.g.., custom training procedure [Line 053], an equivalent standard-RL policy [Line 074]).
  • The paper is not written in a sufficiently self-contained manner, especially it requires the reader to read First-Explore paper to understand proposed First-Explore+ algorithm.
  • The main paper does not sufficiently highlight all the differences over First-Explore. A clear pseudocode of the algorithm would make it easier to understand. The architectural change, which lead to the computational speed improvement is not highlighted clearly in the main body of paper.
  • The paper does not provide detailed ablations of all modifications proposed over First-Explore to isolate contribution of each modification.
  • Missing baselines: The paper does not compare with VariBAD as well as HyperX.
Questions for authors
  • Can the authors expand the empirical evaluation to compare with baseline VariBAD [1] or HyperX.

  • Can the authors provide ablation study for various proposed modifications over First-Explore? Along with ablations, the compute time table should isolate compute time difference due to architectural changes and algorithmic changes over First-Explore.

  • I would like to request a significant change in presentation of the paper, particularly the introduction and methods section to address issues described above.

[1] Zintgraf, Luisa, et al. "VariBAD: A very good method for bayes-adaptive deep rl via meta-learning." arXiv preprint arXiv:1910.08348 (2019).

5 comments

We greatly appreciate your feedback and questions. Thank you for dedicating time to our work. Based on your feedback, we have added VariBAD and HyperX baselines, added a new ablation of exploit-context injection as well as a consolidated ablation section (Appendix F), and substantially improved the presentation of our paper.

Our apologies that you found the paper difficult to follow, and we appreciate your persistence in performing multiple reads. We agree the initial presentation could be greatly improved. Per your request, we have performed a significant change in the presentation of the paper (including the introduction and methods section), clarifying vague descriptions, as well as making the paper more clear in general.

Please see the overall response for more details about these changes. To answer your specific questions:

  1. We have added VariBAD and HyperX experiments, finding that FE+ and E-RL^2 outperform both VariBAD and HyperX on all three tested environments, often by large margins. We are running the remaining Deceptive XLand-MiniGrid comparisons and will share them soon. This highlights how alternative approaches to exploration may not address the specific challenge of costly exploration.
  2. We added ablations of the exploit-context injection to Appendix F.1. We have also added a reimplementation of FE with the GRU architecture (labeled FE in the results). Given the same architecture and training budget, FE performs significantly worse than FE+, suggesting that the new objective is crucial for the greater performance of FE+.
  3. Thank you for providing detailed feedback and identifying specific aspects of the writing that needed to be clarified, we have significantly reworked the writing to incorporate your suggestions.

We greatly appreciate your feedback. We believe this work makes multiple contributions worth sharing with the ICLR community, and in light of the many improvements described above, we ask you to please consider raising your score. Regardless, please let us know if you have any other concerns or questions.

I thank the authors for providing their response. Can the authors highlight the changes in the manuscript compared to the initial submission, so that they can be evaluated?

Thank you for the suggestion.

Here is an anonymous link to the redline PDF, highlighting all changes since the initial submission.

We appreciate your engagement and feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response

We greatly appreciate your feedback and questions. Thank you for dedicating time to our work. Based on your feedback, we have made several additions to the paper:

Three new baselines: VariBAD, HyperX, and a reimplementation of First-Explore that uses the same GRU architecture as the other methods. This new First-Explore baseline is referred to as FE in the results, and the original First-Explore that uses the Transformer architecture (and the increased training steps) is referred to as OFE. All these baselines are included in all four environments.

In our experiments, First-Explore+ significantly outperforms all other methods, including the new baselines. We note that in diagnostic variants of these environments without costly exploration (e.g. Bandits with a 0.0 reward fixed arm, or Dark Treasure Room and Ray Maze with only positive reward goals) these methods successfully learn a near-optimal strategy. These results strengthen our claims and demonstrate the advantage that FE+ has in environments with high exploration costs. The full results are in Table 1.

New Ablation and Dedicated Appendix: We have added the ablation for exploit-context injection (ECI) to Appendix F.1, moved the mode-switching critic rule ablation to Appendix F.2, and added a reference to Appendix E’s Transformer architecture comparison The new ablation shows that ECI allows First-Explore+ to utilize exploit-mode context at inference time without performance degradation. ECI addresses an inconvenient idiosyncrasy of First-Explore, which instead discards exploit-mode context. Our results show that ECI also allows FE+ to achieve substantially higher scores than discarding context in Ray Maze and comparable scores in the remaining environments.

Additionally, the new reimplementation of FE with GRU performs significantly worse than all other baselines in Dark Treasure Room and Ray Maze, suggesting that the new FE+ objective is necessary for FE+’s advantage over FE and OFE.

Presentation: We've reviewed the entire submission and rewritten substantial portions to improve clarity. For example, removing the use of the vague term "deceptive rewards." We have also added additional details:

  • The background section now explains First-Explore more concretely, as well as VariBAD and HyperX
  • The Methods section better clarifies how First-Explore+ differs from First-Explore
  • There is now pseudocode for both methods in Appendix D.
  • The introduction now provides a clear list of contributions.
  • The experiment section now describes each environment.

Thank you for raising these concerns, we believe these changes have dramatically improved the paper, and substantially address your concerns. In light of these improvements, we ask you to please consider raising your scores to support sharing the work with the ICLR community. Regardless, please let us know if you have further questions, or if you have remaining unaddressed concerns.

Dear Reviewers and Area Chair,

We greatly appreciate the time that you have spent reviewing our work, suggesting improvements, and engaging with us during the discussion period. We would like to summarize the result of these discussions and our case for why our work would be valuable to the ICLR community.

During the rebuttal, we have substantively addressed the majority of issues first raised. We have significantly improved the paper’s presentation; clarified the claims; added three baselines (VariBAD, HyperX and a new implementation of First-Explore); and added ablation studies.

Our paper makes multiple concrete contributions, and we believe that these contributions are both interesting to the community and rigorously supported by our experiments.

This paper:

  1. introduces a new algorithm within the First-Explore framework that significantly increases both performance and sample efficiency. These results demonstrate that the framework’s benefits generalize across substantially different algorithmic implementations.

  2. extends the First-Explore framework with two new techniques: exploit-context injection and a dynamic stopping rule, which simplifies the framework and improves performance.

  3. replicates First-Explore with more extensive hyperparameter sweeps, additional baselines, and a new environment we introduce---Deceptive XLand-MiniGrid---with high exploration costs.

  4. identifies a new phenomenon in which training using First-Explore can produce policies whose zero-shot performance exceeds that of policies trained with standard RL.

First-Explore+ represents a dramatic improvement over First-Explore in overall performance, sample efficiency, and simplicity. First-Explore+ outperforms First-Explore (along with the other evaluated methods) in every environment used by the original work, as well as in a new, more challenging setting. We do not claim that First-Explore+ outperforms existing methods outside of these costly-exploration domains, but our strong results suggest that applying First-Explore+ to new domains is a promising direction for future study.

In summary, this paper, as revised, makes surprising, clearly scoped, and well-supported contributions, and we believe that sharing such papers is beneficial to the research community.

We deeply appreciate multiple reviewers raising their scores and commenting on issues being resolved. However, we do not believe the current scores reflect the paper's quality, or that the increases so far are commensurate with how much it has improved in response to your feedback. At the moment, the current ratings place the paper in the 64th percentile of submissions, which is below the 75th percentile acceptance threshold from previous years, jeopardizing its chance of acceptance to ICLR. If you agree that our findings would be valuable to members of the ICLR community, we kindly ask that you update your scores to recommend acceptance.

Regardless, thank you again for all of the feedback and suggestions. They have greatly strengthened our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.