acceptodds
Under review as a conference paper at ICLR 2027

Rising Tide: Overcoming Learnt Exploration Avoidance

Abstract

We introduce Rising Tide, a new algorithmic approach for learning performant policies that, unlike reinforcement learning methods, does not explicitly maximise expected return. Given a base policy, Rising Tide learns a sequence of policies indexed by return threshold, each approximating the base policy restricted to trajectories above that threshold, and trains higher-threshold policies by bootstrapping from lower-threshold ones. This design addresses a failure mode we identify as Learnt Exploration Avoidance (LEA): the tendency of RL agents to actively suppress exploratory behaviour because it initially appears suboptimal. We show that, in LEA-inducing environments, LEA can prevent standard RL methods from reaching optimal behaviour even with effectively unlimited training time. Rising Tide is architecture-agnostic; we demonstrate it with Transformer, LSTM, and fully connected architectures. To evaluate LEA in standard RL, we introduce Tree Climb, a new benchmark designed to isolate LEA. On three established meta-RL domains, Rising Tide delivers consistently strong performance, and on Tree Climb it is the only method to solve either of the two hardest variants.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, exploration, meta-reinforcement learning, in-context reinforcement learning, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EJanOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a failure mode it calls learnt exploration avoidance (LEA). New behaviors start out unpracticed and look worse than what the agent already does well, so RL learns to avoid them. Rising Tide does not maximize expected return directly. It trains one threshold-conditioned network to imitate a base policy (uniform random in the experiments) restricted to episodes whose return is at least the threshold.

Critical strengths

I like the LEA argument, and the method follows from it directly. Training each threshold only on data from the same or lower thresholds is not an ad hoc trick. In a deterministic environment, and in the idealized case, filtering the rollouts of a lower-threshold model gives exactly the right data for the higher one, and the ablations that break this all fail (Table 12). The Tree Climb comparison is fair, with the same hyperparameter search and step budget for every method. The backtracking experiment is a nice touch. Once the action that lets the agent retreat is removed, DQN solves the task (Table 8), which is good evidence for the "avoidance" explanation, at least for DQN. Together with the meta-RL results, which come with the tuning caveat the paper itself states, this is why I lean towards acceptance.

Critical weaknesses
  1. The claims are broader than the evidence. LEA is presented as a general failure mode of RL and Rising Tide as a new class of algorithm. On the standard-RL side, though, everything rests on one benchmark designed by the authors, at a single scale: one depth, one jackpot sequence, and a zero-reward gate of at most three steps. This matters more than usual here. Since Rising Tide imitates the base policy conditioned on return, the share of its lower-threshold models' episodes that get through the gate should shrink roughly exponentially as the gate gets longer. So the short gate may be close to the regime where the method works best. The paper says Rising Tide is not meant to address reward sparsity, which is fair, but then it would help to see where that boundary lies; its own variants only compare no gate with a three-step gate. I would like to see how performance changes with longer gates, and ideally one existing benchmark with a similar structure, such as Deep Sea, which the paper mentions.

  2. I think the novelty claim is somewhat overstated. Selecting high-return trajectories with a threshold and imitating them is not new. Sec. 3 starts from this idea itself, but the paper doesn't cite filtered behavior cloning or the cross-entropy method, which the iterated version with rising thresholds closely resembles. The interesting part seems to be maintaining a whole family of threshold-conditioned models rather than focusing only on the best trajectories. I would make this distinction clear and avoid describing the approach as "a fundamentally novel idea".

Questions for authors
  1. Have you tried a deep-exploration method such as bootstrapped DQN with randomized prior functions on Tree Climb? The way DQN avoids the jackpot path looks like the situation these methods were designed for, so it would help to know whether LEA also defeats them. (suggestion)
  2. How sensitive is Rising Tide to its hyperparameters on Tree Climb? For example, how many of the random-search configurations reach the jackpot? (clarification)
  3. The abstract says the method was demonstrated with Transformer, LSTM and fully connected architectures. Tree Climb uses an MLP and the meta-RL runs appear to use a Transformer, so which experiment used the LSTM? (clarification)
1 comment

Thank you for your feedback and questions. We greatly appreciate the time you have spent reviewing our paper, and we are delighted you support sharing the work with the ICLR community. We are very glad the paper read clearly to you. Please also see our General Response, which summarises the additional experiments and the changes to the paper; we answer your specific questions below.

Does LEA also affect a deep-exploration method such as bootstrapped DQN with randomized prior functions on Tree Climb?

To answer your question, we implemented Bootstrapped DQN with randomized prior functions (BootDQN+prior, Osband et al., Randomized Prior Functions for Deep Reinforcement Learning, NeurIPS 2018), based on the bsuite reference implementation and our DQN baseline. BootDQN+prior succumbs to LEA as much as, if not more than, vanilla DQN. It solved none of the three variants in the hyperparameter search (0 of 50 configurations on each; Appendix D.3), and on Combined Modifications none of its follow-up runs reached the jackpot (Table 2).

BootDQN+prior is designed to address deep-sea style exploration challenges (how to exhaustively explore a limited state space in the presence of a deceptive signal), but it does not help address Tree Climb (how to follow a weak signal in the presence of a strong one). The issue is that Tree Climb has an enormous state space, and so exploring based on novelty causes the agent to explore states away from both the Jackpot Path and the Easy Path.

Intriguingly, on the Constant-Action Easy Path variant, BootDQN+prior discovers the Jackpot in some runs (just as vanilla DQN does), but then unlike vanilla DQN, it unlearns the Jackpot Path, and follows only the Easy Path by the end of training. We believe this effect arises due to BootDQN+prior’s exploration causing it to stumble off of the Jackpot Path, and then, once lost, it is unlikely to be regained.

How sensitive is Rising Tide to its hyperparameters on Tree Climb, and how many random-search configurations reach the jackpot?

Rising Tide tolerated wide variation in our Tree Climb hyperparameter search. On the Combined Modifications variant, successful configurations varied considerably, with δ\delta ranging from 1 to 4, θ\theta from 0.3 to 10, and the entropy weight from 0.003 to 0.1 (the entire swept range). They used uu from 0 to 2, ϵ\epsilon of 0.025 or 1/30, and a learning rate of 3×10−53 \times 10^{-5} or 10−410^{-4}. For every hyperparameter, the successful values reach an edge of the swept range, so the working range extends at least that far and may continue beyond. We also observed configurations transfer between variants of Tree Climb unchanged: every configuration that solved Reward-Gated Jackpot Path also solved Combined Modifications and Constant-Action Easy Path, and the three best configurations solved 29 of 30 runs on newly sampled Jackpot Path sequences (Appendix K.5).

In the same random search, 5 of 50 Rising Tide configurations solved Reward-Gated Jackpot Path and 7 of 50 solved Combined Modifications, against 0 of 400 on each across the other methods, and on Constant-Action Easy Path 28 of 50 did, more than for any other method (16 for DQN, the next highest). Appendix D.3 gives the outcomes for every method and lists each Rising Tide configuration.

Which experiment used the LSTM?

Thank you for identifying that this detail was missing. The bandit experiments used an LSTM. This is currently present in the supplementary material code. We have also added this detail to the paper (Section 4.2).

Thank you for your time, and please let us know if you have further questions. If you feel our response has meaningfully addressed your concerns, we would greatly appreciate it if you could raise your score accordingly.

Reviewer_afvEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a problem in RL, Learnt Exploration Avoidance (LEA), where RL agents learn to avoid potentially useful behaviors because they initially perform worse than already optimized ones. To address this, the authors propose Rising Tide, which learns a family of return-threshold-conditioned policies rather than optimizing a single policy for expected return. Lower-threshold policies generate trajectories that are used to train higher-threshold policies through filtered behavior cloning. This preserves exploratory behaviors while progressively improving higher-return ones. The authors evaluate their method on three meta-RL environments and a new benchmark. Rising Tide performs well on all three meta-RL tasks and solves the two hardest Tree Climb variants in all 10 seeds, while none of the tested baselines succeed.

Critical strengths
  1. An interesting alternative to reward maximization. Rather than encouraging exploration with additional rewards, Rising Tide prevents promising but initially suboptimal behaviors from being discarded. I find this a compelling way to approach exploration.

  2. Convincing evidence of learned exploration avoidance. The Tree Climb experiments are particularly interesting. Removing the backtracking action allows DQN to solve a previously unsolvable variant, suggesting that the failure comes from actively avoiding exploration rather than simply failing to learn the optimal path.

Critical weaknesses
  1. Limited evidence of generality. The experiments focus on environments specifically designed or selected to induce LEA. While useful for isolating the phenomenon, they leave open whether Rising Tide is effective in more realistic RL problems. In particular, the small discrete action spaces and relatively short horizons make it difficult to assess scalability.

  2. The distinction from existing exploration approaches could be stronger. LEA seems closely related to premature convergence and loss of behavioral diversity, which have motivated novelty search, quality diversity, and archive-based exploration. These approaches are discussed but not compared empirically. It is unclear whether Rising Tide offers an advantage over other methods that explicitly preserve alternative behaviors.

Questions for authors
  1. Could the authors compare Rising Tide against a simple archive-based exploration method on Tree Climb? Since the environment is deterministic, replaying promising action sequences seems like a relevant alternative.

  2. How sensitive is the method to reward scaling and threshold selection? The reliance on return-threshold sampling and capping, together with the filter-temperature sensitivity in Ray Maze, raises questions about how easily the method transfers to new environments.

2 comments

Thank you for your feedback and questions. We greatly appreciate the time you have spent reviewing our paper, and we are delighted you support sharing the work with the ICLR community. We are also delighted you consider Rising Tide a “compelling way to approach exploration.” Please see our General Response, which summarises the additional experiments and the changes to the paper; we answer your specific questions below.

How does Rising Tide compare with a simple archive-based exploration method on Tree Climb?

We investigated some simple archive-based heuristics, such as a depth-first search, prioritising exploring from the states with highest return, as well as a timestep-and-return-based archive. A standard depth-first search provably fails to discover the Jackpot within the 3×1093 \times 10^9 environment steps in both the Reward-Gated Jackpot Path and Combined Modifications variants: the Easy Path is discovered first and quickly completed, and the search then explores backward from its end, where there are over 329≥10133^{29} \geq 10^{13} states to exhaust. A simple timestep-and-return-based archive failed similarly.

An archive rule can be configured to solve these specific variants (for example, not exploring from states preceded by four or more unrewarding transitions), but such rules need hand design for each environment; as Lu et al. (Intelligent Go-Explore, ICLR 2025) note, Go-Explore “requires manually designing heuristics to guide exploration.” Rising Tide needs no such environment-specific design, so we compare it with other methods that also learn from scratch. These details are now in Appendix C.1.

How sensitive is the method to reward scaling, threshold selection, capping and the filter temperature, and how easily does it transfer to new environments?

Threshold selection. In our Tree Climb hyperparameter search, successful configurations on Combined Modifications spanned δ\delta from 1 to 4 and θ\theta from 0.3 to 10. In both cases the largest swept value still succeeded, so the working range may extend further. Working configurations also transferred unchanged: every configuration that solved Reward-Gated Jackpot Path also solved the other two variants, and the three best configurations solved 29 of 30 runs on newly sampled Jackpot Path sequences (Appendix K.5). Appendix D.3 lists the outcome of every configuration.

Reward scaling and capping. Several of Rising Tide’s hyperparameters are effectively expressed in reward units. This is not unusual: in Atari, for example, rewards were clipped to a predetermined range to allow learning across many games with a single algorithm (van Hasselt et al., Learning values across many orders of magnitude, NeurIPS 2016). Return capping handles one specific case, a rare and very large return such as Tree Climb’s +1000+1000 jackpot, which would otherwise dominate the sampling of return thresholds (Appendices G.3 and I).

Filter temperature. This is an important parameter, but one that can be tuned like any other, and at the appropriate magnitude the method is stable across a range of values: Ray Maze needs soft filtering and performs best for σ\sigma between 1.33 and 3, and Dark Treasure Room is relatively insensitive for σ\sigma from 0 to 4 (Appendix G.2). A simple default might be a logarithmically spaced sweep between a very small value and the maximum return obtainable in the environment.

Transfer to new environments. As the limitations section acknowledges, we evaluate Rising Tide in controlled domains chosen to make LEA salient. The meta-RL domains (on which Rising Tide performs well) are nonetheless highly stochastic, a property that related methods such as UDRL have been shown to struggle with. A priori, we see no reason to expect the method would not transfer, but establishing this lies outside the scope of the current paper.

Thank you for your time, and please let us know if you have further questions. If you feel our response has meaningfully addressed your concerns, we would greatly appreciate it if you could raise your score accordingly.

Thanks for addressing my questions. I keep my positive rating.

Reviewer_pxDdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies learnt exploration avoidance (LEA), where a well-practised local strategy suppresses practice of initially poor alternatives that could become better. Rising Tide trains a return-threshold-conditioned policy family by imitating episodes that pass a return filter, with learning thresholds at least as high as rollout thresholds. It selects the deployment threshold by mean return. The paper introduces Tree Climb and evaluates three variants alongside three existing meta-RL domains. It reports that Rising Tide alone solves both harder Tree Climb variants among the tested methods. Backtracking interventions, method ablations, and filter-temperature experiments examine the proposed mechanism and its limitations. The backtracking intervention, complete hardest-task separation, and separately tuned ablations jointly support a useful empirical contribution. The evidence is strongest for a controlled exploration failure and an effective threshold-conditioned imitation method, not a general failure of RL or a guaranteed distribution-preserving algorithm

Critical strengths
  1. The backtracking intervention tests a concrete mechanism.
  • On Combined Modifications, DQN's jackpot success changes from 0/10 to 10/10 when backtracking is replaced by an always-incorrect action. (Tables 7–8, lines 1026–1058).
  • The action count is unchanged, and the modification removes rewarding recovery trajectories. (Appendix K.4, lines 1644–1649)
  1. The hardest-task results have substantial internal controls. The results support the combination of threshold conditioning and directional training.
  • All methods receive a 50-configuration search and 10 seed re-evaluation of the top three configurations. Decision-network sizes are approximately 229k–230k parameters, although architectures and total auxiliary capacity differ.
  • Rising Tide succeeds in 10/10 runs on each harder variant, versus 0/10 for every baseline; the paper reports no baseline jackpot anywhere in the initial sweeps for these variants.
  • Four separately tuned method ablations also fail in all ten reported runs on Combined Modifications. (Appendix F.3 and Table 12).
Critical weaknesses
  1. The contribution relative to elite-sample optimization is insufficiently established. The contribution may be a useful extension of elite-sample optimization, but its distinct algorithmic benefit and claimed target distribution are not yet sufficiently separated from that existing framework.
  • Rising Tide repeatedly samples episodes, filters them by return, and fits the retained actions by maximum likelihood. This shares the sample-select-refit structure of the cross-entropy method (CEM), not merely its use of a cross-entropy loss. (Algorithm 1, lines 194–213). The potentially distinctive elements are the threshold-conditioned policy family and directional transfer from lower to higher thresholds. Standard optimization CEM usually updates one sampling distribution using an adaptive elite quantile. These differences mean Rising Tide it is not identical to CEM.
  • The related-work discussion and experiments do not explicitly position the method against CEM or a matched elite-trajectory refitting baseline. The existing ablations support the proposed implementation but do not resolve this comparison. That leaves the claim of a “new algorithm class” based on a “fundamentally novel idea” insufficiently supported. (Section 2, lines 124–145; conclusion, lines 477–485).
  1. The experiments establish a controlled failure case more strongly than the broader exploration claim. The strong controlled result does not yet resolve the method's significance relative to closely related distribution-based optimization. Independent task instances and a more directly relevant comparison would address this more efficiently than simply increasing training time.
  • A designed diagnostic task is appropriate, and the backtracking intervention is informative. However, failure of the tested implementations within a finite budget does not establish a general failure of expected-return optimization or a fundamentally new exploration pathology.
  • Random path sequences were “generated once and held fixed across all experiments.” Training-seed replication therefore does not measure robustness to independently generated task structures.
Questions for authors
  1. How does Rising Tide relate to CEM's elite-sample maximum-likelihood update? Is the key contribution the persistent threshold family, directional transfer, or preservation of a fixed-base conditional distribution? Does a single-distribution elite-trajectory refitting baseline, with the same policy architecture and comparable tuning and interaction budgets, exhibit the reported avoidance?
  2. Is fixed-base-policy conditioning an intended target or a verified property? Can a small enumerable task measure the discrepancy under bootstrapping, policy mixing, shared parameters, and soft filtering?
  3. After fixing configurations, do the results persist on fresh training seeds and separately generated path sequences? (Appendix R, lines 2233–2241; Appendix K.4, lines 1650–1651).
1 comment

Thank you for your feedback and questions. We greatly appreciate the time you have spent reviewing our paper, and we are delighted you support sharing the work with the ICLR community. We are pleased that you consider the Tree Climb experiments to have “substantial internal controls.” Please also see our General Response, which summarises the additional experiments and the changes to the paper; we answer your specific questions below.

How does Rising Tide relate to CEM, what is its key contribution, and does a matched CEM baseline show the same avoidance?

We have trained the Cross-Entropy Method (CEM, Mannor et al., ICML 2003) on Tree Climb, based on the skrl library (Serrano-Muñoz et al., JMLR 2023) implementation.

Empirically, we find that CEM succumbs entirely to LEA in all three Tree Climb variants, achieving a jackpot rate of zero, even in the Constant-Action Easy Path variant. CEM was implemented with the same policy architecture as the other methods, and we then applied the same protocol as was used for the other methods (the same hyperparameter search and the same total number of environment interactions). See the figures, Table 2 and Appendix D.3 in the updated paper.

We now also include a related-work paragraph on elite-sample optimisation.

Rising Tide is distinguished from other methods by learning a persistent family of return-threshold-conditioned models, each trained only on data generated by equal- or lower-threshold models. It further differs from CEM in that it uses a fixed base policy that provides continuing action diversity throughout training.

Is fixed-base-policy conditioning an intended target or a verified property?

Approximating how the base policy would act conditioned on its trajectories meeting or exceeding a return threshold is an intended target, not a verified property. Modelling the base policy this way is a motivating ideal. As you identify, Rising Tide’s objective may diverge from it for several reasons, such as the shared parameters, soft filtering, and the chance of selecting actions according to the base policy.

Do the Tree Climb results persist on fresh training seeds and separately generated path sequences?

Rising Tide's results persist on fresh training seeds and new path sequences. To answer your question, we sampled 10 Jackpot Path sequences, and ran each of the three best Rising Tide configurations on the Combined Modifications variant on all ten (30 runs in total). Rising Tide achieved a 100% jackpot rate in 29/30 runs; the remaining run discovered the Jackpot at the end of training but had not yet learnt to follow it reliably. We now include these results in Appendix K.5.

Thank you for your time, and please let us know if you have further questions. If you feel our response has meaningfully addressed your concerns, we would greatly appreciate it if you could raise your score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We greatly appreciate your feedback, questions, and the time you have dedicated to reviewing our work. We are delighted that you collectively support accepting the paper, that Rising Tide is a “compelling way to approach exploration” (Reviewer afvE), that the paper presentation was clear (Reviewer EJan) and that the Tree Climb experiments have “substantial internal controls” (Reviewer pxDd).

To answer your questions we have run two additional controls: CEM and BootDQN+prior. In both cases, the additional controls exhibit the same pattern as the previous ones, with neither solving the hardest Tree Climb variant, despite a matched architecture, interaction budget, and hyperparameter sweep. These results are now in the paper. They include the full protocol on the hardest variant, and they will be updated with the other variants, as training on the other variants completes.

In response to the concerns about how the paper's novelty was framed (Reviewers pxDd and EJan), we now articulate what distinguishes Rising Tide from other methods, namely that it learns a persistent family of return-threshold-conditioned models, each trained only on data generated by equal- or lower-threshold models. It is these properties together which we describe as defining a new algorithm class. We have also added a comparison with CEM and filtered behaviour cloning to the related work, and removed unnecessary language from the conclusion.

Finally, we have made other amendments and additions in response to your questions and concerns, such as expanding the appendix on archive-based methods. These are detailed in the relevant individual responses.

Thank you for your time, and please let us know if you have further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.