- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper identifies a failure mode it calls learnt exploration avoidance (LEA). New behaviors start out unpracticed and look worse than what the agent already does well, so RL learns to avoid them. Rising Tide does not maximize expected return directly. It trains one threshold-conditioned network to imitate a base policy (uniform random in the experiments) restricted to episodes whose return is at least the threshold.
Critical strengths
I like the LEA argument, and the method follows from it directly. Training each threshold only on data from the same or lower thresholds is not an ad hoc trick. In a deterministic environment, and in the idealized case, filtering the rollouts of a lower-threshold model gives exactly the right data for the higher one, and the ablations that break this all fail (Table 12). The Tree Climb comparison is fair, with the same hyperparameter search and step budget for every method. The backtracking experiment is a nice touch. Once the action that lets the agent retreat is removed, DQN solves the task (Table 8), which is good evidence for the "avoidance" explanation, at least for DQN. Together with the meta-RL results, which come with the tuning caveat the paper itself states, this is why I lean towards acceptance.
Critical weaknesses
-
The claims are broader than the evidence. LEA is presented as a general failure mode of RL and Rising Tide as a new class of algorithm. On the standard-RL side, though, everything rests on one benchmark designed by the authors, at a single scale: one depth, one jackpot sequence, and a zero-reward gate of at most three steps. This matters more than usual here. Since Rising Tide imitates the base policy conditioned on return, the share of its lower-threshold models' episodes that get through the gate should shrink roughly exponentially as the gate gets longer. So the short gate may be close to the regime where the method works best. The paper says Rising Tide is not meant to address reward sparsity, which is fair, but then it would help to see where that boundary lies; its own variants only compare no gate with a three-step gate. I would like to see how performance changes with longer gates, and ideally one existing benchmark with a similar structure, such as Deep Sea, which the paper mentions.
-
I think the novelty claim is somewhat overstated. Selecting high-return trajectories with a threshold and imitating them is not new. Sec. 3 starts from this idea itself, but the paper doesn't cite filtered behavior cloning or the cross-entropy method, which the iterated version with rising thresholds closely resembles. The interesting part seems to be maintaining a whole family of threshold-conditioned models rather than focusing only on the best trajectories. I would make this distinction clear and avoid describing the approach as "a fundamentally novel idea".
Questions for authors
- Have you tried a deep-exploration method such as bootstrapped DQN with randomized prior functions on Tree Climb? The way DQN avoids the jackpot path looks like the situation these methods were designed for, so it would help to know whether LEA also defeats them. (suggestion)
- How sensitive is Rising Tide to its hyperparameters on Tree Climb? For example, how many of the random-search configurations reach the jackpot? (clarification)
- The abstract says the method was demonstrated with Transformer, LSTM and fully connected architectures. Tree Climb uses an MLP and the meta-RL runs appear to use a Transformer, so which experiment used the LSTM? (clarification)
Thank you for your feedback and questions. We greatly appreciate the time you have spent reviewing our paper, and we are delighted you support sharing the work with the ICLR community. We are very glad the paper read clearly to you. Please also see our General Response, which summarises the additional experiments and the changes to the paper; we answer your specific questions below.
Does LEA also affect a deep-exploration method such as bootstrapped DQN with randomized prior functions on Tree Climb?
To answer your question, we implemented Bootstrapped DQN with randomized prior functions (BootDQN+prior, Osband et al., Randomized Prior Functions for Deep Reinforcement Learning, NeurIPS 2018), based on the bsuite reference implementation and our DQN baseline. BootDQN+prior succumbs to LEA as much as, if not more than, vanilla DQN. It solved none of the three variants in the hyperparameter search (0 of 50 configurations on each; Appendix D.3), and on Combined Modifications none of its follow-up runs reached the jackpot (Table 2).
BootDQN+prior is designed to address deep-sea style exploration challenges (how to exhaustively explore a limited state space in the presence of a deceptive signal), but it does not help address Tree Climb (how to follow a weak signal in the presence of a strong one). The issue is that Tree Climb has an enormous state space, and so exploring based on novelty causes the agent to explore states away from both the Jackpot Path and the Easy Path.
Intriguingly, on the Constant-Action Easy Path variant, BootDQN+prior discovers the Jackpot in some runs (just as vanilla DQN does), but then unlike vanilla DQN, it unlearns the Jackpot Path, and follows only the Easy Path by the end of training. We believe this effect arises due to BootDQN+prior’s exploration causing it to stumble off of the Jackpot Path, and then, once lost, it is unlikely to be regained.
How sensitive is Rising Tide to its hyperparameters on Tree Climb, and how many random-search configurations reach the jackpot?
Rising Tide tolerated wide variation in our Tree Climb hyperparameter search. On the Combined Modifications variant, successful configurations varied considerably, with δ ranging from 1 to 4, θ from 0.3 to 10, and the entropy weight from 0.003 to 0.1 (the entire swept range). They used u from 0 to 2, ϵ of 0.025 or 1/30, and a learning rate of 3×10−5 or 10−4. For every hyperparameter, the successful values reach an edge of the swept range, so the working range extends at least that far and may continue beyond. We also observed configurations transfer between variants of Tree Climb unchanged: every configuration that solved Reward-Gated Jackpot Path also solved Combined Modifications and Constant-Action Easy Path, and the three best configurations solved 29 of 30 runs on newly sampled Jackpot Path sequences (Appendix K.5).
In the same random search, 5 of 50 Rising Tide configurations solved Reward-Gated Jackpot Path and 7 of 50 solved Combined Modifications, against 0 of 400 on each across the other methods, and on Constant-Action Easy Path 28 of 50 did, more than for any other method (16 for DQN, the next highest). Appendix D.3 gives the outcomes for every method and lists each Rising Tide configuration.
Which experiment used the LSTM?
Thank you for identifying that this detail was missing. The bandit experiments used an LSTM. This is currently present in the supplementary material code. We have also added this detail to the paper (Section 4.2).
Thank you for your time, and please let us know if you have further questions. If you feel our response has meaningfully addressed your concerns, we would greatly appreciate it if you could raise your score accordingly.