acceptodds
Under review as a conference paper at ICLR 2027

SUN: Reaching for Novelty in Reinforcement Learning

Abstract

Exploration in reinforcement learning (RL) remains a fundamental challenge. Recent goal-conditioned RL strategies (which select goals to encourage broader state coverage) have shown promising results, but none scores a goal by novelty and reachability jointly: the two signals are traded off by hand, applied in sequence, or one is neglected outright. In this paper, we introduce a reachability-aware goalselection framework that explicitly integrates these two aspects, and that can be seamlessly incorporated into any off-policy RL algorithm. To this aim, we propose SUccessor-to-Novelty (SUN), an indicator derived from successor value functions to identify goals that are both novel and reachable. We prove that SUN recovers count-based bonuses in the limit, bounds short-horizon hitting probabilities, and provably rejects unreachable goals. We further present an adaptive goal-selection strategy that leverages these properties, along with an accurate yet lightweight pseudocount to avoid the overhead of classic methods. We back up all our claims with thorough benchmarks: SUN consistently outperforms state-of-the-art methods in standard and novel environments with unreachable or hard-to-reach states, irreversible transitions, obstacles, mazes, and unbounded spaces.

Primary area:
reinforcement learning
Keywords:
exploration, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XKzROpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SUccessor-to-Novelty (SUN), a goal-selection framework for reward-free exploration that jointly considers goal novelty and reachability. SUN defines a multiplicative score using a goal-conditioned successor value function and an inverse visitation count. The authors further introduce an adaptive goal-reselection strategy based on changes in estimated reachability, together with a lightweight neighborhood-based pseudocount estimator. Several theoretical properties are established under oracle assumptions, including count-bonus equivalence, a short-horizon hitting-probability bound, and rejection of unreachable goals. Experiments on 14 environments, including navigation and continuous-control tasks, demonstrate improvements in state coverage and visitation entropy over AdaGoal, DISCOVER, and other variants.

Critical strengths
  1. A simple and well-motivated formulation of reachability-aware exploration.

The multiplicative SUN indicator in Eq. (2) provides an intuitive way to balance novelty and reachability without introducing an additional weighting coefficient. Its connection to discounted goal-reaching values also gives the formulation a meaningful interpretation beyond a purely heuristic combination.

  1. An effective adaptive goal-selection strategy.

The proposed reselection rule addresses an important limitation of episodic goal commitment, particularly when goals become unreachable following stochastic or irreversible transitions. The comparison with episodic and per-step selection in Figure 7 provides convincing evidence for the practical benefit of this mechanism.

  1. A computationally practical novelty estimator.

The neighborhood-based pseudocount shifts density-estimation costs from repeated goal queries to replay-buffer updates. The implementation is conceptually straightforward, and the sensitivity analysis in Appendix H provides useful evidence that the estimator behaves similarly to discretized visitation counts across several environments.

  1. Strong empirical evaluation and informative ablations.

The experiments cover a diverse set of exploration challenges, including unreachable states, irreversible transitions, and continuous control. Comparisons against AdaGoal and DISCOVER are relevant, and the use of 20 random seeds strengthens the results. The indicator and goal-selection ablations help isolate the main contributions.

Critical weaknesses
  1. The theoretical guarantees are limited to an idealized setting.

The main theoretical properties in Appendix A assume exact successor values and frozen visitation counts, with additional assumptions for the deterministic analysis. The count-bonus equivalence is essentially an identity for a fixed goal-specific reward, while unreachability rejection follows from an exact zero-valued reachability estimate. These results do not establish whether learned successor values can reliably reject unreachable goals or preserve the same exploration behavior under approximation errors.

  1. The practical benefits are evaluated mainly through exploration metrics.

The experiments focus on goal coverage and visitation entropy, which are appropriate for reward-free exploration but do not necessarily translate into better downstream policy learning. Although Section 3.4 discusses integration with reward-driven RL, the paper does not directly evaluate whether SUN improves sample efficiency or final returns on downstream tasks. Such evidence would strengthen its practical significance.

  1. Goal selection is restricted to previously visited states.

SUN samples candidate goals from the replay buffer, so it cannot explicitly target unexplored states outside the collected data. This may restrict exploration in environments where reaching novel regions requires extrapolating beyond observed goals. The method also evaluates endpoint novelty rather than the cumulative novelty of trajectories, potentially overlooking informative paths through underexplored regions. The implications of these restrictions deserve further investigation.

  1. The adaptive reselection rule has limited robustness justification.

The rule discards a goal when its estimated successor value falls below the value at selection time. However, such decreases may arise from critic approximation errors, training-induced value changes, or ordinary stochastic fluctuations rather than genuine reachability deterioration. Proposition 1 only establishes stability under exact values and deterministic dynamics. The method would benefit from an analysis of sensitivity to critic noise or alternative reselection thresholds.

  1. Some experimental comparisons would benefit from tighter controls.

Appendix F specifies different candidate-pool sizes for SUN and the episodic baselines, while DISCOVER uses a fixed novelty coefficient depending on the environment family. Although these choices follow the respective implementations, they may influence both exploration performance and computational cost. Additional sensitivity analysis under matched candidate budgets and baseline coefficient tuning would help isolate the benefits of the proposed indicator.

  1. The scalability of pseudocount maintenance remains uncertain.

Although pseudocount queries take constant time, insertion requires neighborhood searches over the replay buffer, with O(NM) complexity for buffer size N and goal dimension M. The implementation also periodically estimates feature scales and may rebuild counts. Since experiments retain all collected samples, cumulative maintenance costs can grow substantially with training duration. The reported runtimes are encouraging, but larger-buffer and higher-dimensional evaluations would better substantiate scalability.

Questions for authors
  1. Can the authors quantify how often learned successor values incorrectly classify unreachable goals as reachable, particularly early in training? How sensitive is SUN's performance to such estimation errors?

  2. Have the authors evaluated whether SUN's improved coverage and visitation entropy lead to better downstream task performance or faster learning of reward-driven policies?

  3. How sensitive is the adaptive reselection mechanism to critic noise and the choice of comparison threshold? Would introducing a tolerance margin improve stability?

  4. How do the results change when candidate-pool sizes and goal-selection computational budgets are matched across methods? Was DISCOVER's novelty coefficient tuned beyond the reported default values?

  5. How does pseudocount maintenance scale with replay-buffer size and goal dimensionality in practice, including feature-scale updates and count rebuilding?

8 comments
AuthorsOpenReview ↗

Overview comment

We thank the reviewer for the careful and constructive review, and acknowledge the contribution of our formulation and empirical evaluation. In response, we have revised our paper and analyzed SUN under learned values, both in theory and by measuring goal selection and reselection against the true value in experiments. We also added a downstream evaluation, sweeps of the candidate pool and of the novelty coefficient β\beta, and a task with a 15-dimensional goal. Below, we will answer each weakness and question with details.

AuthorsOpenReview ↗

Response to W1 and Q1

W1. The theoretical guarantees are limited to an idealized setting... preserve the same exploration behavior under approximation errors.

Q1. Can the authors quantify how often learned successor values incorrectly classify unreachable goals as reachable, particularly early in training? How sensitive is SUN's performance to such estimation errors?

W1: scope of the theory

Oracle results. We agree with this point. Both count-bonus equivalence and unreachability rejection are theoretical results established under idealized oracle assumptions, and we have explicitly clarified this in the revised manuscript.

The former is now called count-scaled value and is described as an identity: SUN is the value of the goal-reaching reward scaled by the inverse count, not the state-wise count bonus of Bellemare et al. (Section 3.1 and Appendix A.2; the abstract is reworded accordingly). Both results are relabeled as propositions, and Appendix A now states for each result which conditions it uses.

Beyond the oracle. For learned values, the revision adds a proposition on goal selection under approximate values (Appendix A.5). Let the error of the learned value be at most η\eta on the candidates, let R\mathcal{R} be the reachable candidates, and let nUn_{\mathcal{U}} be the smallest count among the unreachable ones.

  • No unreachable goal is selected if η<max⁡g∈RVπ(s,g) nU/(ng+nU)\eta < \max_{g \in \mathcal{R}} V^\pi(s,g)\thinspace n_{\mathcal{U}} / (n_g + n_{\mathcal{U}}), and this condition cannot be relaxed.
  • The true score of the selected goal g^\hat g is within η (1/ng^+1/ng⋆)\eta\thinspace(1/n_{\hat g} + 1/n_{g^\star}) of the best true score, attained by g⋆g^\star.

The tolerated error shrinks as unreachable goals become rarer than reachable ones, so rejection is hardest exactly where novelty is highest. This agrees with what we measure.

Q1: how often, and how sensitive

Measurement (Appendix L). In the Gridworlds, where reachability is known exactly, we count how often each algorithm selects a goal that cannot be reached from the state where it is selected (Fig. 8a and Appendix L, Fig. 27). All SUN variants do so in less than 7% of their selections on average, and almost only early in training, while the critic is still inaccurate. The same rate is about 55% for Novelty-Only, 33% for AdaGoal, and 14% for DISCOVER.

Sensitivity. As for the sensitivity of performance to these errors, it is low. The errors are confined to early training (Fig. 27). In ThreeRoom, the adaptive check replaces 83% of the goals that were unreachable when selected, a median of 4 steps after selecting them (68% in FourRoomStuck, and 63% with sticky actions; Appendix L, Fig. 28b). Even the variant that never abandons a goal ends at about the same coverage and entropy in most Gridworlds and Classic Control environments (Appendix Q.2).

AuthorsOpenReview ↗

Response to W2 and Q2

W2. The practical benefits are evaluated mainly through exploration metrics... evidence would strengthen its practical significance.

Q2. Have the authors evaluated whether SUN's improved coverage and visitation entropy lead to better downstream task performance or faster learning of reward-driven policies?

We agree that coverage and entropy are proxies for downstream performance, and the revision adds a downstream evaluation (Appendix Q.1). In all of it, the task policy is trained fully off-policy, using data collected exclusively by the exploration policy.

Gridworlds and Classic Control. The agent learns a task Q-function from the task reward, and we evaluate its greedy policy. We also include an ϵ\epsilon-greedy agent, the only one that acts on the task Q-function during training.

  • In MountainCar, the task where exploration matters the most, SUN is the only algorithm that solves the task.
  • In FourRoomStuck and GridMaze, SUN and ϵ\epsilon-greedy attain a higher return than AdaGoal and DISCOVER, although in GridMaze the difference disappears by the end of training.
  • In Pendulum, the task is easy to learn, and all algorithms reach about the same return.
  • In LunarLander, only ϵ\epsilon-greedy improves during training.

GCRL Control. We also present the goal conditioned policy performance analysis during exploration in Appendix Q.1 and Fig. 45b. Here the task is itself to reach goals, so we evaluate the goal-conditioned policy that the agent learns while exploring, with no additional network. As in DISCOVER, we report its success rate on the task goals of the benchmark over the course of training, in AntMaze-S and ArmPush-H. These goals are never used during training. In AntMaze-S, SUN is the first algorithm to reach the task goals and stays ahead throughout training. In ArmPush-H, DISCOVER improves faster early in training, but SUN catches up after about 4M steps and ends with the highest success rate.

Better exploration is therefore not always better for downstream tasks. In MountainCar and in the GCRL tasks, the more exploration the better, and SUN attains the best results. In LunarLander, this is not the case: the reward-free agents explore the whole state space, for example by flying as high as possible, whereas the task requires the agent to land. How to combine exploration and exploitation is a long-standing problem in RL, and we do not claim to solve it: in these experiments, the task policy never collects data. A more complete method would also act greedily with respect to the task Q-function during training, for example every fixed number of episodes, or once coverage or novelty crosses a threshold. The appendix discusses these options, and we leave them to future work. It also explains why ThreeRoom, Acrobot, and CartPole are not included.

Reviewer_Qr5gOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies goal selection for reward-free exploration in goal-conditioned reinforcement learning. It proposes SUccessor-to-Novelty (SUN), which combines a successor-value reachability estimate with inverse visitation counts to favor goals that are both novel and reachable. The method includes adaptive goal reselection and a replay-buffer neighborhood pseudocount with inexpensive lookup for stored candidates. Under the stated assumptions, the analysis connects SUN to a goal-conditioned count-bonus objective, bounds finite-horizon hitting probabilities, and explains suppression of unreachable goals with exact values. Experiments use DQN and TD3 on gridworlds, classic-control environments, and goal-conditioned navigation and manipulation tasks, measuring coverage and visitation entropy. The reported comparisons, visitation heatmaps, and component ablations suggest broader and more uniform exploration on the evaluated benchmarks.

Critical strengths
  1. The motivation is clear. Novelty alone can favor unreachable goals. Reachability alone can keep the agent near familiar states. SUN addresses this tension directly.
  2. The score is simple and interpretable. The successor-value formulation links it to a count-bonus objective. With exact estimates, unreachable goals receive zero score.
  3. Adaptive goal commitment is a useful practical idea. It lets the agent reconsider a goal when estimated reachability drops, instead of staying committed all episode.
  4. The experiments cover several exploration difficulties. Behavioral diagnostics and component ablations help explain the gains. This supports my current weak-acceptance recommendation.
Critical weaknesses
  1. The comparison with additive scores could be stronger. A well-tuned alternative would help show whether multiplication itself explains the gains.
  2. Some gains may come from the surrounding training and exploration choices. More closely matched comparisons would make the goal-selection contribution clearer.
  3. Robustness to imperfect reachability estimates needs more discussion. The results are encouraging, but noisy and stochastic cases remain less clear. These seem like fixable concerns.
Questions for authors
  1. Does SUN still outperform a tuned or normalized additive score under the same training and reselection setup? A comparable tuning budget for DISCOVER would also help.
  2. Appendix F describes training and behavior-policy differences. Which are required by the baselines? Could you also compare methods with only the goal-selection score changed?
  3. When the adaptive check drops a goal, is that usually a useful correction? Could you compare estimated reachability with actual success on a representative noisy task?
4 comments
AuthorsOpenReview ↗

Overview comment

We thank the reviewer for the constructive feedback and the positive assessment of our motivation, the score's simplicity and interpretability, adaptive goal reselection, and the breadth of the experiments. To address the reviewer’s question and provide a more comprehensive analysis , we added a sweep of the novelty coefficient β\beta for DISCOVER and for SUN Additive (Appendix I), and more closely matched comparisons in which the algorithms share the candidate pool and the selection schedule (Appendices J and Q.2). We also added noisy environments (Appendix K), a proposition on goal selection under an inexact learned value (Appendix A.5), and an analysis of every goal reselection against the true value (Appendix L). Below, we will provide the detailed answers to each weakness and question.

AuthorsOpenReview ↗

Response to W1 and Q1

W1. The comparison with additive scores could be stronger. A well-tuned alternative would help show whether multiplication itself explains the gains.

Q1. Does SUN still outperform a tuned or normalized additive score under the same training and reselection setup? A comparable tuning budget for DISCOVER would also help.

We agree, and the revision adds this comparison (Appendix I). We sweep the novelty coefficient β∈{0.1,1,10,100,1000}\beta \in \lbrace 0.1, 1, 10, 100, 1000 \rbrace, for DISCOVER (μ+βσ\mu + \beta\sigma) and for SUN Additive, whose score is now Vθ+βlog⁡Ntot/ngV^\theta + \beta\sqrt{\log N_{\mathrm{tot}}/n_g}. SUN Additive uses the same training and the same adaptive reselection as SUN, and differs only in the score. Table 3 lists the selected value per environment. The results are in Figs. 23 and 24.

  • SUN Additive barely changes for β≥1\beta \ge 1. With its best β\beta it matches SUN in Gridworlds and Classic Control, but in PointMaze-H and AntMaze-H it stays below SUN for every β\beta, by a wide margin in AntMaze-H.
  • DISCOVER needs a large β\beta: its results are nearly identical for β≤1\beta \le 1, improve up to β=100\beta = 100, and change little at 1,000. Even with its best β\beta, it remains well below SUN in MountainCar, LunarLander, Acrobot, and both AntMazes.
  • Across environments, SUN has the highest median entropy and the best worst-case environment in both metrics, against both additive scores at every β\beta (Fig. 24).

So multiplication does not explain the gains in every environment: a well-tuned sum is competitive in the easier ones. What the product adds is that it has no coefficient, whereas the best β\beta ranges from 1 to 1,000 across environments (Table 3), and that it holds up in AntMaze-H, where both sums fall behind.

The main tables and figures still use the hyperparameters of the submission, and we will regenerate all of them with the tuned values of Table 3: the coefficient β\beta for DISCOVER and SUN Additive, and the pseudocount radius ρ\rho for SUN. Appendix I already shows that the conclusion does not change: SUN remains the best algorithm overall (Figs. 23 and 24).

AuthorsOpenReview ↗

Response to W2 and Q2

W2. Some gains may come from the surrounding training and exploration choices... contribution clearer.

Q2. Appendix F describes training and behavior-policy differences ... Could you also compare methods with only the goal-selection score changed?

We would like to clarify that all algorithms share the network architecture (Fig. 18), the hindsight relabeling, the TD(λ\lambda) targets, the replay buffer, the action noise, and every hyperparameter of Table 4. They differ in three respects, and only the first is required by the baselines.

  1. Ensemble of critics. AdaGoal and DISCOVER use four critics, and SUN uses one. This is required: their scores are the mean and the standard deviation of the ensemble.
  2. Episodic selection. AdaGoal and DISCOVER select the goal once per episode among 2,500 candidates, and act randomly after reaching it. This is the strategy of DISCOVER, not a requirement of the scores.
  3. Goal score. AdaGoal selects the goal with the highest ensemble disagreement σ(s0,g)\sigma(s_0, g), DISCOVER with the highest μ(s0,g)+βσ(s0,g)\mu(s_0, g) + \beta\sigma(s_0, g), and SUN with the highest Vθ(st,g)/ngV^\theta(s_t, g)/n_g.

Our ablations isolate the second and the third. The comparison in which only the score changes is SUN Episodic against AdaGoal and DISCOVER. The three share the selection schedule, the candidate pool, and the behavior after reaching the goal, and they differ in the score and in the ensemble that the two baseline scores need. SUN Episodic outperforms AdaGoal in aggregate: its probability of improvement is 0.66 in coverage and 0.74 in entropy, with both 95% intervals above 0.5 (Table 1). The same table gives 0.63 and 0.78 over DISCOVER. All these numbers use the hyperparameters of the submission, and we will recompute them with the tuned ones, but the new experiments show that SUN still outperforms tuned DISCOVER (see our answer to Weakness 1). Appendix J repeats this comparison for three sizes of the candidate pool (Fig. 25). Appendix Q.2 adds a second comparison in which only the score changes: the three algorithms select a new goal as soon as the current one is reached and never abandon it, and SUN keeps the highest coverage in every Gridworld and Classic Control environment. The comparison in which only the selection strategy changes is SUN Episodic against SUN (Fig. 7).

AuthorsOpenReview ↗

Response to W3 and Q3

W3. Robustness to imperfect reachability estimates needs more discussion...These seem like fixable concerns.

Q3. When the adaptive check drops a goal, is that usually a useful correction? Could you compare estimated reachability with actual success on a representative noisy task?

We added the following experiments and analysis in our revision to answer these questions:

Noise (Appendix K). We add three noisy environments, each with noise applied with probability 20% at every step: sticky actions in FourRoomStuck, where the same action is repeated once or twice; random actions in LunarLander; and null actions in AntMaze-H, where the selected action is not applied. FourRoomStuck was already stochastic, with three tiles where the movement is random with probability 50%. In LunarLander and AntMaze-H, the noise costs every algorithm part of its coverage and entropy, SUN included. The ranking does not change: SUN attains the highest coverage and entropy with and without noise (Fig. 26).

Imperfect reachability. 1) We provide a new proposition (Appendix A.5) states that how much error of the learned value the selection tolerates: no unreachable goal is selected while the error stays below a bound that tightens as unreachable goals become rarer than reachable ones. 2) We also verified this with additional experiments: in the Gridworlds, where reachability is known exactly, all SUN variants select an unreachable goal in less than 7% of their selections on average, and almost only early in training, against about 55% for Novelty-Only, 33% for AdaGoal, and 14% for DISCOVER (Appendix L, Fig. 27). In ThreeRoom, the adaptive check replaces 83% of the goals that were unreachable when selected, a median of 4 steps after selecting them (68% in FourRoomStuck, and 63% with sticky actions; Appendix L, Fig. 28b).

Adaptive check (Appendix L). When the check drops a goal, it is mostly a useful correction, and the revision measures it. In ThreeRoom and in FourRoomStuck, with and without sticky actions, we compute the true visit value by value iteration, compare it with the learned value at every check, and assign a cause to every reselection (Fig. 28).

  • Reselections are rare: they occur in 0.4% to 2.2% of the training steps.
  • In ThreeRoom, 92% of them discard a goal that was unreachable when it was selected, and they stop once the critic is accurate.
  • With sticky actions, which is the noisy task, most reselections follow a true drop of the value caused by the environment: the check does what it is meant to do.
  • In FourRoomStuck without sticky actions, about half are not useful: the true value had not dropped, and the critic estimated a drop. We traced them in the deterministic versions of the environments: in most of them the agent was one step closer to the goal, where the true value differs by about 1% (Fig. 29).

A reselection of the last kind is less harmful than a wrong goal, since the agent only replaces a valid goal with a new one and keeps exploring. Appendix A.8 gives the matching analysis: the check is guaranteed to fire once a goal becomes unreachable if the critic error is below half the value of the goal at selection, but it can fire by mistake in the first steps after a selection.

Reviewer_E6SjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an exploration strategy, SUN, that selects exploration goals from replay by multiplying learned discounted reachability by an inverse visit pseudocount (proxy for novelty). It reconsiders a goal when its estimated value falls below its value at the selection state. Cached neighborhood counts make candidate-count queries constant-time, while insertion costs scale with buffer size and goal dimensionality. The paper establishes properties under exact-value assumptions and evaluates coverage and visitation entropy on gridworlds, classic control, and goal-conditioned control tasks. The combination of reachability-aware scoring, adaptive goal selection, and cached counts is a useful practical contribution. Experiments use 20 seeds and report improvements over AdaGoal, DISCOVER, and random exploration. The targeted tasks and ablations support a practical contribution within low-dimensional state and action spaces. However, generalizability to harder manipulation and image-based tasks is untested.

Critical strengths
  1. The evaluation targets the proposed failure cases.
  • Disconnected rooms, irreversible transitions, and narrow passages directly test the usefulness of reachability-aware exploration. (lines 298–311).
  1. The proposed implementation is practical, and its limitations are explicitly discussed. The method offers a usable contribution with clearly identified boundaries. Disclosure does not resolve the limitations, but helps assess where the method is likely to be useful.
  • The product score separates value learning from novelty estimation; the count procedure describes standardization and eviction.
  • Implementations for both discrete and continuous-action spaces have been presented showing generalization.
  • The authors explicitly acknowledge limitations that count-based novelty becomes uninformative for image observations, the score ignores states traversed en route, and replay-based selection cannot propose unseen goals.
Critical weaknesses
  1. The comparisons do not isolate the advantage over tuned alternatives. The results support the tested system, but do not cleanly establish which design choice causes the advantage.
  • Appendix F states, “Hyperparameters have not been tuned, with the exception of the pseudocount radius”. Yet DISCOVER's weaker performance is partly attributed to its coefficient. (Appendix F, lines 1194–1209).
  • The authors explicitly call the additive control variant as “the unscaled one - the regime Remark 3 predicts should fail.” Its poor result therefore does not rule out a tuned additive score. (Appendix K.6, lines 2048–2051)
  • Episodic selection uses random actions after success, while adaptive SUN selects another goal. Candidate pools also differ: 2,500 versus 256. These are additional differences beyond abandoning unfinished goals.
  1. The evaluation is strong within its scope, but does not establish performance gains in hard-exploration or manipulation tasks. These concerns limit generality.
  • There are no empirical results on hard-exploration Atari, Fetch, or Adroit tasks.
  • AntMaze uses planar position goals, and ArmPush-H uses the cube's planar position. These tests do not establish effectiveness on tasks with dexterous manipulation, richer goal representations, or pixel inputs.
Questions for authors
  1. Does SUN retain its advantage under comparable coefficient-tuning budgets for DISCOVER and a scaled or normalized additive baseline?
  2. What happens when commitment variants share candidate-pool sizes and post-success reselection, differing only in whether they abandon unfinished goals?
3 comments
AuthorsOpenReview ↗

Overview comment and response to W1, Q1, Q2 (part 1)

We thank the reviewer for the constructive feedback and for recognizing SUN as a practical contribution whose evaluation targets the failure cases it is designed for. In response, we swept the novelty coefficient of DISCOVER and SUN Additive (Appendix I), varied the candidate pool (Appendix J), ran the comparison requested in Question 2 (Appendix Q.2), and added HandReach, a task with the Shadow Dexterous Hand and a 15-dimensional goal (Appendix Q.3). Below, we discuss these results in detail.

Weakness 1 and Questions 1 and 2

W1. The comparisons do not isolate the advantage over tuned alternatives ... additional differences beyond abandoning unfinished goals.

Q1. Does SUN retain its advantage under comparable coefficient-tuning budgets for DISCOVER and a scaled or normalized additive baseline?

Q2. What happens when commitment variants share candidate-pool sizes and post-success reselection, differing only in whether they abandon unfinished goals?

We agree with the reviewer that the submission did not separate the choice of the score from an untuned coefficient, and that the episodic variants differ from SUN in more than abandoning goals. The revision addresses both points.

Tuned coefficients. Appendix I sweeps the novelty coefficient β∈{0.1,1,10,100,1000}\beta \in \lbrace 0.1, 1, 10, 100, 1000 \rbrace, for DISCOVER (μ+βσ\mu + \beta\sigma) and for SUN Additive, which now has a coefficient as well (Vθ+βlog⁡Ntot/ngV^\theta + \beta\sqrt{\log N_{\mathrm{tot}}/n_g}). Table 3 lists the selected value per environment. The tuning budgets are thus comparable: five values of β\beta for each additive score, and five values of the pseudocount radius ρ\rho for SUN (Appendix H). The results are in Figs. 23 and 24.

  • DISCOVER needs a large β\beta: its results are nearly identical for β≤1\beta \le 1, improve up to β=100\beta = 100, and change little at 1,000. Even with its best β\beta, it remains well below SUN in MountainCar, LunarLander, Acrobot, and both AntMazes.
  • SUN Additive is less sensitive: its results barely change for β≥1\beta \ge 1. With its best β\beta it matches SUN in Gridworlds and Classic Control, but in PointMaze-H and AntMaze-H it stays below SUN for every β\beta, by a wide margin in AntMaze-H.
  • Across environments, SUN has the highest median entropy and the best worst-case environment in both metrics, against both additive scores at every β\beta (Fig. 24).

What the sweep shows. A tuned additive score therefore matches SUN in the easier environments, but not across all of them. The sentence you quote compared SUN with a single additive score without a coefficient, and this sweep replaces it. The sweep confirms the problem of scale stated in the paper (Section 3.1 and Remark 3): an additive score needs a coefficient, the best β\beta ranges from 1 to 1,000 across environments (Table 3), and a wrong value is costly: with β=0.1\beta = 0.1, SUN Additive explores no better than Reachability-Only in three of the mazes (Fig. 23a). The product has no coefficient.

Note on the main results. The main tables and figures still use the hyperparameters of the submission, and we will regenerate all of them with the tuned values of Table 3: the coefficient β\beta for DISCOVER and SUN Additive, and the pseudocount radius ρ\rho for SUN. Appendix I already shows that the conclusion does not change: SUN remains the best algorithm overall (Figs. 23 and 24).

AuthorsOpenReview ↗

Overview comment and response to W1, Q1, Q2 (part 2)

Episodic against adaptive selection. We conducted the following experiments based on the reviewer's suggestions (two differences besides abandoning goals).

  • Candidate pool. Appendix J reruns the three episodic algorithms with pools of 256, 2,500, and 10,000 candidates (Fig. 25). SUN Episodic is nearly insensitive to the pool size, so with the 256 candidates of SUN it performs about as with 2,500, and the pool does not explain the gap between the two. AdaGoal and DISCOVER lose performance with 256 candidates in most environments and gain little from 10,000, so the 2,500 candidates of the main experiments (Appendix F) do not disadvantage them.
  • Random actions after success. This is the episodic strategy of DISCOVER. SUN Episodic adopts it so that it can be compared with AdaGoal and DISCOVER under the same selection strategy.

The reviewer suggested goal reselection variant (Appendix Q.2). We ran it for SUN, AdaGoal, and DISCOVER: the agent selects a new goal as soon as the current one is reached, never abandons a goal before reaching it, and selects among 2,500 candidates. The figure shows each algorithm next to its episodic version, and next to the adaptive SUN of the main experiments.

  • With respect to its episodic version, SUN is nearly unaffected: the two overlap in almost every environment. Selecting a new goal after success gives a slightly higher entropy in FourRoomStuck and a faster start in Acrobot, and a lower entropy in Pendulum.
  • Adaptive SUN is the best of the three SUN versions in every environment except Pendulum, for the reasons already discussed in Section 4.1.
  • The results of AdaGoal and DISCOVER are mixed. Both improve in Acrobot, DISCOVER by a wide margin, and DISCOVER also attains a higher entropy in FourRoomStuck and CartPole. AdaGoal instead attains a lower entropy in FourRoomStuck and GridMaze, and DISCOVER in ThreeRoom and GridMaze.
  • With any strategy, SUN attains the highest coverage in every environment, and the highest entropy in all but Pendulum and CartPole, where AdaGoal is on par or higher.

Two conclusions follow:

  • First, when the three algorithms share the selection strategy and the candidate pool and differ only in the score, SUN keeps its advantage, with either strategy.
  • Second, the random actions after success do not explain the difference between SUN and SUN Episodic: the variant that selects a new goal after success behaves like SUN Episodic. The difference, a faster start in MountainCar and Acrobot and a higher coverage in LunarLander, appears only with adaptive SUN, which also abandons goals. The clearest example is MountainCar: selecting a new goal after success slightly improves the coverage of all three algorithms, but none of them explores as fast as adaptive SUN. At the end of training, the three versions are otherwise close, in agreement with Appendix L, where the adaptive check replaces the goal in 0.4% to 2.2% of the training steps. Adaptive SUN also selects among fewer candidates (256), but the pool has little effect on SUN Episodic (Appendix J).
  • We did not have enough time to complete this full experiment in the GCRL Control tasks, and we will add it as soon as possible.
AuthorsOpenReview ↗

Response to W2

W2. The evaluation is strong within its scope, but does not establish performance gains ... richer goal representations, or pixel inputs.

To provide the higher dimensional hard-exploration task analysis, in the revision we added a dexterous manipulation task with a high-dimensional goal (Appendix Q.3): HandReach, with the Shadow Dexterous Hand, from the suite that also contains the Fetch tasks (Plappert et al., 2018). The action has 20 dimensions, and the goal is the position of the five fingertips, 15 dimensions in total. After 5M steps and over 10 seeds, SUN has visited 59% of the fingertip cells, against 52% for DISCOVER, 45% for AdaGoal, and 21% for Random. Its entropy ends about 7 nats above AdaGoal and 8 nats above DISCOVER (Figs. 47 and 48).

One finding from this high dimensional experiment is that, in 15 dimensions the fixed-radius pseudocount leaves about one candidate out of five with a count of one, so it cannot rank the most novel candidates. For those candidates only, we refine the count with a kk-nearest-neighbor density estimate of the same quantity (Eq. 36). Insertion is unchanged.

The goal spaces of the paper now range from 2 to 15 dimensions: besides the planar goals you mention, LunarLander(Full) has an 8-dimensional goal with unbounded and binary features.

Pixel inputs, and with them hard-exploration Atari, remain outside the scope. As stated in Section 5, count-based novelty is uninformative for images, and combining the reachability factor of SUN with a learned novelty signal is future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful and constructive reviews. We are glad that they found the SUN score simple and well motivated, and that they consider the combination of reachability-aware scoring, adaptive goal selection, and cached counts a useful practical contribution, supported by informative ablations. We appreciate their suggestions, which helped us improve the paper.

We have uploaded a revised version, and this comment lists what changed with respect to the submission. The details are in our replies to each reviewer. Section, appendix, and figure numbers refer to the revised version.

New experiments

  • Novelty coefficient (Appendix I). We sweep the coefficient β\beta over five values for DISCOVER and for SUN Additive, which now has a coefficient as well.
  • Candidate pool (Appendix J). We rerun the episodic algorithms (SUN Episodic, AdaGoal, DISCOVER) with 256, 2,500, and 10,000 candidates.
  • Noisy environments (Appendix K). We add sticky actions to FourRoomStuck, random actions to LunarLander, and random null actions to AntMaze-H.
  • Goal selection and reselection (Section 4.2 and Fig. 8; Appendix L). Using the true value in the Gridworlds, we measure how often each algorithm selects unreachable goals, and we assign a cause to every goal reselection of SUN.
  • Downstream performance (Appendix Q.1). We evaluate the agents on the tasks themselves, while exploring with SUN and the baselines.
  • Selecting a new goal only after success (Appendix Q.2). As requested by Reviewer E6Sj, all algorithms select a new goal as soon as the current one is reached, never abandon a goal before reaching it, and share the same candidate pool.
  • High-dimensional manipulation (Appendix Q.3). We add HandReach, a dexterous manipulation task with a 15-dimensional goal and a 20-dimensional action.

New analysis

  • Learned values (Appendix A.5). A new Proposition 3 gives the error of the learned value under which unreachable goals are still rejected, and bounds the loss in score of the selected goal.
  • Adaptive check (Appendix A.8). We state when the check is guaranteed to fire, and when it can fire by mistake, under critic error.
  • Scope of the theory (Appendix A). Each result now states the conditions it uses. The count-bonus equivalence is renamed count-scaled value and presented as an identity, and the two oracle statements are relabeled as propositions.
  • Pseudocount and dimension (Appendices B and H). We discuss why the radius must grow with the dimension of the goal space.

Report

  • Aggregate statistics (Fig. 9 and Table 1; Appendix N). We now follow the protocol of Agarwal et al. (2021). The main text reports the interquartile mean and the probability of improvement with stratified bootstrap intervals, in place of the relative improvements of the submission. The appendix adds the median, the mean, the optimality gap, and the scores per environment.
  • Hyperparameters (Appendix F). Table 3 lists the tuned values per environment: the coefficient β\beta for DISCOVER and SUN Additive, and the pseudocount radius ρ\rho for SUN, following the new sweeps.

In addition to this round's revision, we will regenerate all tables and figures with the tuned values of Table 3: the coefficient β\beta for DISCOVER and SUN Additive, and the pseudocount radius ρ\rho for SUN. We will also reorganize the experiment to fit the high-dimensional HandReach with the Shadow Dexterous Hand in the main body.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.