acceptodds
Under review as a conference paper at ICLR 2027

Mastering Atari 2600 Games With Discovered Options

15 views

Abstract

Temporal abstractions, often instantiated as options, have long been regarded as a mechanism for accelerating credit assignment, facilitating exploration, and enabling generalisation in reinforcement learning (RL). However, developing general option discovery methods that are effective in large-scale, high-dimensional domains remains a fundamental challenge. Existing option discovery methods are either confined to relatively simple domains, depend on handcrafted or quasi-symbolic representations, or offer little improvement over learning without options. We present Wayfarer, a general, domain-agnostic, online deep RL agent that discovers options through Laplacian representation learning from high-dimensional observations and leverages them for control. We show that the resulting options simultaneously improve exploration, accelerate credit assignment, and generalise effectively to unseen settings, enabling substantially faster learning of complex policies. Wayfarer achieves state-of-the-art performance among single-stream agents on the most challenging Atari 2600 games, with the largest gains in games that require long-horizon exploration and strategic behaviour, such as Montezuma's Revenge and Private Eye.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, representation learning, option discovery, Laplacian representation, agent-centric representation, Atari

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_vmFBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Wayfarer is an online reinforcement learning system for hard-exploration Atari games that jointly trains three components from a single pixel-based experience stream: (1) an inverse-dynamics-regularised Laplacian representation, (2) intrinsic-reward option policies derived from the Laplacian eigenfunctions, and (3) an IQN-based high-level controller that selects between primitive actions and the discovered options. Rather than introducing a fundamentally new option-discovery objective, the paper's primary contribution is the successful online integration of Laplacian option discovery and exploitation directly from pixels without relying on a pre-collected dataset. The authors validate the system on an established subset of hard-exploration Atari games across five seeds, demonstrating substantial performance gains on several benchmarks (such as Montezuma's Revenge, Private Eye, and Venture) alongside qualitative analyses of representation learning, option behavior, and ablations.

Critical strengths

Effective Online Integration of Option Discovery and Control from Pixels: Discovering and learning when to exploit options online from a single stream of pixel observations—without assuming a pre-collected dataset that already covers the state space—is an important and difficult challenge. Extending beyond random exploratory option execution by jointly learning option selection, alongside motivating the representation via an inverse-dynamics auxiliary task to mitigate uncontrollable features, represents a valuable empirical integration.

Compelling Empirical Gains on Established Hard-Exploration Benchmarks: The system achieves striking improvements on notoriously difficult games within an established benchmark subset (evaluated across five seeds with uncertainty estimates and transparent disclosure of failures). In Table 4, Wayfarer achieves mean returns of approximately 4,187 on Montezuma’s Revenge (vs. 0 for IQN and 1,829 for DreamerV3) and 34,418 on Private Eye (vs. 180 for IQN), demonstrating that the discovered options provide substantial practical leverage in hard-exploration settings.

Critical weaknesses

Mismatch in the Option-Value Backup Target and Unsubstantiated Credit-Assignment Claims: Section 3.1 uses a fixed three-step target for an option's initiation value, bootstrapping over all actions and options regardless of whether the active option has terminated, despite execution following call-and-return semantics. With termination probability p = 0.2, the expected duration is E[T] = 5 and Pr(T > 3) = 0.8^3 = 0.512, meaning that in roughly half of nominal executions the bootstrap assumes a fresh high-level choice while execution remains committed to the option. This is not a valid Bellman target for the option-return random variable, directly undermining the headline claim of accelerated temporal credit assignment. Furthermore, the experimental ablations do not cleanly isolate credit assignment: the comparison against "no option values" conflates value learning with the exploration mechanism (NoisyNet vs. epsilon-greedy), and the epsilon-greedy-matched comparison ("no noisy networks" vs. "no option values") shows only a slight advantage that cannot separate faster reward propagation from altered exploration data distribution.

Overstated Claims Regarding Systematic Generalisation, Exploration, and SOTA Comparability: Several headline claims exceed what the experimental evidence supports:

  1. Exploration and Generalisation: Qualitative exploration rollouts disable option termination, demonstrating persistent skill capacity rather than actual training-time trajectories under the 5-step mean duration. Similarly, transferring two selected options from a single checkpoint to unseen rooms is anecdotal and does not establish systematic generalisation across options, seeds, or starting states.

  2. Baseline Comparability and Missing Full-Budget Control: Protocol differences (sticky actions, action sets, no-op starts, life information) complicate external baseline comparisons, making the internal "no options" ablation critical—yet this control is only evaluated with 3 seeds at 100M frames rather than the headline 5 seeds at 200M frames. Moreover, severe task-level failures on Beam Rider (5,077 vs. 18,830 for IQN), Skiing (-29,709 vs. -17,098 random), and Pitfall! (0 return) show that framing the work as "Mastering" the benchmark or claiming broad single-stream SOTA is stronger than warranted.

Questions for authors

Option-Value Target Formulation: Can you formally define what the fixed 3-step surrogate target represents relative to the true call-and-return SMDP target (the discounted sum of rewards over T steps plus gamma^T times max_o' Q(x_{t+T}, o')), characterize the resulting bias when Pr(T > 3) = 0.512, and clarify whether a duration-consistent or termination-aware intra-option backup was tested?

Full-Budget Internal Control: Do you have, or can you provide, results for the "no options" ablation at the full 200M-frame budget across 5 seeds to give a strictly protocol-matched baseline for the headline results?

Mathematical & Algorithmic Specifications:

  1. In Section 2, symmetry of f(P^pi) alone does not guarantee that I - f(P^pi) is positive semidefinite or has a constant first eigenfunction. What is the exact empirical Laplacian operator and sampling measure/inner product used in ALLO?

  2. The skill reward telescopes to u(x_{t+n}) - u(x_t) only as an undiscounted sum. When the discount factor gamma < 1, what is the exact skill Bellman target and intended objective?

Computational Cost & Reproducibility: What are the parameter counts, hardware requirements, wall-clock training times, and per-cycle update costs (given three encoders, ten skill heads, and batch size 512) compared to standard IQN? Additionally, please specify the exact network dimensions, LayerNorm placement, terminal/truncation handling, and how the single hyperparameter configuration was tuned/selected.

5 comments
AuthorsOpenReview ↗

Response to weaknesses

First of all, we thank the reviewer for their detailed review. We are glad they see online option discovery and control as “an important and difficult challenge”. In particular, we found their questions about the option-value target and empirical Laplacian operator very useful: they led to some important clarifications that have made the paper stronger.

“Mismatch in the Option-Value Backup Target”

Our option-value target is intentionally optimistic and not the call-and-return SMDP target. Please see below for details (as part of our response to your question about the target formulation).

“Unsubstantiated Credit-Assignment Claims…”

The slightly better performance of “no noisy networks” is evidence consistent with our claim of accelerated credit assignment. Note that the ablated variant “no option values” includes noisy layers in the main network and ϵ\epsilon-greedy for exploration with options. This means that the “no option values” agent explores not only using randomly selected options, with ϵ\epsilon-greedy, but also using the noisy value predictions over its primitive actions. Consequently, “no option values”, if anything, likely has an exploration advantage compared to “no noisy networks”, yet “no noisy networks” shows a slight advantage. We thank the reviewer for this criticism, which highlighted the need for us to clarify our text; we have done so in Section 4.2 (lines 448-451 and lines 513-516).

The above said, we agree that our ablations cannot cleanly separate faster reward propagation from changes in the collected data. In Wayfarer, the two are intertwined: options change both what is experienced and which decisions receive credit. The mechanism we propose accelerates credit assignment is that our option trajectories travel more directly to significant events than dithering with primitive actions (e.g., Figure 1), so fewer nn-step backups are needed to credit the decisions that led there. We have made our hypothesis clear in Section 3.1.

“Exploration and Generalisation…”

Yes, the reviewer is right: these figures show the options’ learned policies with termination disabled. Termination in Wayfarer is a fixed probability rather than something the agent learns, so disabling it isolates the part the agent does learn, the option’s policy, and shows it over a longer horizon. This was previously stated in Section 4.2, and we have also now made it explicit in each of the figure captions in Section 4.2 (Figures 4-6). As for training-time behaviours, the option-selection rates in Appendix C.2, measured with termination enabled, exemplify how the agent uses options during training in Montezuma’s Revenge: the agent favours options at the frontier of its experience, and primitive actions once a region (i.e., new room) becomes familiar.

As Section 4.2 states, these figures provide anecdotal evidence. As such, we have tried to provide a diverse range of examples from different games (Montezuma’s Revenge, Private Eye, Pitfall!), across options (Figures 1, 5, 6), across different points in training (Figure 4), and across starting states, including states that the agent has not visited yet (Figure 6). They are existence proofs: they show what options discovered online by Wayfarer can do, how varied they are, and how they develop over training, but being agnostic to how often they occur during training.

“Baseline Comparability and Missing Full-Budget Control”

To further support that discovered options improve the agent’s performance, we have added a full-budget control to the main results (Figure 3, Section 4), “Rainbow (IQN)”, evaluated with 5 seeds at the full budget of 200M frames, as requested by the reviewer. This variant corresponds to the “no options” configuration in our ablations: Wayfarer’s main network alone, acting over primitive actions, without the representation and skill networks. It controls for our changes, and thus separates the contribution of discovering and learning to use options via Wayfarer. The results further support our claims: the full Wayfarer agent is clearly an improvement in aggregate (2.15x median performance, 1.24x IQM, 1.11x mean, and 1.12x optimality gap).

“‘Mastering’ the benchmark or claiming broad single-stream SOTA is stronger than warranted.”

We used the word “mastering” similarly to how the DreamerV3 paper [66] used it, referring to aggregate metrics instead of success in every single game. DreamerV3 also failed to outperform baselines in some Atari 2600 games. To be clear, our use of the word mastery refers to achieving single-stream SotA in aggregate human-normalised performance on the challenging set of Atari 2600 games. On the individual failures: some of these may be attributable to specific design choices: for example, in Skiing, Wayfarer outperforms all other agents when using reward clipping instead of value rescaling. We have added this result to the paper (Appendix I).

AuthorsOpenReview ↗

Response to questions (1/2)

“Option-Value Target Formulation: Can you formally define what the fixed 3-step surrogate target represents relative to the true call-and-return SMDP target (the discounted sum of rewards over T steps plus gamma^T times max_o' Q(x_{t+T}, o')), characterize the resulting bias when Pr(T > 3) = 0.512, and clarify whether a duration-consistent or termination-aware intra-option backup was tested?”

We thank the reviewer for this analysis, which has shown us that the paper does not explain an important choice. The reviewer is right that our target is not the call-and-return target: it values an option as if the agent could re-decide nn steps after initiating it, which makes the option-value target optimistic. We use it because the options keep changing with the representation: we let option values guide initiation and credit the option’s later steps to the primitive actions stored at those steps, distilling option behaviour into primitive-action values over training (as explained in Section 3.1). In our preliminary experiments, intra-option backups drove the option-selection rate down, though these experiments were too limited to draw any conclusions from. We have revised Section 3.1 accordingly, and added the characterisation of the bias below to Appendix E.

Ignoring hh and writing Q(x,o):=Eτ∼U[0,1][Z(x,o,τ)]Q(x,o) := \mathbb{E}_{\tau\sim U[0,1]}\big[ Z\left( x,o,\tau \right) \big]: option values are trained only at initiation, and their target truncates the call-and-return SMDP target at nn steps. In the idealised setting in which the online and target networks are the same, so the double-Q bootstrap reduces to a max⁡\max, and experience replayed follows the current policy, the two differ only if an option is still running at t+nt+n, where we bootstrap from the greedy option at xt+nx_{t+n}. Because termination is memoryless (Pterm=0.2P_{\mathrm{term}}=0.2 at every step), continuing oo is worth the same as re-initiating it at xt+nx_{t+n}, which is one term of the max⁡\max. Our target therefore corresponds to executing options with interruption at the nn-step boundary, in the sense of Sutton et al. (1999) [2], Section 4, and it is optimistic relative to call-and-return. Per backup, the bias from the initiating option is Pr⁡(T>n),γn,E[max⁡o′Qcr(xt+n,o′)−Qcr(xt+n,o),∣,T>n]≥0\Pr(T > n)\\,\gamma^n\\,\mathbb{E}\big[\max_{o'}Q_{\mathrm{cr}}(x_{t+n},o')-Q_{\mathrm{cr}}(x_{t+n},o)\\,\big|\\,T > n\big]\ge 0, with Pr⁡(T>3)=0.512\Pr(T > 3)=0.512, that is, 0.512,γ30.512\\,\gamma^3 times the expected advantage of re-deciding over continuing oo at xt+3x_{t+3}.

“Full-Budget Internal Control: Do you have, or can you provide, results for the ‘no options’ ablation at the full 200M-frame budget across 5 seeds to give a strictly protocol-matched baseline for the headline results?”

Yes, we have added the control, “Rainbow (IQN)”, to the main results (Figure 3, Section 4; per-game results in Appendix H.1), evaluated with 5 seeds at the full budget of 200M frames. These results further support our claim that discovered options improve the agent’s performance: Wayfarer achieves 2.15x median performance, 1.24x IQM, 1.11x mean, and 1.12x optimality gap, compared to this baseline.

“In Section 2, symmetry of f(P^pi) alone does not guarantee that I - f(P^pi) is positive semidefinite or has a constant first eigenfunction. What is the exact empirical Laplacian operator and sampling measure/inner product used in ALLO?”

Again, we thank the reviewer for highlighting a detail that is important to clarify. In particular, we have fixed the sentence in Section 2, which previously implied that the symmetry of f(Pπ)f(P_\pi) alone would guarantee that I−f(Pπ)I - f(P_\pi) is PSD with a constant first eigenfunction. We have also elaborated on the empirical Laplacian operator and inner product used in ALLO (Appendix F with a forward-reference in Section 3.3).

Concretely, ALLO learns from samples and never forms f(Pπ)f(P_\pi): it solves min⁡u∑i12E(x,x′)∼Prep[(ui(x)−ui(x′))2]\min_u\sum_i\tfrac12\mathbb{E}_{(x,x')\sim P_{\mathrm{rep}}}[(u_i(x)-u_i(x'))^2] s.t. ⟨uj,uk⟩ρ:=Ex∼ρ[uj(x)uk(x)]=δjk\langle u_j,u_k\rangle_\rho:=\mathbb{E}_{x\sim\rho}[u_j(x)u_k(x)]=\delta_{jk}, where PrepP_{\mathrm{rep}} is the distribution of replayed pairs (x,x′)=(xt,xt+Δ)(x,x')=(x_t,x_{t+\Delta}) from Section 3.3 and ρ\rho, the distribution of xx, is uniform over the replay buffer. The empirical operator is thus L=diag(ρ)−1(D−W)L=\mathrm{diag}(\rho)^{-1}(\mathbf{D}-\mathbf{W}) with W=12(Prep+Prep⊤)\mathbf{W}=\tfrac12(P_{\mathrm{rep}}+P_{\mathrm{rep}}^\top) and D=diag(W1)\mathbf{D}=\mathrm{diag}(\mathbf{W}\mathbf{1}), which for any PrepP_{\mathrm{rep}} is self-adjoint and PSD on L2(ρ)L^2(\rho) with a constant first eigenfunction (L1=0L\mathbf{1}=0), and equals I−f(Pπ)I-f(P_\pi) when ρ\rho is stationary (the Laplacian of 33; the form given in Section 2).

AuthorsOpenReview ↗

Response to questions (2/2)

“The skill reward telescopes to u(x_{t+n}) - u(x_t) only as an undiscounted sum. When the discount factor gamma < 1, what is the exact skill Bellman target and intended objective?”

We do not discount the intrinsic rewards within each nn-step segment. The intrinsic nn-step return at time tt is an undiscounted sum of one-step eigenfunction changes over the nn-step horizon, from tt to t+nt+n, Gtn,i±:=ri±(xt,xt+n)=±(ui(xt+n)−ui(xt))G_t^{n,i\pm} := r^{i\pm}(x_t,x_{t+n}) = \pm\big(u_i(x_{t+n})-u_i(x_t)\big) (Section 3.2), taking the place of the discounted task return GtnG_t^n. Otherwise, the skill network uses a target of the same form as the main network, without hh and h−1h^{-1} (as ALLO’s orthonormality constraints keep the eigenfunctions, and hence the intrinsic rewards, on a similar scale across environments): yji±=Gtn,i±+γn,Zω−i±(xt+n,a′,τj′)y_j^{i\pm} = G_t^{n,i\pm} + \gamma^n\\, Z^{i\pm}_{\omega^-}(x_{t+n}, a', \tau'_j), with a′a' the greedy primitive action under the online skill network (parameterised by ω\omega). So, discounting applies across nn-step segments and not within them, and the objective is to seek and stay in regions where uiu_i is high (for ++) or low (for −-). We have clarified this by adding the target to Section 3.2.

“What are the parameter counts, hardware requirements…”

With the current configuration, Wayfarer has approximately 43.7M trainable parameters: 2.6M in the representation network, 34.3M in the skill network, and 6.7M in the main network. As the reviewer anticipates, most of the parameters lie in the option heads, at approximately 3.4M each, whereas the encoders have only 0.08M each. Our internal control (Wayfarer’s main network alone, our closest variant to a standard IQN agent) has 6.7M. For reference, DreamerV3, whose aggregate performance Wayfarer matches, uses a 200M-parameter model for Atari 2600 games, about 4.6x Wayfarer’s size [66, extended data table 2].

A Wayfarer run fits on a single L40S (48 GB) or a 40 GB H100 MIG instance. Wayfarer needs <10{<}10\\% more GPU memory than the internal control, because most of the memory requirement comes from the experience replay buffer, which both agents use and which we keep on the GPU. Wayfarer’s buffer holds 10610^6 steps; for reference, DreamerV3’s holds 5×1065{\times}10^6 with additional latent states stored [66, extended data table 5].

Training for 200M frames takes about nine days on an H100 MIG instance, versus about two days for the internal control (about 4.5x overall). Wayfarer's cost could be reduced in multiple ways: eliminating IQN and dueling from the option heads (which alone would cut the parameter count from 43.7M to 25.6M); sharing one encoder across the three networks; lowering the representation network batch size; updating fewer options at every step—for example, selecting one randomly in the style of DCEO [26]—or updating them with a shared batch of data; and replacing the per-option head in the skill network with a single option-conditioned head. We leave this for future work.

This information has been added to Appendix M.

“Additionally, please specify the exact network dimensions…”

  • The exact network dimensions and LayerNorm placement have been added to a table Appendix B, and we will make the code available upon the paper publication. Every convolutional and hidden fully connected layer is followed by LayerNorm, and then a ReLU. Output layers and IQN quantile embeddings have no LayerNorm.
  • Terminal/truncation handling: Main and skill networks: the nn-step return is truncated alongside the bootstrap. The representation network does not sample (xt,xt+Δ)(x_t,x_{t+\Delta}) across episodes.
  • We drew most of our initial hyperparameters from previous work [26, 44, 51, 65]. We refined these hyperparameters and set the remaining ones using preliminary, non-exhaustive sweeps on a few challenging set games. In these sweeps, we observed Wayfarer to be relatively insensitive to several design choices, such as the option termination probability (though we did not test extremely long options), the discounting in P(Δ)P(\Delta), and whether the agent-centric Laplacian representation was learned via a single network (regularised for control-relevant features) or decoupled into separate inverse-dynamics and Laplacian networks. We chose the number of options weighing the cost of training more options. This information has been added to Appendix D.

We hope our clarifications, additional results, and changes to the paper have addressed the reviewer’s main concerns. We are very grateful for the reviewer’s feedback. If the main concerns have been addressed, we would be grateful if the reviewer could increase their score.

Reviewer_EMhoOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Wayfarer, a hierarchical reinforcement learning algorithm where options are the policies which climb the eigenfunctions of the (state-)graph Laplacian. The ALLO graph-drawing objective is used, with the key addition of regularizing the latent representation used for the graph-drawing objective using multi-step inverse dynamic prediction in order to suppress from the latent input information which cannot be controlled by the agent. Building on top of a base of Rainbow+IQN, they show State of the Art results (on 200M environment frames budget) on Private Eye and Venture specifically and the Atari 2600 challenging set more generally. Through ablation studies and specific examples, they convincingly argue that the Laplacian-based options help exploration, credit assignment, and generalization.

Critical strengths
  1. Proposal of a novel algorithm – Wayfarer – which can learn Laplacian-eigen-options from high-dimensional observations and effectively use those options for exploration, credit assignment, and generalization. A non-trivial achievement.

  2. SotA performance of that algorithm at 200M environment frames on Private Eye and Venture, hard exploration environments.

Critical weaknesses
  • Some statements in the paper are mistaken, false, or misleading. Please refer to the corrections section below.
Questions for authors

Corrections:

  1. Misleading; lines 70-71: "Moreover, the representation is typically learned offline, from a pre-collected dataset". This is highly misleading or even false, of the literature cited by the paper itself, [16,18,20,22,26,78] are learned online and [19] is compatible with online learning.
  2. Missing; lines 66-67: "Laplacian-derived options [15–25]" should also cite [26,78] which also have Laplacian-derived options.
  3. Missing; lines 68-70:
    1. "The Laplacian representation has been used to facilitate exploration [15–20]", here [26, 78] should be added to the citation.
    2. credit assignment [21, 22], here [78] should be added to the citation.
  4. False; lines 71-72: "The exception that learns it online", this is entirely false, as mentioned above in point 1, many methods learn online.
  5. Overstated; lines 72-75: "Indeed, to our knowledge, no option-discovery method—whether grounded in reward, in given subgoals, or in the structure of the environment itself—has been shown to be able to effectively tackle exploration, credit assignment, and generalisation at once when processing high-dimensional observations", for the reasons mentioned above this is an overstatement which needs to be corrected.
  6. Misleading; line 75: "let alone doing so online, from a single stream of experience". This is very misleading: 1) as previously mentioned many methods learn online 2) there is nothing in the other methods which are reliant on multiple streams of experience; when other methods use parallel environments it is simply to make there experiments go faster but all methods would work equally well with a single stream of experience.
  7. Naming Error; lines 1314-1319: HNEO should actually be called SHNEO here (Scalable Hierarchical Novel Eigen Options), though this is an understandable mistake since the authors of SHNEO themselves mistakenly used "HNEO" in their own article.
  8. Misleading; 1318-1318: "HNEO builds on PQN [79] and learns from many parallel instances of the environment, whereas Wayfarer learns from a single stream of experience." While this is technically true, this is misleading: 1) as they show in their appendix F, SHNEO works just as well with DQN on a single stream of experience as with PQN and 2) in the methods cited generally, a single stream of experience vs parallel environments is not a significant difference; the methods can be run with a single stream of experience if needed by replacing parallel environments with a replay buffer like is done in Wayfarer. To say that this is a meaningful distinction is extremely misleading.

If the above are corrected, I would be very happy to increase my score for what is otherwise an excellent and impressive article.

Clarification Requests:

  • There could be more explanations as to why the IQN ZZ are transformed with hh and why does hh take this form? What happens if the ZZ 's are not transformed with hh? (line 208)
  • line 296: "applies a symmetric log transformation to the representation outputs." This is not clear, please clarify (i.e. please give equations and where this fits in the algorithm otherwise this creates a reproducibility problem).
  • Does fig. 16 represent attribution as calculated in appendix G.1? If yes, it should say so in the caption.

Suggestions:

  1. Wayfarer uses a lot various tricks which are not particularly well justified: Rainbow (vs. simply DQN), h, IQN; how necessary are those? Does Wayfarer still work if one uses simply DQN instead of Rainbow? If one doesn't use h? Doesn't use IQN?
  2. [78] should be cited as the archival version of the paper (https://neurips.cc/virtual/2026/loc/sydney/poster/150071 ), rather than the workshop version.
4 comments
AuthorsOpenReview ↗

Response to corrections

First of all, we thank the reviewer for a thorough and generous review. We appreciate that they consider Wayfarer “a non-trivial achievement” and that they found the article “excellent and impressive”. We are very grateful for the precise corrections regarding previous work.

Corrections

We have accepted most of the corrections, and thank the reviewer for bringing these points to our attention. The changes mostly affected one paragraph of the introduction (“The Laplacian representation has been used to facilitate…”). Our comment about Laplacian representations being typically learned offline was indeed imprecise, as we were trying to distinguish between learning a representation at the same time as the option itself. We have removed this comment from the text.

“There is nothing in the other methods which are reliant on multiple streams of experience; when other methods use parallel environments it is simply to make there experiments go faster but all methods would work equally well with a single stream of experience”

We agree that not all methods we reference strictly require parallel environments. We would, however, be cautious about treating single versus parallel streams as interchangeable. Beyond speeding up training, the number of parallel environments has been found to positively affect representation quality, state space coverage, network expressivity and plasticity, training stability, and overall effectiveness of learning (Mayor et al., 2025). We were unable to check the reviewer’s claim regarding SHNEO’s Appendix F, as the archival version does not seem to be publicly available yet, and the workshop version of the paper does not include the PQN vs. single-stream DQN comparison. Such a finding would be quite interesting and warrant further investigation. Can the reviewer provide us with a link to the version with the referred Appendix F? Once we have verified the result, we can change our phrasing in the extended related work.

“Overstated; lines 72-75: Indeed, to our knowledge, no option-discovery method—whether grounded in reward, in given subgoals, or in the structure of the environment itself—has been shown to be able to effectively tackle exploration, credit assignment, and generalisation at once when processing high-dimensional observations”

We are unaware of other methods that have tackled all three simultaneously, and that is what we are referring to in the text. Of the papers listed by the reviewer: 16 does not focus on credit assignment nor generalisation; 18 does not focus on generalisation; 20 does not focus on generalisation; 22 does not learn options; 26 does not focus on credit assignment nor generalisation; and 78 (now 27) does not focus on generalisation (quote from the paper: “Since our focus is exploration rather than generalization”) and is considered contemporaneous work by ICLR. That said, we incorporated 27 into our revision of the introduction, and we acknowledged in our extended related work that 27 appears to be the closest in design to Wayfarer (Appendix K).

AuthorsOpenReview ↗

Response to clarification requests and suggestions

Clarifications

“There could be more explanations as to why the IQN Z are transformed with h and why does h take this form? What happens if the Z's are not transformed with h?”

We thank the reviewer for the questions: hh is the invertible value transformation of Pohlen et al. [58], also used in R2D2, NGU, and Agent57 [59, 60, 30], as an alternative to reward clipping. It reduces the scale of the optimisation target by compressing large values, while including more reward information than merely the sign. This enables learning in domains where sign information is insufficient, and it stabilises learning in domains with large reward magnitudes. We have now elaborated on this in Section 3.1.

The canonical alternative in the ALE is reward clipping. To test whether Wayfarer’s performance depends on hh, we replaced it with reward clipping (results added to Appendix I). We found that, in aggregate, Wayfarer may not need hh, meaning that we could further simplify the algorithm. Getting rid of hh eliminates some of the game-specific failures, like Skiing, and preserves our gains in Montezuma’s Revenge. However, in games with specific reward structures, like Private Eye, where the reward signals include -1000, -300, -1, 100, 5000, and 15000, the sign of the reward is not enough to do well. Investigating this further is a great avenue for future work. We have kept hh as the default, because it gives us a more generally applicable agent beyond the challenging set and the ALE: a transformation like hh leaves the objective unchanged, whereas reward clipping does not.

“line 296: applies a symmetric log transformation to the representation outputs. This is not clear, please clarify”

We apply symlog⁡(z):=sign⁡(z)log⁡(1+∣z∣)\operatorname{symlog}(z):=\operatorname{sign}(z)\log(1+|z|) element-wise to the Laplacian head’s outputs, so it outputs symlog⁡(fϕ(fψ(x)))\operatorname{symlog}(f_{\phi}\big(f_{\psi}(x)\big)). It is not strictly needed, but it compresses large values (doing so slightly more aggressively than hh), which we found to stabilise the gradient norms of the representation network when the data distribution shifts suddenly (e.g., when a new room is discovered in Montezuma’s Revenge). We thank the reviewer for the clarifying question: we have added the symlog definition to Section 3.3, hopefully clarifying the ambiguity.

“Does fig. 16 represent attribution as calculated in appendix G.1?”

Yes, the attributions in both Figures 16 and 17 use the method outlined in Appendix G.1 (now J.1). We have added this information to both captions.

Suggestions

“Does Wayfarer still work if one uses simply DQN instead of Rainbow? If one doesn't use h? Doesn't use IQN?”

Notice that DCEO [26] in a way has already demonstrated that Laplacian-based options work well with DQN. One of our goals with Wayfarer was to push the limits of performance while incorporating additional capabilities into the agent. Regardless, we agree that these choices deserve more justification. Rainbow and IQN are the strongest established single-stream, model-free agents on the ALE benchmark; and using IQN instead of C51 as the distributional component means that we do not commit to any prior assumptions about the support of the returns. The hh-transformation, as defined above, was also used to make the algorithm more general, since it allows for learning in environments where the sign of the reward is not informative enough. We have added more justification for using Rainbow and IQN to Section 2 and elaborated on the use of hh in Section 3.1.

Due to the high computational requirements of running ALE experiments, let alone training a full Wayfarer agent, we were only able to ablate what we considered to be the most important design choices for Wayfarer in discovering options and learning to use them: noisy networks, agent-centric regularisation, learning option values, and a baseline control without options. For the rebuttal, we also added results for reward clipping versus hh (Appendix I), as mentioned in one of our responses above, and a full internal control variant (e.g., Figure 3, Section 4).

We hope our revisions have addressed all of the reviewer’s concerns. If so, we would be extremely grateful if they would consider updating their score, as mentioned in the review.

Reviewer_EMhoOpenReview ↗

Mostly satisfied

I am mostly satisfied the author's modifications and clarifications.

One point remains which I would very much like to see in the camera-ready version if necessary: the ablation of Rainbow to DQN, that would allow the reader to see what the specific contributions of this paper add over DQN and help untangle if the Rainbow contributions are really necessary.

I have furthermore looked at all the other reviewers' comments and the rebuttal of the authors to those comments. One criticism of other reviewers which I very much agree with is Reviewer vmFB's Option-Value Target Formulation comment. I think the authors have adequately updated the paper to address that concern.

This is a significant non-trivial piece of work and considering the fixes I increase my score to 4 – clear accept.

We thank the reviewer for their thoughtful feedback and reassessment. We agree that such an ablation would be informative, and hope to include it in the camera-ready version.

Reviewer_FvRZOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this work, the authors present Wayfarer, a domain-agnostic online deep RL agent that discovers options from high-dimensional input data. The discovered options are then used to simultaneously improve not only exploration but also to accelerate credit assignment and generalize to new input data faster than learning new complex policies. To achieve this Wayfarer utilizes 3 neural networks, a main network to learn from task rewards, a skill network to learn a pair of options for individual eigenfunctions, and a representation network to learn eigenfunctions by approximating the Laplacian representation of the agentic-centric state. Experience and rewards connect the 3 networks, and they update from a single stream of input data in the same cycle using their individual encoders. To demonstrate the utility of the architecture, the authors experiment on the most challenging 10 Atari 2600 games that require long-horizon exploration and strategic behavior to achieve high scores. Compared to other approaches, such as Dreamer V3, IQN, and Rainbow, the performance of Wayfarer is shown to be superior in terms of exploration in some games, such as Montezuma's Revenge and Private Eye.

Critical strengths
  • The paper is well-written and provides ample background and motivation for the use of options to improve the exploration of long-horizon environments with sparse rewards. The authors also clearly describe the method's contributions relative to prior options work and how it uses a single input stream with multiple networks to build the option representation and act in the environment.
  • The formulation is well-motivated by the mathematical models described by the authors.
  • The virtuous loop of evolving dynamics between the networks is an interesting observation. The main network's choices shape the experience, the experience refines the representation, and the representation is used to learn options that feed back into the main network's choices.
  • The figures in the experiments section give a great visual presentation of the exploration performed by the agents in the environments acting under the influence of the wayfarer options. They illustrate how the options influence exploration in the same space and how exploration generalizes to unseen new environments.
  • The aggregate performance on the hard Atari 2600 environments demonstrates the improved general performance of Wayfarer across the dataset in an efficient way using a single stream of data.
Critical weaknesses
  • The method is shown to be superior in some environments but not a universal win overall. Specifically, on the Pitfall it was shown that an agent with Wayfarer options was able to discover different eigenfunctions to explore the environment but did not make progress.
  • While the use of 3 different networks to perform several updating functions is interesting the computational overhead per step is notably higher per step.
  • At this point, Atari 2600 is becoming a dated benchmark. However, with respect to previous work on learning options it provides the most informative comparison with previous work in the area.
Questions for authors
  • How does Wayfarer compare to the other agents over the wider Atari corpus in terms of learning rate and peak achieved score? The focus on the more challenging environments is interesting, but I'm curious how that behavior translates to the previously mastered environments and how long Wayfarer takes to achieve similar levels of performance.
  • How was the number of options chosen? Would the use of more options impact the environment's behavior in some way?
4 comments
AuthorsOpenReview ↗

Response to weaknesses

We thank the reviewer for their encouraging review, and are glad they found the paper well-written and the virtuous loop between representation, option, and task learning interesting.

“The method is shown to be superior in some environments but not a universal win overall [...]”

Wayfarer indeed does not outperform the baselines in every game, and we tried to report this transparently. Wayfarer’s gains are mainly in games that require long-horizon exploration and strategic behaviour (Montezuma’s Revenge, Private Eye, Venture). DreamerV3 is much stronger in Beam Rider, and no agent achieves a positive return in Pitfall!. In aggregate, Wayfarer outperforms the model-free baselines and matches DreamerV3 (Figures 3 and 12).

To our knowledge, no single-stream agent has achieved a positive return in Pitfall!, let alone within 200M frames. Presumably, this is because the game requires extremely deep exploration and persistence in the face of negative rewards. As the reviewer notes, Wayfarer discovers different options to explore the environment, but it is unable to explore deeply enough. Increasing exploration (e.g., by tuning option length or initial noise in the noisy network layers) might help, but it would likely come at the cost of performance on other games. Based on our ablations, learning option values helps balance this cost relative to using options only for exploration (e.g., DCEO-style [26]), but it does not eliminate the cost entirely. Resolving this trade-off is an important direction for future work. The reason we reported the analysis in Pitfall! is because we believe Wayfarer is a promising approach to tackle this open problem. We have added a few sentences to our conclusion discussing option termination as an important direction for future work.

“While the use of 3 different networks to perform several updating functions is interesting the computational overhead per step is notably higher per step.”

The reviewer is right; Wayfarer adds computational overhead compared to our model-free baselines. Note that DreamerV3 does the same but in different ways, and Wayfarer matches DreamerV3’s performance with ~4.5x fewer parameters [66]. We use the same frame budget, so the extra cost is in the learning updates. The overhead could be reduced in multiple ways, which we leave to future work. For example: eliminating IQN and dueling from the option heads, which alone would cut the parameter count from 43.7M to 25.6M; sharing one encoder across the three networks; lowering the representation network batch size; updating fewer options at every step—for example, selecting one randomly in the style of DCEO [26]—or updating them with a shared batch of data; and replacing the per-option head in the skill network with a single option-conditioned head. We have added information on the computational overhead to Appendix M.

“At this point, Atari 2600 is becoming a dated benchmark.”

We chose to evaluate Wayfarer in the ALE exactly because it provides the most informative comparison with previous work in the area, as the reviewer points out. Although evaluating agents in Atari 2600 games has been common practice in the field for over 10 years, there are still many research questions for which the ALE is a great evaluation platform, as Pablo Castro argued a few years ago. The challenging set is particularly relevant here: even the strongest single-stream agents struggle on aspects of the benchmark, with aggregate scores far below saturation (Wayfarer: median 0.69, IQM 0.65, mean 0.54).

AuthorsOpenReview ↗

Response to questions

“How does Wayfarer compare to the other agents over the wider Atari corpus in terms of learning rate and peak achieved score?”

We are also curious about this; however, we have not yet run Wayfarer across the 57-game suite. Given Wayfarer’s computational cost, we prioritised the challenging set to study whether the discovered options help in environments that require deep exploration and/or effective credit assignment. We narrowed down the scope to this subset of games to be able to run a comprehensive set of ablations, whose hyperparameters have been tuned, and multiple seeds per game—the importance of which has been noted in the field (Colas et al., 2018; Agarwal et al., 2021). During the rebuttal period, we have focused on running experiments that would complete the ablations: a full internal control and reward clipping. For these reasons, we are unable to run Wayfarer systematically on the full set of games with the remaining computational budget we have. Anecdotally, based on preliminary data, we observe that in some games we outperform the baselines, in some games we are tied, and in some we are worse. We suspect that our relative gains would be smaller in aggregate, because most of the games left out are reactive and will thus benefit less from temporally-extended actions.

Also, note that the challenging set already includes some games our single-stream baselines master, and they mostly suggest that options are not getting in the way: the synergy between noisy networks for exploration and learning option values results in the agent being able to explore using its options, but not being forced to do so, if they are not needed, because it can learn not to use them (e.g., Figure 15: Freeway, Pong, Surround). Notably, this means that Wayfarer can potentially get the best of both worlds.

“How was the number of options chosen? Would the use of more options impact the environment's behavior in some way?”

We evaluated the number of options based on both previous work and a preliminary, non-exhaustive hyperparameter sweep on a few challenging set games. Anecdotally, Wayfarer seemed to be relatively robust to the number of options, and our final choice of K=10K=10 (two for each of the five non-constant eigenfunctions) also took into consideration the computational cost of training more options. Our starting point was DCEO [26], where as few as K=5K=5 options have been shown to work on Montezuma’s Revenge. Our initial runs ruled out very small repertoires (1-3 options). We then ran a small sweep comparing 5 eigenfunctions (uni- and bidirectional, so 5 and 10 options) against 10 and 20 eigenfunctions (unidirectional, so 10 and 20 options). Bidirectional options showed a slight advantage, and increasing the number of options seemed beneficial, but learning 20 options was slower than learning 10. Achieving gains with even a small number of options makes sense in our case: for exploration, the smoothest eigenfunctions tend to matter most, because they capture the environment’s large-scale structure, while less smooth ones capture local structure. We thank the reviewer for pointing out that this had not been specified; we have specified our hyperparameter tuning in Appendix D.

We hope our rebuttal answers the reviewer’s key questions. If so, we would be extremely grateful if they would consider raising their score.

We thank the reviewer again for their encouraging review. As the discussion period nears its end, we would be grateful to hear whether our responses have addressed the reviewer's main concerns, or whether there are any remaining points we could clarify.

I want to thank the authors for their thorough responses to all the reviewers. Based on their feedback, I will raise my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.