- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Wayfarer is an online reinforcement learning system for hard-exploration Atari games that jointly trains three components from a single pixel-based experience stream: (1) an inverse-dynamics-regularised Laplacian representation, (2) intrinsic-reward option policies derived from the Laplacian eigenfunctions, and (3) an IQN-based high-level controller that selects between primitive actions and the discovered options. Rather than introducing a fundamentally new option-discovery objective, the paper's primary contribution is the successful online integration of Laplacian option discovery and exploitation directly from pixels without relying on a pre-collected dataset. The authors validate the system on an established subset of hard-exploration Atari games across five seeds, demonstrating substantial performance gains on several benchmarks (such as Montezuma's Revenge, Private Eye, and Venture) alongside qualitative analyses of representation learning, option behavior, and ablations.
Critical strengths
Effective Online Integration of Option Discovery and Control from Pixels: Discovering and learning when to exploit options online from a single stream of pixel observations—without assuming a pre-collected dataset that already covers the state space—is an important and difficult challenge. Extending beyond random exploratory option execution by jointly learning option selection, alongside motivating the representation via an inverse-dynamics auxiliary task to mitigate uncontrollable features, represents a valuable empirical integration.
Compelling Empirical Gains on Established Hard-Exploration Benchmarks: The system achieves striking improvements on notoriously difficult games within an established benchmark subset (evaluated across five seeds with uncertainty estimates and transparent disclosure of failures). In Table 4, Wayfarer achieves mean returns of approximately 4,187 on Montezuma’s Revenge (vs. 0 for IQN and 1,829 for DreamerV3) and 34,418 on Private Eye (vs. 180 for IQN), demonstrating that the discovered options provide substantial practical leverage in hard-exploration settings.
Critical weaknesses
Mismatch in the Option-Value Backup Target and Unsubstantiated Credit-Assignment Claims: Section 3.1 uses a fixed three-step target for an option's initiation value, bootstrapping over all actions and options regardless of whether the active option has terminated, despite execution following call-and-return semantics. With termination probability p = 0.2, the expected duration is E[T] = 5 and Pr(T > 3) = 0.8^3 = 0.512, meaning that in roughly half of nominal executions the bootstrap assumes a fresh high-level choice while execution remains committed to the option. This is not a valid Bellman target for the option-return random variable, directly undermining the headline claim of accelerated temporal credit assignment. Furthermore, the experimental ablations do not cleanly isolate credit assignment: the comparison against "no option values" conflates value learning with the exploration mechanism (NoisyNet vs. epsilon-greedy), and the epsilon-greedy-matched comparison ("no noisy networks" vs. "no option values") shows only a slight advantage that cannot separate faster reward propagation from altered exploration data distribution.
Overstated Claims Regarding Systematic Generalisation, Exploration, and SOTA Comparability: Several headline claims exceed what the experimental evidence supports:
-
Exploration and Generalisation: Qualitative exploration rollouts disable option termination, demonstrating persistent skill capacity rather than actual training-time trajectories under the 5-step mean duration. Similarly, transferring two selected options from a single checkpoint to unseen rooms is anecdotal and does not establish systematic generalisation across options, seeds, or starting states.
-
Baseline Comparability and Missing Full-Budget Control: Protocol differences (sticky actions, action sets, no-op starts, life information) complicate external baseline comparisons, making the internal "no options" ablation critical—yet this control is only evaluated with 3 seeds at 100M frames rather than the headline 5 seeds at 200M frames. Moreover, severe task-level failures on Beam Rider (5,077 vs. 18,830 for IQN), Skiing (-29,709 vs. -17,098 random), and Pitfall! (0 return) show that framing the work as "Mastering" the benchmark or claiming broad single-stream SOTA is stronger than warranted.
Questions for authors
Option-Value Target Formulation: Can you formally define what the fixed 3-step surrogate target represents relative to the true call-and-return SMDP target (the discounted sum of rewards over T steps plus gamma^T times max_o' Q(x_{t+T}, o')), characterize the resulting bias when Pr(T > 3) = 0.512, and clarify whether a duration-consistent or termination-aware intra-option backup was tested?
Full-Budget Internal Control: Do you have, or can you provide, results for the "no options" ablation at the full 200M-frame budget across 5 seeds to give a strictly protocol-matched baseline for the headline results?
Mathematical & Algorithmic Specifications:
-
In Section 2, symmetry of f(P^pi) alone does not guarantee that I - f(P^pi) is positive semidefinite or has a constant first eigenfunction. What is the exact empirical Laplacian operator and sampling measure/inner product used in ALLO?
-
The skill reward telescopes to u(x_{t+n}) - u(x_t) only as an undiscounted sum. When the discount factor gamma < 1, what is the exact skill Bellman target and intended objective?
Computational Cost & Reproducibility: What are the parameter counts, hardware requirements, wall-clock training times, and per-cycle update costs (given three encoders, ten skill heads, and batch size 512) compared to standard IQN? Additionally, please specify the exact network dimensions, LayerNorm placement, terminal/truncation handling, and how the single hyperparameter configuration was tuned/selected.
5 comments
Response to questions (1/2)
“Option-Value Target Formulation: Can you formally define what the fixed 3-step surrogate target represents relative to the true call-and-return SMDP target (the discounted sum of rewards over T steps plus gamma^T times max_o' Q(x_{t+T}, o')), characterize the resulting bias when Pr(T > 3) = 0.512, and clarify whether a duration-consistent or termination-aware intra-option backup was tested?”
We thank the reviewer for this analysis, which has shown us that the paper does not explain an important choice. The reviewer is right that our target is not the call-and-return target: it values an option as if the agent could re-decide steps after initiating it, which makes the option-value target optimistic. We use it because the options keep changing with the representation: we let option values guide initiation and credit the option’s later steps to the primitive actions stored at those steps, distilling option behaviour into primitive-action values over training (as explained in Section 3.1). In our preliminary experiments, intra-option backups drove the option-selection rate down, though these experiments were too limited to draw any conclusions from. We have revised Section 3.1 accordingly, and added the characterisation of the bias below to Appendix E.
Ignoring and writing : option values are trained only at initiation, and their target truncates the call-and-return SMDP target at steps. In the idealised setting in which the online and target networks are the same, so the double-Q bootstrap reduces to a , and experience replayed follows the current policy, the two differ only if an option is still running at , where we bootstrap from the greedy option at . Because termination is memoryless ( at every step), continuing is worth the same as re-initiating it at , which is one term of the . Our target therefore corresponds to executing options with interruption at the -step boundary, in the sense of Sutton et al. (1999) [2], Section 4, and it is optimistic relative to call-and-return. Per backup, the bias from the initiating option is , with , that is, times the expected advantage of re-deciding over continuing at .
“Full-Budget Internal Control: Do you have, or can you provide, results for the ‘no options’ ablation at the full 200M-frame budget across 5 seeds to give a strictly protocol-matched baseline for the headline results?”
Yes, we have added the control, “Rainbow (IQN)”, to the main results (Figure 3, Section 4; per-game results in Appendix H.1), evaluated with 5 seeds at the full budget of 200M frames. These results further support our claim that discovered options improve the agent’s performance: Wayfarer achieves 2.15x median performance, 1.24x IQM, 1.11x mean, and 1.12x optimality gap, compared to this baseline.
“In Section 2, symmetry of f(P^pi) alone does not guarantee that I - f(P^pi) is positive semidefinite or has a constant first eigenfunction. What is the exact empirical Laplacian operator and sampling measure/inner product used in ALLO?”
Again, we thank the reviewer for highlighting a detail that is important to clarify. In particular, we have fixed the sentence in Section 2, which previously implied that the symmetry of alone would guarantee that is PSD with a constant first eigenfunction. We have also elaborated on the empirical Laplacian operator and inner product used in ALLO (Appendix F with a forward-reference in Section 3.3).
Concretely, ALLO learns from samples and never forms : it solves s.t. , where is the distribution of replayed pairs from Section 3.3 and , the distribution of , is uniform over the replay buffer. The empirical operator is thus with and , which for any is self-adjoint and PSD on with a constant first eigenfunction (), and equals when is stationary (the Laplacian of 33; the form given in Section 2).
Response to questions (2/2)
“The skill reward telescopes to u(x_{t+n}) - u(x_t) only as an undiscounted sum. When the discount factor gamma < 1, what is the exact skill Bellman target and intended objective?”
We do not discount the intrinsic rewards within each -step segment. The intrinsic -step return at time is an undiscounted sum of one-step eigenfunction changes over the -step horizon, from to , (Section 3.2), taking the place of the discounted task return . Otherwise, the skill network uses a target of the same form as the main network, without and (as ALLO’s orthonormality constraints keep the eigenfunctions, and hence the intrinsic rewards, on a similar scale across environments): , with the greedy primitive action under the online skill network (parameterised by ). So, discounting applies across -step segments and not within them, and the objective is to seek and stay in regions where is high (for ) or low (for ). We have clarified this by adding the target to Section 3.2.
“What are the parameter counts, hardware requirements…”
With the current configuration, Wayfarer has approximately 43.7M trainable parameters: 2.6M in the representation network, 34.3M in the skill network, and 6.7M in the main network. As the reviewer anticipates, most of the parameters lie in the option heads, at approximately 3.4M each, whereas the encoders have only 0.08M each. Our internal control (Wayfarer’s main network alone, our closest variant to a standard IQN agent) has 6.7M. For reference, DreamerV3, whose aggregate performance Wayfarer matches, uses a 200M-parameter model for Atari 2600 games, about 4.6x Wayfarer’s size [66, extended data table 2].
A Wayfarer run fits on a single L40S (48 GB) or a 40 GB H100 MIG instance. Wayfarer needs more GPU memory than the internal control, because most of the memory requirement comes from the experience replay buffer, which both agents use and which we keep on the GPU. Wayfarer’s buffer holds steps; for reference, DreamerV3’s holds with additional latent states stored [66, extended data table 5].
Training for 200M frames takes about nine days on an H100 MIG instance, versus about two days for the internal control (about 4.5x overall). Wayfarer's cost could be reduced in multiple ways: eliminating IQN and dueling from the option heads (which alone would cut the parameter count from 43.7M to 25.6M); sharing one encoder across the three networks; lowering the representation network batch size; updating fewer options at every step—for example, selecting one randomly in the style of DCEO [26]—or updating them with a shared batch of data; and replacing the per-option head in the skill network with a single option-conditioned head. We leave this for future work.
This information has been added to Appendix M.
“Additionally, please specify the exact network dimensions…”
- The exact network dimensions and LayerNorm placement have been added to a table Appendix B, and we will make the code available upon the paper publication. Every convolutional and hidden fully connected layer is followed by LayerNorm, and then a ReLU. Output layers and IQN quantile embeddings have no LayerNorm.
- Terminal/truncation handling: Main and skill networks: the -step return is truncated alongside the bootstrap. The representation network does not sample across episodes.
- We drew most of our initial hyperparameters from previous work [26, 44, 51, 65]. We refined these hyperparameters and set the remaining ones using preliminary, non-exhaustive sweeps on a few challenging set games. In these sweeps, we observed Wayfarer to be relatively insensitive to several design choices, such as the option termination probability (though we did not test extremely long options), the discounting in , and whether the agent-centric Laplacian representation was learned via a single network (regularised for control-relevant features) or decoupled into separate inverse-dynamics and Laplacian networks. We chose the number of options weighing the cost of training more options. This information has been added to Appendix D.
We hope our clarifications, additional results, and changes to the paper have addressed the reviewer’s main concerns. We are very grateful for the reviewer’s feedback. If the main concerns have been addressed, we would be grateful if the reviewer could increase their score.
Response to weaknesses
First of all, we thank the reviewer for their detailed review. We are glad they see online option discovery and control as “an important and difficult challenge”. In particular, we found their questions about the option-value target and empirical Laplacian operator very useful: they led to some important clarifications that have made the paper stronger.
Our option-value target is intentionally optimistic and not the call-and-return SMDP target. Please see below for details (as part of our response to your question about the target formulation).
The slightly better performance of “no noisy networks” is evidence consistent with our claim of accelerated credit assignment. Note that the ablated variant “no option values” includes noisy layers in the main network and ϵ-greedy for exploration with options. This means that the “no option values” agent explores not only using randomly selected options, with ϵ-greedy, but also using the noisy value predictions over its primitive actions. Consequently, “no option values”, if anything, likely has an exploration advantage compared to “no noisy networks”, yet “no noisy networks” shows a slight advantage. We thank the reviewer for this criticism, which highlighted the need for us to clarify our text; we have done so in Section 4.2 (lines 448-451 and lines 513-516).
The above said, we agree that our ablations cannot cleanly separate faster reward propagation from changes in the collected data. In Wayfarer, the two are intertwined: options change both what is experienced and which decisions receive credit. The mechanism we propose accelerates credit assignment is that our option trajectories travel more directly to significant events than dithering with primitive actions (e.g., Figure 1), so fewer n-step backups are needed to credit the decisions that led there. We have made our hypothesis clear in Section 3.1.
Yes, the reviewer is right: these figures show the options’ learned policies with termination disabled. Termination in Wayfarer is a fixed probability rather than something the agent learns, so disabling it isolates the part the agent does learn, the option’s policy, and shows it over a longer horizon. This was previously stated in Section 4.2, and we have also now made it explicit in each of the figure captions in Section 4.2 (Figures 4-6). As for training-time behaviours, the option-selection rates in Appendix C.2, measured with termination enabled, exemplify how the agent uses options during training in Montezuma’s Revenge: the agent favours options at the frontier of its experience, and primitive actions once a region (i.e., new room) becomes familiar.
As Section 4.2 states, these figures provide anecdotal evidence. As such, we have tried to provide a diverse range of examples from different games (Montezuma’s Revenge, Private Eye, Pitfall!), across options (Figures 1, 5, 6), across different points in training (Figure 4), and across starting states, including states that the agent has not visited yet (Figure 6). They are existence proofs: they show what options discovered online by Wayfarer can do, how varied they are, and how they develop over training, but being agnostic to how often they occur during training.
To further support that discovered options improve the agent’s performance, we have added a full-budget control to the main results (Figure 3, Section 4), “Rainbow (IQN)”, evaluated with 5 seeds at the full budget of 200M frames, as requested by the reviewer. This variant corresponds to the “no options” configuration in our ablations: Wayfarer’s main network alone, acting over primitive actions, without the representation and skill networks. It controls for our changes, and thus separates the contribution of discovering and learning to use options via Wayfarer. The results further support our claims: the full Wayfarer agent is clearly an improvement in aggregate (2.15x median performance, 1.24x IQM, 1.11x mean, and 1.12x optimality gap).
We used the word “mastering” similarly to how the DreamerV3 paper [66] used it, referring to aggregate metrics instead of success in every single game. DreamerV3 also failed to outperform baselines in some Atari 2600 games. To be clear, our use of the word mastery refers to achieving single-stream SotA in aggregate human-normalised performance on the challenging set of Atari 2600 games. On the individual failures: some of these may be attributable to specific design choices: for example, in Skiing, Wayfarer outperforms all other agents when using reward clipping instead of value rescaling. We have added this result to the paper (Appendix I).