acceptodds
Under review as a conference paper at ICLR 2027

Value-Directed Exploration for Efficient Model-Based Reinforcement Learning

Abstract

Efficient exploration remains a central challenge in reinforcement learning, particularly when environment interactions are costly. Curiosity-motivated algorithms guide exploration using model uncertainty as an intrinsic reward, however resolving errors in rarely visited or return-insensitive regions does not necessarily improve policy performance. Consequently, such strategies may allocate exploration effort inefficiently with respect to the task objective. To address this, we propose a value-directed optimistic exploration strategy that evaluates model uncertainty through its effect on future value, thereby prioritizing information acquisition in regions where reducing uncertainty can most affect which actions or policies are preferred. Our analysis yields a value-weighted information gain that can be substantially smaller than standard information gain when value sensitivity concentrates on a low-dimensional subset of the dynamics, giving regret bounds whose leading term tightens with this reduction and otherwise recovers value-agnostic scaling. Empirically, value-directed exploration improves performance in environments with task-irrelevant distractors while matching strong baselines on standard benchmarks.

Primary area:
reinforcement learning
Keywords:
Model-based reinforcement learning, exploration, optimism

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_U8t1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VDO-MBRL, an optimistic model-based exploration method that weights model epistemic uncertainty by its estimated effect on future value, replacing the value-agnostic uncertainty bonus of SOMBRL. Two practical bonuses are introduced: a gradient-based variant that weights ensemble uncertainty along value-gradient directions, and a gradient-free variant that measures the spread of predicted next-state values. Theoretically, under GP/RKHS assumptions, the authors define a value-weighted information gain ΓN⋆\Gamma^\star_N and derive regret bounds whose leading term scales with it. They also give a tightened SOMBRL bound and a lower bound showing that calibration cannot depend on ΓN⋆\Gamma^\star_N alone. Empirically, VDO-MBRL outperforms SOMBRL on author-constructed noisy-TV variants of PointMaze and MiniWorld, and performs comparably on 12 state-based DMC tasks and four vision-based DMC/Atari tasks.

Critical strengths
  1. Coherent theoretical treatment within the GP/RKHS framework. The paper provides an upper bound, a sharpened SOMBRL analysis, and a lower bound clarifying what cannot be improved. This is the main source of value in the submission. However, its relevance to the evaluated algorithm is limited, since the practical method departs substantially from the analyzed bonus (see Weaknesses).
Critical weaknesses
  1. The empirical evidence rests on author-constructed environments. All gains appear only on six self-designed noisy-TV settings. Across 16 standard benchmarks, VDO-MBRL shows no improvement over SOMBRL while costing 1.41.4–1.9×1.9\times more wall-clock time. SOMBRL is the only exploration baseline. Noise-robust exploration methods that the paper itself cites (Mavor-Parker et al., 2022; Jarrett et al., 2023; Hou et al., 2026) are absent, so it remains unclear whether value direction is needed at all.

  2. There is a large theory–practice gap. The guarantees apply to the full bonus with second-order and transfer corrections under GP calibration. The implementation keeps only the first-order term with ensembles, and adds SOMBRL warmup, running-mean normalization, clipping, and annealing. The theory therefore does not cover the evaluated algorithm.

  3. The method relies on its own tuning. VDO uses task-specific warmup lengths (50k–1M steps), with separate values for VDO-grad, which SOMBRL does not require. This makes it hard to attribute the gains to the mechanism rather than to tuning.

Questions for authors
  1. Can you compare against noise-robust exploration methods (e.g., Mavor-Parker et al., 2022; Jarrett et al., 2023; Hou et al., 2026) on the noisy-TV tasks? Without them, the necessity of value direction is not established.

  2. Can you report ablations that remove the warmup, normalization, and clipping from VDO, or alternatively add the same components to SOMBRL? This would isolate the contribution of value weighting.

  3. Ensemble disagreement on irreducible Gaussian noise should decay with data. Why does SOMBRL remain attracted to the TV? Please report SOMBRL's bonus magnitude at the TV cell over training, so that ensemble miscalibration can be ruled out.

  4. How does the method relate to Information-Directed Sampling (Russo & Van Roy) and the Uncertainty Bellman Equation (O'Donoghue et al., 2018), which also direct exploration by value-relevant information? Neither is discussed.

  5. Since the weights w⋆w^\star are not bounded by one (Eq. 81), ΓN⋆≤ΓN\Gamma^\star_N \le \Gamma_N is not guaranteed. Can you estimate both quantities on the evaluated tasks to show that the claimed reduction actually occurs?

Suggestion:

  1. Evaluation on established hard-exploration benchmarks (e.g., Montezuma's Revenge) would show whether the benefit extends beyond constructed distractors.
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you very much for your careful review of our paper and the specific suggestions for strengthening it. We are pleased to hear that you find our theoretical treatment coherent, including the regret upper bound, the sharpened SOMBRL analysis, and the lower bound clarifying what cannot be improved. We address your comments below and hope this will help improve your evaluation of the paper.

Weaknesses:

  1. The empirical evidence rests on author-constructed environments. Performance gains appear only on six self-designed noisy TV settings. Across 16 standard benchmarks, VDO-MBRL shows no improvement over SOMBRL while costing 1.41.4-1.9×1.9\times more wall-clock time. SOMBRL is the only exploration baseline and noise-robust exploration methods are absent, so it is unclear whether value direction is needed at all.

Response: Thank you for the opportunity to address this point. We agree that evaluations beyond our constructed distractor settings and comparisons with other exploration approaches are important for establishing the broader benefits of VDO-MBRL. To this end, we have additionally evaluated VDO-MBRL on three new visual tasks, specifically River Raid, Montezuma's Revenge, and RoboDesk Push Green, bringing the total number of unmodified benchmarks considered to 19 (consisting of 12 state-based DMC tasks and 7 vision-based tasks across DMC, Atari, and RoboDesk). On Montezuma's Revenge, an established hard-exploration benchmark, both VDO variants learn more efficiently than SOMBRL and on Push Green both achieve higher returns than SOMBRL (Figure 4), showing that value-directed exploration can also benefit tasks without constructed distractors. Across all unmodified benchmarks, VDO-MBRL consistently remains among the strongest methods, although many of these tasks pose limited exploration challenges, as indicated by the strong performance of MBPO on most state-based DMC tasks (Figure 11). For this reason, we focused our visual evaluation on tasks where prior work reported benefits from exploration, as discussed in our response to Reviewer DH7D (Weakness 2). Our theory predicts the largest improvements when model uncertainty contains a substantial task-irrelevant component, so that the undirected information gain grows while the value-weighted information gain remains small (Appendix G). The noisy TV settings instantiate this structure directly by letting the agent choose between task-irrelevant uncertainty and task reward, while controlling whether the distractor is stochastic or learnable and how many dimensions it contains. These settings therefore provide a targeted test of the mechanism that complements, rather than replaces, the standard benchmarks for showing the benefits of VDO-MBRL.

To address the point about limited baselines, we have also added the following six new exploration baselines: the noise-aware method AMA (Mavor-Parker et al., 2022), the learning-progress method LPM (Hou et al., 2026), MaxInfoRL, and optimistic model-based methods H-UCRL, LOVE, and OWM (as described in Appendix J with results given in Figure 12). We evaluate these baselines on state-based Hopper Hop, the Switchback maze with a d=64d=64 stochastic noisy TV, visual Finger Spin, and visual MiniWorld with a deterministic TV, and we select AMA as the representative noise-aware method since Curiosity in Hindsight (Jarrett et al., 2023). On the stochastic-TV Switchback maze, AMA performs competitively with VDO, which is consistent with its objective of discounting irreducible noise. However, on deterministic-TV MiniWorld, both AMA and LPM achieve substantially lower returns than VDO, which is consistent with the learnable TV dynamics not being discounted as aleatoric noise by AMA and still providing learning progress for LPM. VDO additionally outperforms the optimistic baselines on all evaluated tasks and MaxInfoRL on both noisy TV tasks, as discussed further in our responses to Reviewer P4NP (Weakness 4 and Question 1). Overall, VDO-MBRL is the only method that performs consistently well across these tasks, which demonstrates that value direction provides benefits beyond noise filtering and learning progress, particularly when task-irrelevant uncertainty is reducible.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

Finally, regarding compute time, we would like to point out that the multipliers in Table 2 are measured relative to the exploration-free MBPO or DreamerV3 backbone rather than SOMBRL, which itself requires 1.11.1-1.4×1.4\times the backbone training time. On the visual tasks, where training is most expensive, VDO-grad requires training time comparable to SOMBRL on most tasks, so the more consistent performance of value-directed exploration comes at little additional cost over undirected exploration. On the state-based tasks, VDO adds a moderate overhead relative to SOMBRL, although all runs complete in under 14 hours. As discussed in our response to Reviewer DH7D (Weakness 2), we have also reduced the cost of visual VDO-spread by using fewer Monte Carlo samples, bringing its training time on Hopper Hop to within 4% of SOMBRL.

  1. There is a large theory-practice gap as the guarantees apply to the full bonus with second-order and transfer corrections under GP calibration while the implementation keeps only the first-order term with ensembles, and adds SOMBRL warmup, running-mean normalization, clipping, and annealing.

Response: We agree that the regret bound applies to the full theoretical bonus under GP calibration and does not cover the practical variants. We would like to clarify that the purpose of the theory is not to provide guarantees for neural network implementations in high-dimensional control settings where calibrated confidence bounds on the dynamics are generally unavailable, but rather to identify and address a key obstacle to efficient exploration, namely that model uncertainty can be largely irrelevant to the task. In a tractable setting with calibrated models, we derive an exploration bonus from how dynamics errors affect future values and establish a regret bound that characterizes when task relevance improves exploration efficiency. These insights guide the design of our practical algorithm, even though the formal guarantees do not extend to it.

As discussed in our response to Reviewer P4NP (Weakness 2), the calibrated GP assumption is needed to construct confidence bounds on dynamics prediction errors, which the optimism guarantee relies on. This gap between the calibrated model analysis and scalable implementation is shared by H-UCRL and SOMBRL, which derive their regret bounds under the same calibration assumption and evaluate practical variants that replace GP uncertainty with neural network ensemble disagreement. We follow the same approach, which preserves the central role of epistemic dynamics uncertainty in guiding exploration. Our practical bonus retains the first-order value-sensitivity term and omits the second-order and transfer corrections required for the optimism guarantee, since these add complexity and compute time. We also observe that VDO-grad with a quadratic uncertainty correction motivated by the full bonus learns comparably to the first-order version (Figure 15), so we retain the simpler bonus.

The remaining implementation components mentioned stabilize training without changing which uncertainty the bonus prioritizes. More specifically, running-mean normalization and annealing are shared with SOMBRL (Appendix I) and rescale the bonus by a common factor at each update, which preserves the relative value weighting across transitions and clipping limits the effect of outliers from the learned critic. We also find that the SOMBRL warmup is not required on the state-based tasks, where both VDO variants remain effective with zero warmup (Figure 14), as discussed in our response to Weakness 3. In summary, our theoretical analysis establishes when task relevance can improve exploration guarantees and our experiments test whether these benefits persist in practical settings that fall outside the assumptions required for the analysis.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

  1. The method relies on its own tuning as VDO uses task-specific warmup lengths with separate values for VDO-grad which SOMBRL does not require. So it is hard to attribute the gains to the mechanism rather than to tuning.

Response: Thank you for pointing this out. We agree that it is important to separate the contribution of value weighting from that of warmup tuning. To this end, we have added warmup sweeps for both VDO variants on state-based Hopper Hop, the Switchback maze with a d=64d=64 noisy TV, and visual MiniWorld with a deterministic TV (Figure 14). On the state-based tasks, both VDO approaches remain effective across all warmup durations, including zero warmup, so their gains there do not depend on the chosen warmup values, while VDO-grad benefits from a longer warmup on visual MiniWorld. We also compare against a matched SOMBRL->no-bonus baseline that uses the SOMBRL bonus for the same warmup period and then stops adding the bonus. Both VDO methods generally outperform this baseline, which also shows that their performance improvements cannot be attributed to the initial period of undirected exploration alone. SOMBRL does not need a warmup since its bonus does not depend on a learned critic. Moreover, we observe that task reward is usually not discovered before the switch (Table 3) and both VDO methods learn effectively with zero warmup even when their initial critics are deliberately misleading (Figure 15). VDO-MBRL therefore does not rely on an accurate critic when value-directed exploration begins, which limits its sensitivity to the warmup length.

Finally, we also select the exploration coefficient λ\lambda based on SOMBRL's performance and reuse the value for VDO-MBRL without further tuning. Our new λ\lambda sweep in Figure 13 shows that both VDO methods remain effective across a range of coefficients, while SOMBRL achieves similar returns at every value on the Switchback maze, which are all below the VDO variants (see our response to Reviewer P4NP, Question 5). Together, these ablations support that the gains of VDO-MBRL arise from the value-directed weighting of model uncertainty rather than from tuning.

Questions:

  1. Can you compare against noise-robust exploration methods (e.g., Mavor-Parker et al., 2022; Jarrett et al., 2023; Hou et al., 2026) on the noisy-TV tasks? Without them, the necessity of value direction is not established.

Response: Thank you for the suggestion. Yes, as described in our response to Weakness 1, we have added the noise-aware method AMA (Mavor-Parker et al., 2022) and the learning-progress method LPM (Hou et al., 2026) as baselines. We find that AMA performs competitively with VDO on the stochastic noisy TV, however both AMA and LPM achieve substantially lower returns than VDO on MiniWorld with the deterministic noisy TV. These results show the benefit of value direction when task-irrelevant uncertainty is reducible, which is a case that noise-robust exploration is not designed to address. We select AMA as the noise-aware baseline since it integrates with our probabilistic dynamics models, whereas Curiosity in Hindsight (Jarrett et al., 2023) would require a separate representation learning architecture.

  1. Can you report ablations that remove the warmup, normalization, and clipping from VDO, or alternatively add the same components to SOMBRL? This would isolate the contribution of value weighting.

Response: Yes, we have included ablations on the warmup period as described in our response to Weakness 3 and given in Figure 14 of the updated paper. The results show that both VDO methods remain effective even with zero warmup on the state-based tasks and generally outperform a matched SOMBRL->no-bonus baseline. For normalization and annealing, our experiments already follow the second suggestion: as described in Appendix I, SOMBRL uses the same running-mean normalization and uncertainty-based annealing as VDO. With these components shared, the main remaining difference between the two methods is whether model uncertainty is weighted by its relevance to task value, so the comparison with SOMBRL isolates this contribution. The only VDO-specific component is clipping, which caps the value-directed signal at a running 99th percentile to limit outliers from the learned critic. SOMBRL does not compute this critic signal and the clip affects only the largest values, leaving the value weighting of all other transitions unchanged.

Reviewer_P4NpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VDO-MBRL, a value-directed optimistic exploration method for model-based RL. Instead of using the magnitude of model uncertainty as the exploration bonus, as in SOMBRL, the method weights model uncertainty by the sensitivity of the value function to the predicted next state, derived from the performance difference lemma and a first-order Taylor expansion. Two practical variants are given: VDO-grad, which uses value gradients, and VDO-spread, which uses the spread of predicted values across an ensemble. The paper also defines a value-weighted information gain and derives a regret bound based on it. Experiments on state-based PointMaze and visual MiniWorld environments with noisy TV distractors show improvements over SOMBRL, and results on DMC and Atari show performance comparable to SOMBRL.

Critical strengths
  1. The paper targets a relevant problem. Uncertainty-driven exploration can be attracted by task-irrelevant stochasticity, and focusing exploration on uncertainty that affects decisions is a reasonable direction.
  2. The method is simple and easy to plug in. Both variants only add a task critic on top of an ensemble dynamics model and can be used with MBPO and DreamerV3 with moderate overhead.
  3. The theoretical analysis is fairly complete. The paper gives a regret bound with the value-weighted information gain, tightens the SOMBRL bound, and uses a lower-bound example to show the limits of the approach.
Critical weaknesses
  • The novelty is limited. Using model uncertainty through its effect on value is not new in model-based RL. DOVE and LOVE (Seyde et al., 2020; 2021) explore with a UCB on values predicted by an ensemble of dynamics models, with the same motivation as this paper and essentially the same signal as VDO-spread. POMBU, value-distributional MBRL, Epistemic MCTS, and model-value inconsistency also propagate model uncertainty to values, and Appendix C.1 shows that the VDO bonus is a first-order linearization of H-UCRL. Most of these works are not cited, and the paper should clarify what is new beyond them.
  • The theory gives limited support to the practical algorithm. β\beta in Theorem 5.1 still depends on ΓN\Gamma_N (Appendix H), so the gain over the tightened SOMBRL bound is only a ΓN⋆/ΓN\sqrt{\Gamma^\star_N/\Gamma_N} factor in the leading term, with an extra ΓN2\Gamma_N^2 term. In addition, the analysis assumes a calibrated GP and a known reward, while the implementation uses an NN ensemble, a learned critic, only the first-order term, and several heuristics.
  • The method depends on the critic and a task-specific warmup. Since the critic is unreliable early in training, VDO uses the SOMBRL bonus for a warmup of 50k to 1M steps, and the noisy TV curves show a similar early TV peak before the switch. It is unclear how the method behaves when no reward is found during warmup or the critic is wrong, in which case it may suppress needed exploration.
  • The baselines and experiments are not sufficient. SOMBRL is the direct predecessor of VDO and shares most of its components, so the main comparison is closer to an ablation of the bonus. More direct baselines such as LOVE, H-UCRL, noise-robust curiosity methods, and Policy-Shaped Prediction are not compared, although several of them are cited. The main evidence comes from two hand-designed noisy TV environments, with no results on existing distractor benchmarks such as the Distracting Control Suite, and ablations on warmup and λ\lambda are missing.
Questions for authors
  1. LOVE has the same motivation and essentially the same signal as VDO-spread, and Appendix C.1 shows that VDO-grad is a linearization of H-UCRL. What is new beyond these works? How does VDO compare with LOVE on the visual tasks and with H-UCRL on the state-based tasks?
  2. How sensitive is the method to the warmup length? What happens without warmup, or when no reward is found during warmup? Is there a mechanism to fall back to undirected exploration when the critic is wrong?
  3. Can the authors compare with the cited noise-robust curiosity methods, Luis et al. (2023), Policy-Shaped Prediction, and MaxInfoRL?
  4. Optimistic World Models (Mete et al., 2026, arXiv:2602.10044) also performs optimistic exploration on DreamerV3, but it biases the learned dynamics toward high reward instead of using an uncertainty bonus, so it is not obviously affected by a noisy TV. How does VDO compare with it in the visual noisy TV setting?
  5. Was λ\lambda for SOMBRL tuned for each distractor dimension? Does SOMBRL use the same normalization, clipping, and annealing as VDO?
  6. Does the advantage over SOMBRL hold on the Distracting Control Suite? In the multi-channel noisy TV example of Appendix G, does regret scale with the number of distractor dimensions as predicted?
7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you very much for your time spent reviewing our paper and for the detailed feedback. We are glad that you find the problem to be relevant, approach easy to use, and theoretical analysis complete. We address your comments below and hope this will help improve your evaluation of the paper.

Weaknesses:

  1. The novelty is limited as other works use model uncertainty through its effect on value. These works should be cited and the paper should clarify what is new beyond them.

Response: Thank you for highlighting these related works. We agree that using model uncertainty through its effect on value has been studied previously, particularly in DOVE/LOVE, and we have expanded the related work discussion in the revised paper to include these works. We would like to clarify that our main theoretical contribution is to characterize when and why value-directed exploration can improve learning efficiency. Specifically, we introduce a value-weighted information gain and derive an instance-dependent regret bound whose leading term can be substantially smaller when model uncertainty concentrates in task-irrelevant directions. We establish conditions under which this improvement occurs, illustrate the distinction through a multi-channel distractor example, and characterize the remaining dependence on global information gain. Furthermore, our new analysis also applies to existing optimistic algorithms, including H-UCRL and LC$^3$ and provides a sharper understanding of their exploration complexity.

At the algorithm level, VDO-MBRL differs from existing works in how value-relevant uncertainty is computed and incorporated into exploration. Our practical variants use the dynamics ensemble and task critic to estimate how uncertainty in predicted next states affect future task value and then add this estimate as an exploration bonus to the reward, allowing these methods to be easily integrated with existing model-based RL algorithms. VDO-spread does indeed use a similar uncertainty signal to the DOVE/LOVE works, as both approaches estimate value uncertainty induced by ensemble dynamics predictions. However, VDO-spread measures disagreement between the values of ensemble-predicted next states and uses this as a local exploration bonus, whereas DOVE/LOVE estimate uncertainty over predicted trajectory returns and optimize an upper confidence bound on these returns. Similarly, VDO-grad can be interpreted as a first-order approximation of H-UCRL's optimistic model perturbations, as discussed in Appendix C.1, but avoids expanding the action space with hallucinated controls whose dimension scales with the state space. The other approaches mentioned propagate model-induced uncertainty through Q-value estimates (POMBU), learn distributions over values (value-distributional MBRL), propagate uncertainty through tree search (Epistemic MCTS), or estimate uncertainty from inconsistencies between model-based value predictions over horizons (model-value inconsistency). These approaches share important motivations with VDO but differ in how uncertainty is estimated and incorporated into policy improvement. These formulations also introduce different estimation and optimization challenges, such as propagating uncertainty over longer horizons, calibrating optimistic value estimates, or jointly optimizing over plausible dynamics, which can affect practical performance and sensitivity to hyperparameter choices.

In summary, our contribution is not the general principle of using value uncertainty for exploration, but rather a theoretical characterization of its benefits and limitations, together with a simple and effective bonus-based approach that applies across different MBRL architectures. To strengthen the empirical comparison with prior optimistic exploration methods, we have additionally evaluated LOVE and OWM on visual tasks and H-UCRL on state-based tasks, with the results provided in Figure 12 of Appendix J.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

  1. The theory gives limited support to the practical algorithm as the analysis assumes a calibrated GP and a known reward, while the implementation uses an NN ensemble, a learned critic, only the first-order term, and several heuristics. Also β\beta in Theorem 5.1 still depends on ΓN\Gamma_N, so the gain over the tightened SOMBRL bound is only a factor of ΓN⋆/ΓN\sqrt{\Gamma^\star_N/\Gamma_N} with an extra ΓN2\Gamma_N^2 term.

Response: Thank you for the opportunity to clarify the connection between our theoretical analysis and practical algorithms. We agree that the regret guarantee does not directly extend to our neural-network implementations and we would like to explain the role of the assumptions and approximations involved. Our intent with the theory is to identify and address a practical obstacle to efficient exploration: model uncertainty can be largely irrelevant to the task. We analyze a tractable setting with this challenge, derive an exploration bonus from how the dynamics errors affect future values, and establish a regret bound that characterizes when task relevance can improve exploration efficiency.

The calibrated GP assumption allows us to construct confidence bounds on dynamics prediction errors and establish a rigorous optimism guarantee and assuming known rewards isolates the contribution of dynamics uncertainty. These assumptions are consistent with the theoretical settings studied in H-UCRL and SOMBRL, which similarly derive regret bounds using calibrated models. To scale to high-dimensional continuous and visual control settings, we follow the same approach adopted in their practical implementations: replacing GP-based uncertainty estimates with disagreement among neural-network dynamics ensembles. This preserves the central idea of using epistemic dynamics uncertainty to guide exploration and allows us to evaluate VDO-MBRL with scalable model-based RL algorithms such as MBPO and DreamerV3. It is also important to clarify that our theory does not require knowledge of the true optimal value function as the bonus uses the continuation value of a candidate policy under the learned mean dynamics model, which we approximate in practice using a learned extrinsic critic for the current policy. The practical VDO-grad also retains the first-order value-sensitivity term and omits the higher-order corrections required for the theoretical optimism guarantee. We omit these terms because we found their influence on performance to be small while contributing to the complexity and compute time of the algorithm. For instance, we provide learning curves for VDO-grad with a quadratic uncertainty correction motivated by the theoretical bonus in Figure 15 of the revised paper, which shows this small performance change. This is why we retain the simpler first-order bonus in our practical implementation.

Regarding the regret bound, we agree that the calibration coefficient βN\beta_N retains dependence on the undirected information gain ΓN\Gamma_N. As discussed in Appendix H, this dependence cannot generally be replaced by the value-weighted information gain ΓN⋆\Gamma_N^\star without additional assumptions. We would like to point out that our analysis first tightens SOMBRL's original regret bound from O~(ΓN3/2N)\widetilde{\mathcal O}(\Gamma_N^{3/2}\sqrt N) to O~(ΓNN)\widetilde{\mathcal O}(\Gamma_N\sqrt N), matching the information-gain dependence of Kakade et al. (2020) (in Appendix E). Relative to this tightened bound that we contribute, VDO-MBRL further reduces the leading term by ΓN⋆/ΓN\sqrt{\Gamma_N^\star/\Gamma_N} with an additional ΓN2\Gamma_N^2 term, which can yield a substantially tighter guarantee. Our analytical noisy TV example presented in Appendix G makes this concrete. We find that adding DD task-irrelevant channels increases the undirected information gain while leaving the value-weighted information gain independent of DD so that our complete bound becomes O~(ND+D2)\widetilde O(\sqrt{ND}+D^2), compared with O~(DN)\widetilde O(D\sqrt N). Once the additive correction of ΓN2\Gamma_N^2 is dominated, which occurs when NN is sufficiently larger than D3D^3 up to logarithmic factors, this yields an improvement of order D\sqrt D. Thus, retaining dependence on undirected information gain still allows a substantial improvement through task relevance. Our noisy TV experiments provide complementary evidence that this principle benefits the practical algorithms.

In summary, our theoretical analysis establishes when task relevance can substantially improve exploration guarantees, while our controlled distractor experiments investigate whether these benefits persist beyond the assumptions required for the analysis. We believe that together these provide a principled understanding of the advantages and limitations of value-directed exploration and empirical evidence that this approach remains effective with scalable neural-network implementations.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

  1. The method depends on the critic and a task specific warmup. It is unclear how the method behaves when no reward is found during warmup or the critic is wrong, in which case it may suppress needed exploration.

Response: Thank you for raising this concern. As discussed in our response to Reviewer DH7D (Weakness 1), we added experiments investigating the sensitivity of VDO-MBRL to warmup durations and robustness to inaccurate critics. The warmup sweeps are given in Figure 14 across three diverse environments: state-based Hopper Hop, Switchback Maze with a d=64d=64 noisy TV, and vision-based MiniWorld with a deterministic TV. We also compare against a SOMBRL baseline that no longer adds the exploration bonus after the warmup period to isolate the exploration benefits after warmup. The results show that the VDO methods remain effective across a range of warmup durations on the state-based tasks, including with zero warmup, and generally outperform the matched baseline. We also find that VDO-grad has greater warmup sensitivity on visual MiniWorld where longer initial exploration improves performance.

We additionally find that successful learning does not require discovering the task reward during the warmup period. As reported in Table 3 of Appendix J, only 3/25 VDO-grad and 2/25 VDO-spread seeds encounter task reward before the switch on the Switchback maze task, yet both variants subsequently learn effectively. This is possible because the extrinsic critic continues updating after the warmup period, which allows newly discovered rewards to inform future exploration. We further investigate the effect of inaccurate critics on value-directed exploration by setting Twarm=0T_{\text{warm}}=0 and initializing critics to favor either the noisy TV or an incorrect goal. The results in Figure 15 demonstrate that both VDO methods learn to reach the true goal across these conditions.

Overall, these results show that VDO-MBRL can continue exploring effectively even when the task reward has not been discovered during warmup or the critic assigns relevance to incorrect regions, indicating that such inaccuracies need not suppress exploration.

Reviewer_DH7DOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents, VDO-MBRL, that directs exploration toward dynamics uncertainty that affects task value. Its practical variants use either value gradients to weight next-state uncertainty or disagreement in the values of ensemble-predicted next states. A theoretical version with additional correction terms yields a regret bound involving value-weighted information gain while retaining dependence on global information gain. The practical methods use learned extrinsic critics with MBPO and DreamerV3. Experiments cover state and visual distractor tasks, twelve state-based DMC tasks, two visual DMC tasks, and two Atari games. The paper reports improved resistance to distractors and broadly comparable performance to SOMBRL on unmodified benchmarks.

Critical strengths
  1. The theoretical contribution clearly states the assumptions and its limitations. These qualifications make the theoretical contribution useful independently of the practical approximation.
  • Theorem 5.1 separates a value-weighted leading term from a global correction (lines 319–333)
  • The analysis accounts for planning under the mean model while return errors accumulate under true dynamics. (lines 287–295)
  • The discussion explicitly retains global calibration dependence for general kernels. (lines 334–341)
  1. The experiments distinguish task irrelevance from irreducible noise. Harder-maze comparisons against plain MBPO support a benefit beyond simply removing curiosity throughout training.
  • PointMaze varies distractor dimension, layout, and position. MiniWorld tests deterministic and stochastic screens (Figure 2). The deterministic screen tests whether learnable but irrelevant dynamics still distract the agent.
  • Methods share the learner and uncertainty estimator; the exploration coefficient is chosen for SOMBRL (a chosen baseline) and reused for VDO-MBRL.
Critical weaknesses
  1. Critic dependence and the limits of task-directed exploration are insufficiently tested. An inaccurate critic can weaken the value-directed dynamics bonus. This is a potential failure mode, not a demonstrated failure or proof that all exploration disappears.
  • Appendix I states that VDO “starts with the unweighted SOMBRL bonus during a task-specific warmup period” (Appendix I, lines 2366–2370). Warmup differs by variant: 50k/100k steps for spread/gradient on state distractors and 200k/300kon visual distractors (Table 1).
  • No warmup sweep or matched post-warmup bonus-switch-off experiment was found in the reported experimental sections. The appendix only reports variations on layouts and distractor placement (Appendix J).
  • Relevance to a learned current-policy critic is not necessarily relevance to future policy improvement. The paper motivates warmup as allowing this critic to learn, but does not establish a persistent-exploration safeguard when its estimates remain misleading. For changing rewards or dynamics (non-stationary environments), previously unimportant regions could become useful; adaptation in that setting is outside the stated guarantees.
  1. Broader robustness and computational benefit are less established. The evidence supports a controlled mechanism more strongly than a broad practical advantage over distractor-resistant methods.
  • Unmodified visual benchmarks comprise Finger Spin and Hopper Hop from DMC, and Robotank and Enduro from Atari. No explicit rationale for selecting these four tasks has been presented. The separate state-based DMC evaluation broadens coverage, but does not resolve visual-task representativeness. (Figure 10 and Appendix J)
  • The principal exploration baseline is SOMBRL. A direct test of the claimed distinction from noise filtering would include a cited noise-aware method, such as Mavor-Parker et al. (2022) or Curiosity in Hindsight (Jarrett et al., 2023), and a learning-progress method, such as Kim et al. (2020) or Hou et al. (2026).
  • Visual Hopper Hop takes 119.7 hours for VDO-spread, 78.4 for SOMBRL, and 79.6 for VDO-grad. The cost concern therefore applies more strongly to spread than gradient (Table 2).
Questions for authors
  1. How were warmup lengths selected? Do gains persist with a shared sweep, including zero warmup, and against switching off the SOMBRL bonus after the same warmup?
  2. What fraction of seeds has found task reward when VDO starts? What happens when reward discovery occurs after the scheduled switch? (Appendix I)
  3. Does a cited noise-aware method (Mavor-Parker et al. or Jarrett et al.) or learning-progress method (Kim et al. or Hou et al.) close the gap on the existing deterministic and stochastic MiniWorld tasks? Would QFLEX provide a useful complementary comparison on continuous control? Please explain compatibility or adaptation constraints rather than treating all these methods as interchangeable bonuses.
  4. Why were Finger Spin, Hopper Hop, Robotank, and Enduro selected? What selection criteria and resource constraints determined this visual subset?
  5. Please clarify whether the practical method has any safeguard against permanently overlooking regions assigned low relevance by an inaccurate critic. What would be needed for continued exploration in non-stationary environments? This is only for scope clarification, not a request for new algorithm targeting non-stationary environments.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you very much for reviewing our paper and for the constructive feedback. We are pleased to hear that you find the theoretical contributions independently useful and appreciate the positive assessment of our controlled distractor experiments. We address your comments below and hope this will help improve your evaluation of the paper.

Weaknesses:

  1. Critic dependence and the limits of task-directed exploration are insufficiently tested. An inaccurate critic can weaken the value-directed dynamics bonus, which is a potential failure mode.

Response: Thank you for highlighting this insightful point. Indeed, in our experiments we observed a dependence on critic learning to realize successful task-directed exploration. Upon your recommendation, we have run additional experiments to investigate this dependence in terms of performance sensitivity to different warmup values as well as robustness to inaccurate critics when value-directed exploration begins.

First, we have swept the value warmup steps TwarmT_{\text{warm}} for VDO-grad and VDO-spread and the suggested SOMBRL variant (we call this SOMBRL -> no bonus) on state-based Hopper Hop, state-based Switchback maze with a d=64d=64 noisy TV, and vision-based MiniWorld with the deterministic TV. We selected these environments to cover diverse exploration challenges across state-based and visual control where the weaker performance of exploration-free baselines suggests that warmup choices may have a greater impact on learning. Both VDO variants use the SOMBRL bonus for the first TwarmT_{\text{warm}} steps before switching to value-directed bonuses whereas SOMBRL -> no bonus does not add the exploration bonus after the same period. The results are given in Figure 14 of the revised paper and they show that both VDO variants remain effective across a range of warmup durations on the state-based tasks including with zero warmup. On MiniWorld, VDO-grad demonstrates greater sensitivity to the warmup duration and benefits from a longer initial exploration. Both variants generally outperform the SOMBRL -> no bonus baseline, which shows that their benefits cannot be attributed solely to the initial period of undirected exploration.

Next, we investigate robustness to inaccurate initial critics for the same Switchback maze when setting Twarm=0T_{\text{warm}}=0 so that value-directed exploration begins immediately. We compare randomly initialized critics with deliberately inaccurate critics that favor either the noisy TV or an incorrect goal, with the results shown in Figure 15. Despite differences in learning speed and variability, we observe that both VDO variants learn to solve the task effectively across these conditions, which suggests that inaccurate value estimates do not necessarily prevent successful exploration. We additionally evaluate VDO-grad with a quadratic uncertainty correction motivated by our theoretical bonus, which also learns effectively under these initial conditions.

Overall, these experiments demonstrate that VDO can remain effective across different warmup durations and recover from initially misleading value estimates. As pointed out, a remaining challenge is that even an accurate current-policy critic may downweight model uncertainty that matters under an improved policy or becomes relevant when changes in rewards or dynamics alter the optimal value function. Although we do not consider non-stationary environments in this work, VDO-MBRL's practical robustness suggests this may be a promising setting for future work. Furthermore, our theoretical bonus construction offers a principled approach to address this by using a quadratic uncertainty correction that maintains exploration even when value gradients are small.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

  1. The broader robustness and computational benefits are less established as the evidence supports a controlled mechanism more strongly than a broad practical advantage over distractor-resistant methods. There is no rationale for the selection of the unmodified visual benchmarks that are given and a more thorough experimental test would include a cited noise-aware method and a learning-progress method. Also there is a concern about computational cost for VDO-spread where visual Hopper Hop takes 119.7 hours to run.

Response: Thank you for these suggestions. We have expanded our evaluations to include additional visual control tasks, added comparisons with noise-aware and learning-progress methods, and updated the computational cost analysis.

We initially selected Finger Spin, Hopper Hop, and Enduro because the SOMBRL paper reports benefits from exploration on these visual tasks and also included Robotank after preliminary runs showed stable early learning. Given the substantially higher computational cost of visual experiments, we prioritized running experiments in these settings rather than evaluating the full benchmark suite. We have now added River Raid, Montezuma's Revenge (following Reviewer U8t1's suggestion), and Push Green to expand the evaluation to include seven unmodified visual tasks across DMC, Atari, and RoboDesk. The new results are given in Figure 4, which shows that VDO remains competitive across these tasks, with more sample-efficient learning on Montezuma's Revenge and performance improvements on Push Green.

We agree that adding comparisons with noise-aware and learning-progress methods can help clarify the benefits of value-directed exploration beyond filtering irreducible noise or rewarding improvements in dynamics prediction. Following your suggestion, we have added comparisons with Aleatoric Mapping Agents (AMA; Mavor-Parker et al., 2022) and Learning Progress Monitoring (LPM; Hou et al., 2026) along with additional exploration baselines in Figure 12 of Appendix J. We evaluate these additional baselines on state-based Hopper Hop, state-based Switchback maze with a d=64d=64 noisy TV, vision-based Finger Spin, and vision-based MiniWorld with the deterministic TV. We selected Hopper Hop, Switchback Maze, and MiniWorld for the exploration challenges described above and included Finger Spin as a simpler visual control task to verify that the additional baselines can learn effectively in a less demanding setting for exploration. We find that VDO achieves higher average returns than AMA and LPM on Hopper Hop and MiniWorld, while AMA is competitive with VDO on Switchback and both baselines achieve high returns on Finger Spin. AMA's strong performance on Switchback is consistent with its objective of discounting prediction error from irreducible TV noise and LPM is also less susceptible to the noisy TV since unpredictable transitions offer little learning progress. However, LPM's bonus appears insufficient to sustain the exploration needed to reliably navigate this challenging maze and reach the goal. On the MiniWorld task with a deterministic TV, both AMA and LPM are more strongly drawn to the TV whose dynamics are learnable despite being task-irrelevant. AMA's aleatoric correction does not suppress this distractor and LPM can reward improvements in predicting its dynamics, leaving both methods susceptible to task-irrelevant exploration. These results therefore highlight the benefit of directing exploration toward uncertainty that matters for task value, beyond noise filtering or learning progress alone.

Regarding the computational cost concern, we have updated Table 2 with revised measurements for VDO-spread on the vision-based tasks that now use 16 Monte Carlo samples rather than the previous 64 when estimating ensemble members' next state values. This reduces training time by 32% for VDO-spread on Hopper Hop and results in broadly comparable computation time with VDO-grad and SOMBRL that come with very minor changes in performance, which are reflected in the updated Figure 4.

Questions:

  1. How were warmup lengths selected? Do gains persist with a shared sweep?

Response: We selected warmup lengths through per-task tuning based on preliminary performance when sweeping over candidate warmup durations. To make this more transparent, we have extended the sweeps and provide the results on three different tasks in Figure 14 of the updated PDF. As discussed in our response to Weakness 1, VDO-MBRL generally outperforms the matched bonus-switch-off baseline and remains effective even with zero warmup on state-based tasks, though VDO-grad benefits from a longer warmup on visual MiniWorld tasks.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

  1. What fraction of seeds have found task reward when VDO starts? What happens when reward discovery occurs after the scheduled switch?

Response: We give the fraction of seeds that have discovered task reward before value-directed exploration begins in Table 3 of Appendix J. In challenging noisy TV settings, reward discovery often occurs only after the switch. For example, on Switchback only 3/25 VDO-grad and 2/25 VDO-spread seeds encountered task reward during the first TwarmT_{\text{warm}} steps, while on MiniWorld with a stochastic TV no variant discovered reward before the switching point. Despite this, both VDO-MBRL approaches subsequently learn effectively, with the extrinsic critic continuing to update as new rewards are encountered and informing value-directed exploration. These findings demonstrate that, in practice, VDO-MBRL can learn well even if task reward has not been discovered during the initial undirected exploration period.

  1. Does a cited noise-aware method or learning-progress method close the gap on existing deterministic and stochastic MiniWorld tasks? Would QFLEX provide a useful complementary comparison on continuous control?

Response: Following your suggestion, we have added comparisons with the noise-aware method AMA (Mavor-Parker et al., 2022) and the learning-progress method LPM (Hou et al., 2026) on state-based Hopper Hop, state-based Switchback maze with a stochastic noisy TV, vision-based Finger Spin, and MiniWorld with a deterministic noisy TV (results are given in Figure 12 in Appendix J). As discussed in our response to Weakness 2, neither method closes the performance gap with the deterministic noisy TV where the distractor dynamics are learnable but task-irrelevant. In the Switchback maze with a stochastic noisy TV, we find that AMA performs competitively with VDO, and LPM achieves lower returns while still spending less time in the TV cell. As both state-based and vision-based stochastic noisy TV environments introduce irreducible, task-irrelevant uncertainty, these mazes provide the same underlying noise-filtering challenge. We therefore selected stochastic-TV Switchback and deterministic-TV MiniWorld as complementary tests for irreducible noise and learnable but irrelevant dynamics to assess this within our computational budget.

QFLEX (Wei et al., 2026) could provide an interesting complementary method for high-dimensional continuous control, however it addresses a different exploration challenge. More specifically, QFLEX uses value-guided flows to sample promising actions in high-dimensional action spaces, whereas VDO-MBRL uses task values to prioritize which uncertain aspects of the dynamics are worth exploring. These approaches therefore operate at different levels, with QFLEX improving action selection and VDO-MBRL directing exploration toward task-relevant dynamics uncertainty. The methods could be combined using VDO's value-directed uncertainty bonus to train the critic guiding QFLEX's flow policy, which would combine high-dimensional action selection with task-relevant exploration of uncertain dynamics. We leave this for future work though and prioritize comparisons with methods that more directly investigate uncertainty-driven exploration, including AMA and LPM for noise-aware and learning-progress exploration, as well as H-UCRL, LOVE, OWM, and MaxInfoRL for alternative approaches to model-based optimism and uncertainty quantification.

  1. Why were Finger Spin, Hopper Hop, Robotank, and Enduro selected? What selection criteria and resource constraints determined this visual subset?

Response: As discussed in our response to Weakness 2, we initially selected Finger Spin, Hopper Hop, and Enduro because the SOMBRL paper reported benefits from exploration on these visual tasks, and included Robotank after preliminary experiments showed stable early learning. Since vision-based training is substantially more computationally expensive than state-based, with a single Hopper Hop run requiring approximately 80 hours (Table 2), our computational budget limited the initial selection to tasks where exploration was likely to matter and multi-seed evaluations were feasible. We have since expanded the visual benchmark with River Raid, Montezuma's Revenge, and RoboDesk Push Green for a total of seven unmodified visual tasks (the results are given in Figure 4).

AuthorsOpenReview ↗

Rebuttal by Authors Continued

  1. Please clarify whether the practical method has any safeguard against permanently overlooking regions assigned low relevance by an inaccurate critic. What would be needed for continued exploration in non-stationary environments?

Response: The full theoretical bonus in Equation 15 provides a principled safeguard against overlooking uncertain regions when their estimated value sensitivity is small by including value-independent quadratic uncertainty corrections, which incentivize exploration where there is remaining model uncertainty. Under our calibrated GP assumptions, these terms also support the theoretical optimism guarantee. While the practical VDO variants do not inherit this guarantee, our new experiments provide encouraging evidence of robustness to inaccurate critics. As described in our response to Weakness 1, Figure 15 shows that both variants learn successfully even with zero warmup and inaccurate initial critics. We additionally evaluate VDO-grad with a quadratic uncertainty correction motivated by the theoretical bonus, which also learns effectively across these conditions.

Our paper considers stationary environments where the underlying rewards and dynamics remain fixed and inaccurate relevance estimates can improve with additional experience. In non-stationary environments, changes in rewards or dynamics can alter which regions are relevant to the task and invalidate previously learned estimates. To adapt to such changes, VDO-MBRL would need to continue exploring regions assigned low relevance by the current critic, potentially by retaining or reintroducing the value-independent SOMBRL bonus. A theoretical extension would require confidence bounds that remain calibrated under changing dynamics and a regret analysis accounting for the environmental changes. The robustness demonstrated by our critic-initialization experiments suggests this is a promising direction for future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of Improvements

We would like to thank the reviewers again for their time and valuable comments. We have uploaded a revised version of the manuscript incorporating the following main changes highlighted in blue:

  • Added six exploration baselines, including the noise-aware method AMA, the learning-progress method LPM, MaxInfoRL, and the optimistic model-based methods H-UCRL, LOVE, and OWM, evaluated on state-based and visual tasks with and without noisy TV distractors (Appendix J, Figure 12).
  • Expanded the unmodified visual benchmarks from four to seven tasks by adding River Raid, Montezuma's Revenge, and RoboDesk Push Green, bringing the total number of unmodified benchmarks to 19 (Figure 4).
  • Added sensitivity analyses for the exploration coefficient, the warmup duration with a matched SOMBRL->no-bonus baseline, and misleading critic initializations without warmup, together with the fraction of seeds that discover reward before value-directed exploration begins (Appendix J, Figures 13–15, Table 3).
  • Reduced the computational cost of visual VDO-spread and updated the runtime comparison (Table 2).
  • Expanded the related work to discuss approaches that propagate model uncertainty to value estimates or direct exploration toward decision-relevant information, and clarified that the reduction from the value-weighted information gain is instance-dependent (Section 5).
  • Added an analysis of SOMBRL's ensemble disagreement over training, showing how its decay depends on the number of distractor dimensions (Appendix J, Figure 16).

We believe that these changes have substantially enhanced the quality of the paper and invite the reviewers to reconsider our submission, potentially adjusting their scores. If there are any remaining concerns, please let us know and we will address them!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.