- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VDO-MBRL, an optimistic model-based exploration method that weights model epistemic uncertainty by its estimated effect on future value, replacing the value-agnostic uncertainty bonus of SOMBRL. Two practical bonuses are introduced: a gradient-based variant that weights ensemble uncertainty along value-gradient directions, and a gradient-free variant that measures the spread of predicted next-state values. Theoretically, under GP/RKHS assumptions, the authors define a value-weighted information gain and derive regret bounds whose leading term scales with it. They also give a tightened SOMBRL bound and a lower bound showing that calibration cannot depend on alone. Empirically, VDO-MBRL outperforms SOMBRL on author-constructed noisy-TV variants of PointMaze and MiniWorld, and performs comparably on 12 state-based DMC tasks and four vision-based DMC/Atari tasks.
Critical strengths
- Coherent theoretical treatment within the GP/RKHS framework. The paper provides an upper bound, a sharpened SOMBRL analysis, and a lower bound clarifying what cannot be improved. This is the main source of value in the submission. However, its relevance to the evaluated algorithm is limited, since the practical method departs substantially from the analyzed bonus (see Weaknesses).
Critical weaknesses
-
The empirical evidence rests on author-constructed environments. All gains appear only on six self-designed noisy-TV settings. Across 16 standard benchmarks, VDO-MBRL shows no improvement over SOMBRL while costing – more wall-clock time. SOMBRL is the only exploration baseline. Noise-robust exploration methods that the paper itself cites (Mavor-Parker et al., 2022; Jarrett et al., 2023; Hou et al., 2026) are absent, so it remains unclear whether value direction is needed at all.
-
There is a large theory–practice gap. The guarantees apply to the full bonus with second-order and transfer corrections under GP calibration. The implementation keeps only the first-order term with ensembles, and adds SOMBRL warmup, running-mean normalization, clipping, and annealing. The theory therefore does not cover the evaluated algorithm.
-
The method relies on its own tuning. VDO uses task-specific warmup lengths (50k–1M steps), with separate values for VDO-grad, which SOMBRL does not require. This makes it hard to attribute the gains to the mechanism rather than to tuning.
Questions for authors
-
Can you compare against noise-robust exploration methods (e.g., Mavor-Parker et al., 2022; Jarrett et al., 2023; Hou et al., 2026) on the noisy-TV tasks? Without them, the necessity of value direction is not established.
-
Can you report ablations that remove the warmup, normalization, and clipping from VDO, or alternatively add the same components to SOMBRL? This would isolate the contribution of value weighting.
-
Ensemble disagreement on irreducible Gaussian noise should decay with data. Why does SOMBRL remain attracted to the TV? Please report SOMBRL's bonus magnitude at the TV cell over training, so that ensemble miscalibration can be ruled out.
-
How does the method relate to Information-Directed Sampling (Russo & Van Roy) and the Uncertainty Bellman Equation (O'Donoghue et al., 2018), which also direct exploration by value-relevant information? Neither is discussed.
-
Since the weights are not bounded by one (Eq. 81), is not guaranteed. Can you estimate both quantities on the evaluated tasks to show that the claimed reduction actually occurs?
Suggestion:
- Evaluation on established hard-exploration benchmarks (e.g., Montezuma's Revenge) would show whether the benefit extends beyond constructed distractors.
5 comments
Rebuttal by Authors Continued
Finally, regarding compute time, we would like to point out that the multipliers in Table 2 are measured relative to the exploration-free MBPO or DreamerV3 backbone rather than SOMBRL, which itself requires - the backbone training time. On the visual tasks, where training is most expensive, VDO-grad requires training time comparable to SOMBRL on most tasks, so the more consistent performance of value-directed exploration comes at little additional cost over undirected exploration. On the state-based tasks, VDO adds a moderate overhead relative to SOMBRL, although all runs complete in under 14 hours. As discussed in our response to Reviewer DH7D (Weakness 2), we have also reduced the cost of visual VDO-spread by using fewer Monte Carlo samples, bringing its training time on Hopper Hop to within 4% of SOMBRL.
- There is a large theory-practice gap as the guarantees apply to the full bonus with second-order and transfer corrections under GP calibration while the implementation keeps only the first-order term with ensembles, and adds SOMBRL warmup, running-mean normalization, clipping, and annealing.
Response: We agree that the regret bound applies to the full theoretical bonus under GP calibration and does not cover the practical variants. We would like to clarify that the purpose of the theory is not to provide guarantees for neural network implementations in high-dimensional control settings where calibrated confidence bounds on the dynamics are generally unavailable, but rather to identify and address a key obstacle to efficient exploration, namely that model uncertainty can be largely irrelevant to the task. In a tractable setting with calibrated models, we derive an exploration bonus from how dynamics errors affect future values and establish a regret bound that characterizes when task relevance improves exploration efficiency. These insights guide the design of our practical algorithm, even though the formal guarantees do not extend to it.
As discussed in our response to Reviewer P4NP (Weakness 2), the calibrated GP assumption is needed to construct confidence bounds on dynamics prediction errors, which the optimism guarantee relies on. This gap between the calibrated model analysis and scalable implementation is shared by H-UCRL and SOMBRL, which derive their regret bounds under the same calibration assumption and evaluate practical variants that replace GP uncertainty with neural network ensemble disagreement. We follow the same approach, which preserves the central role of epistemic dynamics uncertainty in guiding exploration. Our practical bonus retains the first-order value-sensitivity term and omits the second-order and transfer corrections required for the optimism guarantee, since these add complexity and compute time. We also observe that VDO-grad with a quadratic uncertainty correction motivated by the full bonus learns comparably to the first-order version (Figure 15), so we retain the simpler bonus.
The remaining implementation components mentioned stabilize training without changing which uncertainty the bonus prioritizes. More specifically, running-mean normalization and annealing are shared with SOMBRL (Appendix I) and rescale the bonus by a common factor at each update, which preserves the relative value weighting across transitions and clipping limits the effect of outliers from the learned critic. We also find that the SOMBRL warmup is not required on the state-based tasks, where both VDO variants remain effective with zero warmup (Figure 14), as discussed in our response to Weakness 3. In summary, our theoretical analysis establishes when task relevance can improve exploration guarantees and our experiments test whether these benefits persist in practical settings that fall outside the assumptions required for the analysis.
Rebuttal by Authors Continued
- The method relies on its own tuning as VDO uses task-specific warmup lengths with separate values for VDO-grad which SOMBRL does not require. So it is hard to attribute the gains to the mechanism rather than to tuning.
Response: Thank you for pointing this out. We agree that it is important to separate the contribution of value weighting from that of warmup tuning. To this end, we have added warmup sweeps for both VDO variants on state-based Hopper Hop, the Switchback maze with a noisy TV, and visual MiniWorld with a deterministic TV (Figure 14). On the state-based tasks, both VDO approaches remain effective across all warmup durations, including zero warmup, so their gains there do not depend on the chosen warmup values, while VDO-grad benefits from a longer warmup on visual MiniWorld. We also compare against a matched SOMBRL->no-bonus baseline that uses the SOMBRL bonus for the same warmup period and then stops adding the bonus. Both VDO methods generally outperform this baseline, which also shows that their performance improvements cannot be attributed to the initial period of undirected exploration alone. SOMBRL does not need a warmup since its bonus does not depend on a learned critic. Moreover, we observe that task reward is usually not discovered before the switch (Table 3) and both VDO methods learn effectively with zero warmup even when their initial critics are deliberately misleading (Figure 15). VDO-MBRL therefore does not rely on an accurate critic when value-directed exploration begins, which limits its sensitivity to the warmup length.
Finally, we also select the exploration coefficient based on SOMBRL's performance and reuse the value for VDO-MBRL without further tuning. Our new sweep in Figure 13 shows that both VDO methods remain effective across a range of coefficients, while SOMBRL achieves similar returns at every value on the Switchback maze, which are all below the VDO variants (see our response to Reviewer P4NP, Question 5). Together, these ablations support that the gains of VDO-MBRL arise from the value-directed weighting of model uncertainty rather than from tuning.
Questions:
- Can you compare against noise-robust exploration methods (e.g., Mavor-Parker et al., 2022; Jarrett et al., 2023; Hou et al., 2026) on the noisy-TV tasks? Without them, the necessity of value direction is not established.
Response: Thank you for the suggestion. Yes, as described in our response to Weakness 1, we have added the noise-aware method AMA (Mavor-Parker et al., 2022) and the learning-progress method LPM (Hou et al., 2026) as baselines. We find that AMA performs competitively with VDO on the stochastic noisy TV, however both AMA and LPM achieve substantially lower returns than VDO on MiniWorld with the deterministic noisy TV. These results show the benefit of value direction when task-irrelevant uncertainty is reducible, which is a case that noise-robust exploration is not designed to address. We select AMA as the noise-aware baseline since it integrates with our probabilistic dynamics models, whereas Curiosity in Hindsight (Jarrett et al., 2023) would require a separate representation learning architecture.
- Can you report ablations that remove the warmup, normalization, and clipping from VDO, or alternatively add the same components to SOMBRL? This would isolate the contribution of value weighting.
Response: Yes, we have included ablations on the warmup period as described in our response to Weakness 3 and given in Figure 14 of the updated paper. The results show that both VDO methods remain effective even with zero warmup on the state-based tasks and generally outperform a matched SOMBRL->no-bonus baseline. For normalization and annealing, our experiments already follow the second suggestion: as described in Appendix I, SOMBRL uses the same running-mean normalization and uncertainty-based annealing as VDO. With these components shared, the main remaining difference between the two methods is whether model uncertainty is weighted by its relevance to task value, so the comparison with SOMBRL isolates this contribution. The only VDO-specific component is clipping, which caps the value-directed signal at a running 99th percentile to limit outliers from the learned critic. SOMBRL does not compute this critic signal and the clip affects only the largest values, leaving the value weighting of all other transitions unchanged.
Rebuttal by Authors
Thank you very much for your careful review of our paper and the specific suggestions for strengthening it. We are pleased to hear that you find our theoretical treatment coherent, including the regret upper bound, the sharpened SOMBRL analysis, and the lower bound clarifying what cannot be improved. We address your comments below and hope this will help improve your evaluation of the paper.
Weaknesses:
Response: Thank you for the opportunity to address this point. We agree that evaluations beyond our constructed distractor settings and comparisons with other exploration approaches are important for establishing the broader benefits of VDO-MBRL. To this end, we have additionally evaluated VDO-MBRL on three new visual tasks, specifically River Raid, Montezuma's Revenge, and RoboDesk Push Green, bringing the total number of unmodified benchmarks considered to 19 (consisting of 12 state-based DMC tasks and 7 vision-based tasks across DMC, Atari, and RoboDesk). On Montezuma's Revenge, an established hard-exploration benchmark, both VDO variants learn more efficiently than SOMBRL and on Push Green both achieve higher returns than SOMBRL (Figure 4), showing that value-directed exploration can also benefit tasks without constructed distractors. Across all unmodified benchmarks, VDO-MBRL consistently remains among the strongest methods, although many of these tasks pose limited exploration challenges, as indicated by the strong performance of MBPO on most state-based DMC tasks (Figure 11). For this reason, we focused our visual evaluation on tasks where prior work reported benefits from exploration, as discussed in our response to Reviewer DH7D (Weakness 2). Our theory predicts the largest improvements when model uncertainty contains a substantial task-irrelevant component, so that the undirected information gain grows while the value-weighted information gain remains small (Appendix G). The noisy TV settings instantiate this structure directly by letting the agent choose between task-irrelevant uncertainty and task reward, while controlling whether the distractor is stochastic or learnable and how many dimensions it contains. These settings therefore provide a targeted test of the mechanism that complements, rather than replaces, the standard benchmarks for showing the benefits of VDO-MBRL.
To address the point about limited baselines, we have also added the following six new exploration baselines: the noise-aware method AMA (Mavor-Parker et al., 2022), the learning-progress method LPM (Hou et al., 2026), MaxInfoRL, and optimistic model-based methods H-UCRL, LOVE, and OWM (as described in Appendix J with results given in Figure 12). We evaluate these baselines on state-based Hopper Hop, the Switchback maze with a d=64 stochastic noisy TV, visual Finger Spin, and visual MiniWorld with a deterministic TV, and we select AMA as the representative noise-aware method since Curiosity in Hindsight (Jarrett et al., 2023). On the stochastic-TV Switchback maze, AMA performs competitively with VDO, which is consistent with its objective of discounting irreducible noise. However, on deterministic-TV MiniWorld, both AMA and LPM achieve substantially lower returns than VDO, which is consistent with the learnable TV dynamics not being discounted as aleatoric noise by AMA and still providing learning progress for LPM. VDO additionally outperforms the optimistic baselines on all evaluated tasks and MaxInfoRL on both noisy TV tasks, as discussed further in our responses to Reviewer P4NP (Weakness 4 and Question 1). Overall, VDO-MBRL is the only method that performs consistently well across these tasks, which demonstrates that value direction provides benefits beyond noise filtering and learning progress, particularly when task-irrelevant uncertainty is reducible.