- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to adaptively balance the benefits of long-horizon planning with accumulating model errors in model-based reinforcement learning. For each candidate action, their proposed method uses split-conformal calibration to construct a calibrated return interval over plausible dynamics models for each planning horizon level and executes the action whose lower bound exceeds the upper bound of all others. When no such action exists, on each iteration, they predict the influence of extending the rollout and whether that will allow any action to be certified for stability enough to justify the additional interval uncertainty and the cost of computation. The resulting algorithm gives an adaptive planning horizon for each state that reflects the uncertainty in estimated returns and separation between competing actions. This is evaluated on synthetic tasks that are designed to isolate the horizon selection rule.
Critical strengths
- The principle behind this paper that planning depth should depend on the uncertainty that matters for selecting the best action is interesting and well aligned with decision directed planning. Using calibrated return intervals to certify decision stability makes sense as a stopping criterion.
- The experiments are designed to vary the action-value margins, uncertainty, and delayed rewards while keeping the predictions fixed across methods. This is a well controlled analysis that demonstrates how the appropriate planning horizon depends on how clearly the candidate actions can be differentiated in terms of their expected returns.
Critical weaknesses
- The theoretical guarantee relies on conformal calibration using the environment return targets for all candidate actions. In the DMC experiments, these are obtained by resetting the simulator and executing each candidate action separately, which is generally infeasible for realistic online RL. In addition, there is limited empirical validation as most experiments use synthetically generated return predictions rather than a learned world model. This method would be most useful if it could be effectively applied in more practical settings even with some approximations.
- Proposition 1 appears to bound the probability of incorrectly certifying an action but does not guarantee the correctness of decisions made through the fallback mechanism. In the experiments, only 42% of decisions are certified and 50% use the fallback, making this an important gap to address.
Questions for authors
- How could the method be calibrated in a realistic RL setting? How could calibration be maintained as the world model and policy change? Perhaps an offline-to-online RL setting where the offline data is used for initial calibration could be possible.
- It would be very interesting to see if the adaptive decision-directed planning mechanism also improves training sample efficiency compared with fixed-horizon or uncertainty-based planning, could this be shown?
- How much improvement comes from the learned continuation predictor?