Robust-Power-HOOT: Monte Carlo Tree Search for Continuous Stochastic MDPs Under Model Ambiguity

We study Monte Carlo Tree Search (MCTS) for continuous-action stochastic MDPs under model ambiguity, where the model used for planning may differ from the execution environment. Existing methods typically address at most two of three challenges: continuous-action exploration, stochastic value aggregation, and distributional robustness. We propose Robust-Power-HOOT, which combines HOO-style continuous-action search, polynomial exploration bonuses, power-mean backups, and conservative distributionally robust Bellman updates. Exact robust optimization at every backup is costly in large search trees; instead, we use closed-form upper bounds on robust penalties to compute conservative lower estimates of robust values. We prove that the root estimate converges at rate $O(n^{-\zeta })$, where $\zeta \in (0,1/2)$, to the projected finite-horizon value induced by these lower-bound updates. We decompose the gap to the exact robust objective into finite-horizon truncation, state abstraction, lower-bound approximation, and finite-sample errors, and identify regimes in which the penalty bounds are tight. Across seven continuous-control benchmarks with controlled planning–execution mismatch, the method often degrades less as execution noise increases.