- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper takes the next step in the evolution starting from the Hierarchical Optimistic Optimization (HOO) to HOOT (application of HOO to trees), POLY-HOOT (HOOT with polynomial bonus term and proof of convergence) and considers the setting of robust optimization where the transition dynamics may be chosen by an adversary. It combines the idea of power-mean backups of Stochastic-Power-HOOT which works in continuous spaces with the robust optimization approach from Robust-Power-UCT which only works in discrete spaces. Thus it fills an important gap in the literature. The paper provides reasonable theoretical guarantees and empirical results in multiple domains under different divergence measures.
Critical strengths
The paper addresses the important problem of solving robust stochastic MDPs and gives an algorithm with theoretical guarantees.
The appendix and the main text provide the full proofs of the claims.
The approach is illustrated in multiple domains with good performance.
Critical weaknesses
Unfortunately, the paper is difficult to read by wider audiences than the experts. Some of these issues can be easily fixed by introducing the main ideas without overwhelming the reader with jargon. For example, the abbreviation of HOO was never explained, and the introduction has loss equations with terms whose meanings were left to the reader's imagination. The introduction should be more basic and should not assume expert knowledge. The equations are not needed at this stage. More interesting would have been an application-oriented motivation. Why is this generalization important in practice apart from its natural mathematical advance and elegance? The authors can see the original HOO paper for a good example.
Most of the paper is filled with claims and proofs which would work well in a 'theory conference' or in a journal. I suggest that the authors focus on the intuition behind the theorems and proofs in the main paper and relegate most proofs to the appendix.
The paper also spends little time in interpreting the empirical results. While they generally seem strong, it is not clear what to make of the results of different divergences, what trends are there if any, and how to explain them. The readers are looking for insights, not just some pretty curves.
Questions for authors
-
Can you provide a summary of the different claims and the novelty of proof techniques (if any) at a high level before introducing all the formal machinery? is it basically a composition of familiar proof methods and algorithmic tools?
-
Can you provide a more robust discussion of the experimental results potentially using some ablation studies?
-
Can your theory support the differences in the empirical performance of the different divergences?
Our response to Reviewer 47Ms
We want to thank the reviewer for the careful reading and for seeing the gap that this paper fills.
About presentation and motivation. It is our fault: we used acronyms and equations in Section 1 before defining them, and we have fixed this. On the practical motivation, we want to position the story behind the work. We started from two lines of work. Continuous-action tree search (HOO, POLY-HOOT, Stochastic-Power-HOOT) plans as if the simulator were exact, and robust tree search (Robust-Power-UCT) plans for the worst model but needs a finite action set. In control, the actions are torques or steering commands, a grid over them grows as km (about 108 on Humanoid), and the simulator never matches the real system exactly (Tobin et al., 2017; Peng et al., 2018). So we explicitly ask whether one tree search can handle continuous actions, stochastic rollouts, and model mismatch at once, with a guarantee.
Novelty of the proof (Q1). Yes, we largely compose known tools, and we now open Section 5 with a paragraph that names them. We needed three additional steps. Our closed-form backups estimate a conservative value, and its Bellman operator need not be monotone or a contraction, so we propagate errors only through the exact robust operator (Prop. 4, Cor. 1). Our plug-in penalty depends nonlinearly on child values that are still converging, so we prove its concentration separately (Lemma 1, App. B). And continuous successor states almost never repeat, so we aggregate them and bound the projection bias (Prop. 1).
Ablations (Q2). We now discuss the checks of Appendix H.1 in Section 6. The clearest one is Table H.15. When we replace the closed-form W1 backup by the exact one, the return rises on CartPole (7.4→19.1) and Pendulum and moves both ways on the MuJoCo tasks. We read this as a sign that our approximation has a cost on some tasks, although this experiment alone cannot show that penalty tightness causes it.
Divergences (Q3). Partly. Prop. 4 says that the approximation error grows with the looseness of the bound, and our diagnostics follow this order. χ2 gives the tightest bound (0 to 1% gap, Table H.14) and the smallest worst-case swing over the radius sweep (32%, against 40% for TV and 50% for W1, Table H.17). Our theory cannot predict which divergence gives the highest return, since that depends on how well the ambiguity set matches the real mismatch.
We would be grateful if the reviewer could look again at Sections 1, 5, and 6.
Our references for comments above. Tobin, J. et al. Domain randomization for transferring deep neural networks from simulation to the real world. IROS, 2017. Peng, X. B. et al. Sim-to-real transfer of robotic control with dynamics randomization. ICRA, 2018.