- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
this paper establishes a robust DPP for finite-horizon MDPs under general set-valued transition kernels and randomized policies and proposes a tractable directional actor–critic algorithm.
Critical strengths
the proofs are sound and rigorous, algorithms are clearly motivated and the numerics are extensive and informative. moreover the contribution to the literature is non-trivial
Critical weaknesses
the paper is a bit abstract, and while i have been able to follow the general ideas, i believe the target audience of this paper may be limited at iclr.
Questions for authors
do the authors have a route forward to incorporate function approximation into this work? can they convince me that it wouldn't make the current obsolete?
response to Reviewer Pxv1
Thank you for the review. We answer both points below. New proofs are in Appendix G of the revised PDF, and new experiments are in our anonymous repository.
1. “Abstract nature and limited ICLR audience.”
Existing robust policy-gradient methods can use the gradient from one selected worst-case kernel. When several worst-case laws are optimal, this gradient can point in the wrong direction. Theorem 3.5 give the exact one-sided derivative by considering all optimal laws. This issue occurs in our experiments. At the exact dual optimum, 13–35% of transport cells have multiple optimal destinations (Appendix E, Table 13). In 1,332 deliberatly constructed tie tests across 12 shared parameter settings, the selected-kernel gradient wrongly predicted ascent 92 times (6.9%), reaching 39.6% in one setting. Our derivative gave the correct sign in every test.
The correct direction can also improve performance. Using the same exact-objective line search, our method achieves higher final robust value than selected-kernel policy gradient in 10 of 12 settings (paired 95% intervals). It also beats an RNAC-style baseline based on Zhou et al. (2023) in 10 of 12 settings and nominal continuation in all 12 settings.We also implement a sampled actor–critic with a sensitivity critic on LQ control, pendulum and lost-sales inventory tasks.
2. “Route to function approximation. Would it make this work obsolete?”
No. We already use function approximation through a neural policy, robust value critic and sensitivity critic. Theorem 3.5 tells what sensitivity these components should estimate. It applies to compact, potentially continuous spaces under the stated regularity assumptions. The exact ascent guarantee, however, requires a finite model.
We also made improvements toward larger problems.First, the exact ascent of Algorithm 1 (Theorem 3.8, Appendix C) is now much cheaper. Proposition G.1 replaces the exponential enumeration of near-active branches with one polynomial-size LP and Proposition G.2 replaces the global constants C1,C2 of Theorem C.1(iv) with a backtracking step. On the benchmarks of Table 3 (Appendix E.1.2), a run now takes less than 0.1 s instead of up to 8 minutes. The method now runs with up to 30 states where listing branches is infeasible and the objective never decreased in any of our runs (see repository). Second, Proposition G.3 bounds the error from estimating the reference model by O(N−1/2), using the dual of Proposition 3.2. An update that improves the estimated objective improves the true objective up to an error of twice this bound. Appendix E.1.3 shows that the error decreases as the sample size grows. Third, if the value critic has error et at stage t, the total propagated value error is at most ∑tet. Apendix E.1.3 also measures the effect of critic-label noise.
The exact method therefore gives us a reliable reference for developing and testing approximate methods