- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies discounted tabular distributionally robust reinforcement learning with -divergence uncertainty sets under the model-based setting. The paper analyzes the uncertinaty for both SA- and S-rectangular uncertinty sets. The claimed total sample complexities are that closes the uncertinaty size regime which matches the minimal optimality for all regimes and eliminate the dependence. The proposed analysis uses the second moments of adversarial density ratios to obtain a variance-sensitive Bellman error bound.
Critical strengths
-
The problem close the parameter regime gap Previous work has achieved minimax optimality in both small- and large-parameter regimes; however, these results involve terms of the form . This paper addresses this gap by introducing a variance-sensitive framework that exploits the second-moment structure of adversarial density ratios. This framework enables us to control robust Bellman estimation errors and their propagation while maintaining a linear dependence on the divergence.
-
The technique novelty is clear. The technique contribution variance-sensitive analysis of the -robust Bellman operator. The analysis exploits the second-moment structure of worst-case likelihood ratios to relate estimate error to the adversarial variance and the uncertainty radius, providing a promising route to the claimed linear dependence on the radius. Extending this approach to S-rectangular uncertainty sets is also technically meaningful.
Critical weaknesses
My main concern is an unjustified change of probability measure in Appendix B, page 21. Lines 1099–1105 bound the variance of the normalized dual function under the interpolated nominal distribution , whereas the Bernstein relateds steps in lines 1113-1118 requires the variance under the nominal transition kernel . Lines 1120-1121 then substitute the former bound for the latter without establishing a comparison between these two variances. At the empirical endpoint , both the dual threshold and its normalizer are defined under , so the preceding KKT identifier dnes not directly justify the required bound under , which affects the variance-sensitive estimation bound used to support the main upper bound result.
Questions for authors
- What inequality or high-probability argument justifies transferring the variance bound from to , particularly at , where the dual threshold and normalizer depend on the empirical samples?
- Can you provide a more detailed derivation and propagate any additional error terms through the sample-complexity analysis, showing that the stated rates remain valid without introducing dependence on the minimum positive transition probability? It is unclear for me how the minimal support probability parameter is eliminated.
- In my opinion, the primary contribution of this paper lies in establishing convergence guarantees for the "intermediate-gap" regime where , while simultaneously eliminating the dependence on . Including a table in the "Related Work" section that compares the different regimes would significantly enhance the paper's readability.
If the authors resolve this issue and show that the stated convergence rates are preserved, I would reconsider my assessment and consider raising my score.