- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies distributionally robust average-reward reinforcement learning with a generative model. It covers two rectangular sets and four uncertainty metrics for robustness considerations. The authors establish sufficient conditions under difference metrics to guarantee the existence of an optimal Bellman equation and a constant optimal robust gain. By estimating the nominal transition kernel with samples from a generator, this paper provides a finite sample complexity bound for obtaining a robust -optimal policy under average reward criteria. The bound depends on the the span of the optimal bias function, which is further upper bounded under different settings.
Critical strengths
-
This paper gives a finite sample complexity analysis under a weak communication assumption, which is weaker than previous work that assumes irreducibility, ergodicity, or unichain, enabling more MDPs to be consider under this setting.
-
This paper targets a broader scope of settings in robust RL, by handling both - and -rectangular sets under KL, divergence, and TV, and Wasserstein distance.
Critical weaknesses
-
In section 4.1, the main theoretical findings for the TV and Wasserstein distance are a natural extension of prior work. While for KL and divergence, assumption 1 seems restrictive. For example, by the definition of , it satisfies , which may have negligible value when the state space is large. In this case, can be close to 0 and the uncertainty set considered may be too restrictive in a robust RL problem.
-
The experimental results are limited. They validate the rate of decrease in gain estimation error under a constructed MDP instance. However, since this paper focuses on the robust RL setting, an experiment comparing with the corresponding non-robust algorithm would better fit the purpose of this paper.
Questions for authors
-
Is the extra factor and assumption 1 necessary for policy learning, particularly for TV/Wasserstein uncertainty?
-
In algorithm 1, how to solve for the empirical optimal discounted Bellman equation for both - and -rectangular sets? What is the compuational complexity?
Thank you again for your detailed review. We hope our clarifications are helpful and look forward to hearing your comments. We would be happy to address any remaining concerns.