Minimax-Optimal Robust Reinforcement Learning with ${\chi }^2$-divergence Uncertainty Set under $\mathcal{S}\mathcal{A}$- and $\mathcal{S}$-Rectangularity

We study the sample complexity of model-based distributionally robust reinforcement learning (DR-RL) with ${\chi }^2$-divergence defined uncertainty set under balanced nominal generative-model sampling. For $\mathcal{S}\mathcal{A}$-rectangular uncertainty set, existing upper and lower bounds do not match in the uncertainty level $\rho$. We close this gap by establishing a tighter upper bound of $\tilde{\mathcal{O}}\left(\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma {)}^3{\epsilon }^2}\left(1+\frac{\rho }{1-\gamma }\right)\right)$, which matches with the minimax lower bound and thus we establish the minimax optimality. Here, $\gamma$ is the discount factor, $\mathcal{S}$ and $\mathcal{A}$ are the state and action spaces, and $\epsilon$ is the optimality gap. Our proof develops a novel variance-sensitive framework that exploits the second-moment structure of adversarial density ratios to control robust Bellman estimation and to propagate statistical errors while preserving dependence linear in the divergence. For $\mathcal{S}$-rectangular uncertainty set with shared state-wise budget of $|\mathcal{A}|\rho$, we further establish an upper bound of $\tilde{\mathcal{O}}\left(\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma {)}^3{\epsilon }^2}\left(1+\frac{|\mathcal{A}|\rho }{1-\gamma }\right)\right)$ and also a matching minimax lower bound, and therefore we establish the minimax optimality also for $\mathcal{S}$-rectangular uncertainty set. Our results show that the sample complexity shall not depend on the minimum of non-zero transition probability of the nominal transition kernel, and provide the correct scaling with the uncertainty level $|\mathcal{A}|\rho$.