Reinforcement Learning for Quantal Stackelberg Equilibrium in Mean-Field Games

We study online reinforcement learning in episodic Stackelberg mean-field games, where a leader interacts with an infinite population of boundedly rational, myopic followers governed by a quantal-response model. The leader knows neither the followers' reward function nor their realized rewards and must infer their response model from behavioral observations. We analyze the complete observation (CO) setting, where the leader observes the exact joint state-action distribution of the population, and then extend the analysis to finite-sample observation (FSO), where only $N$ followers are observed per step and the true population distributions remain unobserved. We propose an algorithm combining kernelized optimistic least-squares value iteration with a confidence set based on maximum likelihood estimation for the followers' reward parameter. Under our stated assumptions, for a leader kernel of rank $d_L$ and a follower feature dimension $d_F$, the algorithm achieves high-probability regret $\tilde{O}(d_L^{3/2}{H}^2\sqrt{T})$ under FSO relative to the optimal CO policy, where $T$ is the number of episodes and $H$ is the horizon length. With fixed response and regularization parameters, this rate holds when $d_F\lesssim \sqrt{d_L}$, $T\gtrsim d_F^2$, and $N=\tilde{\Omega }(|\mathcal{Z}|T)$, where $\mathcal{Z}$ is the followers' state-action space. We further establish a CO leader-side lower bound $\Omega (d_{L}H\sqrt{T})$, showing that the $\sqrt{T}$ dependence is unavoidable. For the FSO commitment-rule class in this paper, we prove a fixed-dimensional lower bound $\Omega (T/\sqrt{N})$; hence $N=\tilde{\Omega }(T)$ is necessary for uniform expected $\tilde{O}(\sqrt{T})$ regret within this class.