Pure, Mixed and Behavioral Strategy Learning in Mean Field Games

Mean Field Games (MFGs) provide a scalable framework for multi-agent reinforcement learning, but traditionally rely on two assumptions: agents are homogeneous (i.e., they use identical policies) and their policies are stochastic (i.e., their actions are random). We challenge the necessity for these two foundational assumptions by establishing equivalence between standard MFGs and a novel system with heterogeneous agents playing deterministic strategies. We prove that for any behavioral strategy Nash equilibrium (NE) in an MFG, there exists an equivalent mixed strategy NE over a population of agents with diverse, deterministic policies. This is analogous to classical game-theoretical insights from Kuhn's theorem, suitably adjusted to MFGs for the first time. The equivalence is constructive, enabling the approximation of Nash equilibria in large $N$-agent games using only pure strategies, which are simpler to implement and analyze. Our insight reframes the role of stochasticity in MFGs as a population-level sufficient condition rather than a necessary requirement for individual agents. Algorithmically, this perspective gives rise to a new class of learning methods, including Fictitious Play and finite-agent variants that operate with pure policies. We explore the performance, scalability, and behavioral implications of this framework on multi-agent reinforcement learning tasks, leveraging deep reinforcement learning for general tractability.