Offline Equilibrium Computation in Convex Markov Games

Convex Markov games extend Markov games to utilities beyond expected returns, capturing essential notions such as fairness, safety, and imitation. This expressiveness stems from evaluating the global game status through concave functions of state–action occupancies. However, this non-linearity makes policy updates dependent on the current occupancy, formalized as a shadow reward. Consequently, computing equilibria of convex Markov games requires tracking the current occupancy at every update, through fresh environment interactions or an exact transition model, making the process computationally expensive. We ask whether such equilibria can instead be approximated purely from logged multi-agent play. We propose cMG-DICE, which combines the shadow reward formulation with stationary distribution correction estimation (DICE) so that each agent's best response reduces to a single minimization over logged transitions. We demonstrate the efficacy of this framework by approximating equilibria under diverse utilities and initial joint policies on tabular games and the Clean Up environment from a single dataset, bypassing the need to restart costly online optimization from scratch for each new equilibrium.