UMARM: Unified Multi-Agent Reward Matching

Behavior cloning, contrastive representation learning, and preference optimization are independently-motivated approaches behavioral modeling. We show that under a player-conditioned reward $r(s,a,z,c)$, these objectives share common structure: the behavior cloning optimum recovers the InfoNCE-optimal critic up to action-independent terms (Theorem 1), a principled adaptive schedule with super-linear decay guarantees convergence to the behavioral and preference objectives alone (Theorem 2), and the combined loss is equivalent to a MaxEnt goal-conditioned RL problem whose optimal reward recovers the player-conditioned data distribution (Theorem 3). UMARM (Unified Multi-Agent Reward Matching) instantiates this framework for multi-agent behavioral modeling, jointly learning a continuous player latent $z_p$ and a conditioned policy $\pi (a\mid s,z_p,c)$ from raw trajectories alone — no skill ratings, agent identifiers, or per-agent parameters are required. Applied to chess across $\sim 4.3$ million players spanning novice to grandmaster, UMARM achieves $56.9\%$ top-1 move prediction accuracy, outperforming the parameter-matched Maia-3 model. Constructing $z_p$ zero-shot from moves in the current game, accuracy reaches $56.1\%$ overall and $62.9\%$ after 40 moves of context. ELO ratings are recoverable from the latent geometry ($R^2=0.79$) despite never being supplied, and the preference head enables match outcome prediction between unrelated opponents. All capabilities emerge from the unified objective without task-specific supervision.