Summary:This paper introduces Unified Multi-Agent Reward Matching (UMARM), an offline self-supervised framework for modeling heterogeneous human behavior across large agent populations. The authors theoretically connect behavior cloning (via negative log-likelihood, NLL), contrastive representation learning (InfoNCE), and preference optimization (Bradley-Terry). They prove that under a player-conditioned reward rθ(s,a,z,c), the NLL optimum aligns with the InfoNCE-optimal player critic up to action-independent terms (Theorem 1), derive an adaptive contrastive schedule with superlinear decay ensuring asymptotic convergence to behavioral and preference objectives (Theorem 2), and demonstrate equivalence to MaxEnt goal-conditioned RL (Theorem 3). In practice, UMARM embeds each player into a continuous latent vector zp via a shared player encoder and conditions a 5M Maia-3 Transformer backbone without requiring scalar skill ratings (ELO), explicit agent IDs, or per-player fine-tuning.
Contributions:
- Unified Theoretical Framework: Formally bridges behavior cloning, contrastive learning, and preference optimization under a shared player-conditioned reward structure. It provides theoretical proofs showing the alignment of NLL and InfoNCE optima (Theorem 1), derives an adaptive decaying auxiliary schedule with convergence guarantees (Theorem 2), and establishes equivalence to goal-conditioned RL (Theorem 3).
- Novel Decoupled Multi-Task Architecture: Implements a unified loss LNLL+Lprefer+λCL(t)LCL where gradients from all three objectives couple exclusively at the player encoder latent zp. A separate preference head performs quality calibration without directly biasing move-prediction logits.
- Zero-Shot & Metadata-Free Behavioral Modeling: Eliminates the need for hand-crafted ELO ratings, per-player parameters (e.g., PEFT/LoRA), or meta-learning adaptation steps, supporting zero-shot policy inference for unseen players from current in-game moves.
Validation:
- Move Prediction Accuracy: Evaluated on ~4.15M positions across 5,000 held-out Lichess players spanning ~4.3M total players from novice to grandmaster. UMARM achieves a 56.94% top-1 move-matching accuracy, outperforming the architecturally identical Maia-3 5M baseline (55.64%) and slightly exceeding the parameter-matched Maia-3 23M model (56.72%), while shuffling the player latent drops accuracy to 53.74%.
- Skill Recovery via Latent Geometry: Linear ridge regression probes fit from zp to unobserved ELO achieve R2=0.79 (MAE 172), outperforming a non-linear MLP (R2=0.77), confirming that player skill naturally organizes into a clean linear subspace without supervision.
- Zero-Shot Move Prediction: When constructing zp on the fly using solely within-game moves without historical reference games, top-1 accuracy climbs monotonically from 52.1% (2–5 moves) to 62.9% (41+ moves), surpassing the 5-game few-shot baseline (56.9%) beyond 20 moves.
- Match Outcome Prediction: Using the decoupled preference head hξ(zp) to predict outcomes between arbitrary player pairs achieves 61.4% accuracy on validation games, significantly surpassing standard logistic ELO baseline fitting (56.1%).
Response to Reviewer RGnF
We thank the reviewer for the careful reading of the proofs and for finding the setting practically relevant. We would like to clarify that UMARM's advance is one model of any player's policy, with a theory for its loss and schedule, not an integration of [1] and [2]. The method, contributions, trained model and all submitted experimental results are unchanged.
W1: Appendix A
W2: Novelty
We would like to clarify that the advance is not an integration of [1] and [2]: as noted in Sections 1 and 2, [2] identifies players from their games, and [1] finetunes to each player. UMARM is one model of any player's policy:
W3: Maia4All comparison
We would like to clarify that the paper reports Maia4All's published figures for context only, not as a direct comparison, as the submission stated; this is in part because Maia4All trains parameters on between 20 to 2500 games per player whereas UMARM and our comparisons do no test-time training. Our direct comparison is Maia-3 5M, on our test set, with UMARM's backbone, data source and window.
W4: Style beyond skill
We add a rating-matched control, Matched: the latent of a different player rated within 30 ELO points (Table 1). Top-1 accuracy (%):
Matching the rating removes about three quarters of the 3.20-point Shuffled drop, consistent with the linear ELO probe (Table 2). Under Matched, accuracy still drops by 0.77 points on average and by at least 0.52 in every bracket, supporting that the latent carries player information beyond the rating.
We hope the minor revisions and the rating-matched control address these concerns, and if so, we would appreciate it if the reviewer would consider updating the score.