acceptodds
Under review as a conference paper at ICLR 2027

UMARM: Unified Multi-Agent Reward Matching

Abstract

Behavior cloning, contrastive representation learning, and preference optimization are independently-motivated approaches behavioral modeling. We show that under a player-conditioned reward , these objectives share common structure: the behavior cloning optimum recovers the InfoNCE-optimal critic up to action-independent terms (Theorem 1), a principled adaptive schedule with super-linear decay guarantees convergence to the behavioral and preference objectives alone (Theorem 2), and the combined loss is equivalent to a MaxEnt goal-conditioned RL problem whose optimal reward recovers the player-conditioned data distribution (Theorem 3). UMARM (Unified Multi-Agent Reward Matching) instantiates this framework for multi-agent behavioral modeling, jointly learning a continuous player latent and a conditioned policy from raw trajectories alone — no skill ratings, agent identifiers, or per-agent parameters are required. Applied to chess across million players spanning novice to grandmaster, UMARM achieves top-1 move prediction accuracy, outperforming the parameter-matched Maia-3 model. Constructing zero-shot from moves in the current game, accuracy reaches overall and after 40 moves of context. ELO ratings are recoverable from the latent geometry () despite never being supplied, and the preference head enables match outcome prediction between unrelated opponents. All capabilities emerge from the unified objective without task-specific supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
reward matching, rl, behavior cloning, contrastive learning, preference optimization, unification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RGnFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

UMARM learns a player representation from reference games and uses it to predict individual chess behavior. It combines behavior cloning, contrastive learning, and outcome-based preferences, with theoretical alignment claims and experiments on Lichess data.

Critical strengths
  • Learning individual behavior from limited reference games is practically relevant, and the results on unseen players support this application.
  • The same player representation is useful for move prediction, skill estimation, and outcome prediction.
Critical weaknesses
  • The theoretical justification has substantive gaps. Appendix A does not justify density-ratio preservation or cancellation of player-dependent offsets, and its parameter-convergence argument uses conditions absent from Theorem 2.
  • Methodological novelty is limited: player-history encoding [2] and few-shot personalization [1] are established. The main advance is their integration with joint training objectives and an adaptive contrastive schedule.
  • Maia4All [1] results use a different test set and reference-game budget. These results do not establish a direct comparison with Maia4All.
  • Individual style beyond skill is insufficiently validated. Random-player shuffling changes both, while the latents strongly predict ELO. This control does not isolate the contribution of personal style.

References:

[1] Learning to Imitate with Less: Efficient Individual Behavior Modeling in Chess
[2] Detecting Individual Decision-Making Style: Exploring Behavioral Stylometry in Chess

Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer RGnF

We thank the reviewer for the careful reading of the proofs and for finding the setting practically relevant. We would like to clarify that UMARM's advance is one model of any player's policy, with a theory for its loss and schedule, not an integration of [1] and [2]. The method, contributions, trained model and all submitted experimental results are unchanged.

W1: Appendix A

[...] Appendix A does not justify density-ratio preservation or cancellation of player-dependent offsets, and its parameter-convergence argument uses conditions absent from Theorem 2.

  • Density-ratio preservation. We now state it as a condition of Theorem 1's critic statement ("when the views identify their players", p. 5), and add a player-independent term b(τj)b(\tau_j) that cancels when players are compared.
  • Player-dependent offsets. We now name them and state where they cancel: they are the candidate player's summed soft values (V=log⁡ZV = \log Z), which the InfoNCE normalization does not absorb. Under the view condition, the softmax over actions removes them from log-likelihood differences of the player's moves. In reward differences they remain as the soft-value difference that Theorem 1 now states and Lprefer\mathcal L_{prefer} models.
  • Parameter convergence. We strengthen Theorem 2 with a standard stochastic-gradient argument, which replaces the gradient-flow argument and its parameter-convergence conclusion. The proof is shorter, and the statement is more realistic: it assumes no convergence of either loss and bounds the contrastive weight and the loss gap at every step, given only loss floors below the initial losses. Its SGD stationarity bound lists its standard assumptions: LNLL+Lprefer\mathcal L_{NLL}+\mathcal L_{prefer} is LL-smooth, ∥∇LCL∥≤M\Vert \nabla\mathcal L_{CL}\Vert \le M, gradients are unbiased with variance at most σ2\sigma^2, and the step size is at most 1/(4L)1/(4L) .

W2: Novelty

Methodological novelty is limited: player-history encoding [2] and few-shot personalization [1] are established. The main advance is their integration with joint training objectives and an adaptive contrastive schedule.

We would like to clarify that the advance is not an integration of [1] and [2]: as noted in Sections 1 and 2, [2] identifies players from their games, and [1] finetunes to each player. UMARM is one model of any player's policy:

  • No rating and no per-player optimization. The policy is conditioned on both players' latents in place of their ratings, computed from 5 reference games or the current game's moves.
  • Downstream capabilities beyond move prediction. The same latents recover ELO, never given to the model (R2R^2 = 0.79, Table 2), and predict outcomes between players (61.4% against 56.1% for an ELO baseline, Table 4).
  • Theory for the loss and the schedule. Behavior cloning at its optimum recovers the InfoNCE-optimal critic up to action-independent terms when the views identify their players (Theorem 1). The outcome preference between two players reduces to a reference-free Bradley-Terry likelihood on their soft values (Proposition 1), not a DPO-style loss: the unified loss differs from the naive sum with LDPO\mathcal L_{DPO} in nearly every component. The adaptive schedule leaves only these two losses as the contrastive gap closes (Theorem 2).

W3: Maia4All comparison

Maia4All [1] results use a different test set and reference-game budget. These results do not establish a direct comparison with Maia4All.

We would like to clarify that the paper reports Maia4All's published figures for context only, not as a direct comparison, as the submission stated; this is in part because Maia4All trains parameters on between 20 to 2500 games per player whereas UMARM and our comparisons do no test-time training. Our direct comparison is Maia-3 5M, on our test set, with UMARM's backbone, data source and window.

W4: Style beyond skill

Individual style beyond skill is insufficiently validated. Random-player shuffling changes both, while the latents strongly predict ELO; this control does not isolate the contribution of personal style.

We add a rating-matched control, Matched: the latent of a different player rated within 30 ELO points (Table 1). Top-1 accuracy (%):

BracketNormalMatchedShuffled
Skilled54.9554.2952.33
Advanced58.5458.0255.61
Master59.6058.3354.91
Avg56.9456.1753.74

Matching the rating removes about three quarters of the 3.20-point Shuffled drop, consistent with the linear ELO probe (Table 2). Under Matched, accuracy still drops by 0.77 points on average and by at least 0.52 in every bracket, supporting that the latent carries player information beyond the rating.

We hope the minor revisions and the rating-matched control address these concerns, and if so, we would appreciate it if the reviewer would consider updating the score.

Reviewer_NT7DOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies a chess player-conditioned move conditioned problem: given historical data, we seek a conditioned policy that can model the observed player, resulting in a policy that matches the strength of the observed behavior. In this setting, the authors derive a unification argument combining behavior cloning, preference based learned and contrastive learning.

Given this unification, the authors develop UMARM, a framework that learns to model player’s characteristics in a latent space and correctly mimic any player from its derived latent.

Critical strengths

The aim of simplifying training agents from demonstrations by unifying multiple common losses is intellectually interesting. Developing a method able to recover the ELO ratings of players based on learned latent representations is also a nice property of the method presented.

Critical weaknesses

There are two major related weaknesses of the paper, linked to the presentation of the results.

  1. Overall writing

The paper is mainly written in an informal way and doesn’t flow naturally, making it hard to judge the substance of the results. Examples of this writing issues include:

  • sentences of the form “X; Y” are overly used, breaking the narrative flow
  • terms like “InfoNCE”, “preference head”, “quality-calibrated gradients”, “NLL” and “p_data” are used but never defined in the main text. p(i) “alternates between white and black” with no formalization
  • the expectation for the NLL loss takes (tau, z, c) from D, but D doesn’t contain latent variables, the contrastive loss function depends on elements that are only vaguely defined “B is the set of contrastive views in a batch, zi is an ℓ2-normalized view — a latent for one player computed from a subset of that player’s reference games — P+(i) is the set of other views in B of view i’s player, κ is the contrastive temperature” (views and contrastive views are not defined, contrastive temperature doesn’t refer to any formula, etc.)

This does not constitute an exhaustive list of such writing issues.

  1. Unclear and limited contributions

Beside the phrasing, the contributions of this paper are unclear and fail to be correctly demonstrated. The main point of the paper is the unification of the different objectives, which is defined as the combination of the three losses defined on page 4 (L_NLL, L_CL, L_DPO) into a combination of only two losses. The motivation for such a reduction is not given. Moreover,

  • Reduction 1 (page 4) is not a reduction but rather a choice of loss. “Our preference term L_prefer coincides with the preference loss of Contrastive Preference Optimization (Xu et al., 2024), which arrives at the same form via an upper-bound argument on DPO; we adopt it directly as a likelihood. The CPO upper bound can be vacuous in chess, where blunders produce arbitrarily sharp distributions that a reference policy would never generate.”
  • Theorem 1 (page 5) cannot be understood without the definition of p_data, which is not given in the main text. Without its definition, Theorem 1 still seems to be the standard decomposition of a softmax policy with partition Z. I additionally find the insights given in the paper unclear, given that it stays informal and InfoNCE is not defined in the work: “This alignment applies to optima and not optimization dynamics: the NLL gradient for player separation is indirect, while InfoNCE provides direct gradient on the player latents.”
  • Theorem 2 (page 5) claims a “unification” where the L_CL loss term is redundant asymptotically under strong simultaneous convergence assumptions, while the paper itself admits “We retain L_CL as an optimization aid with an adaptive coefficient”, and therefore no unification is retained for the experiments. One motivation itself for the reduction is the naive optimization one could apply using a weighted sum of losses “Naively, one would optimize the weighted sum L_NLL + λ_CL L_CL + λ_DPO L_DPO, treating the three as independent regularizers. We now show this is unnecessary: these objectives share sufficient structure that two of the three terms can be simplified or absorbed.”, while in the rest of the paper the loss considered is even more simple by fixing λ_DPO = 1.
Questions for authors
  1. What would a unification buy over the naive sum of losses? This is the main point of the paper and it is not clearly motivated.
  2. How does such a unification appear in the training loss used for the experiments?
  3. How would your methodology apply to “multi-agent” systems, as mentioned in the title of the work? The current application to chess seems to treat the problem as a single agent problem, by modeling only one player in the game of chess.
4 comments
AuthorsOpenReview ↗

Response to Reviewer NT7D (part 1)

We thank the reviewer for the review. We would like to clarify that UMARM's unification is a shared reward structure of the three objectives, of which the two-term loss is a consequence, that its preference loss is neither DPO's loss nor a DPO-style loss, and that UMARM models both players of each game and more importantly every single chess player (or agent) using the exact same model with per-player latent inputs.

Q1 / W2a: What the unification is, and what it buys over the naive sum

What would a unification buy over the naive sum of losses? This is the main point of the paper and it is not clearly motivated.

[...] the unification of the different objectives, which is defined as the combination of the three losses defined on page 4 (L_NLL, L_CL, L_DPO) into a combination of only two losses.

The unification buys a loss that needs no reference policy (Reduction I), whose preference term is the outcome likelihood between two players and keeps the behavior-cloning optimum for fixed latents, and whose contrastive weight vanishes as the contrastive gap closes (Theorem 2). We would like to clarify that the unification is a shared reward structure, and the two-term loss is its consequence --- a naive sum of losses might introduce conflicting objectives and in fact we show via an added counterexample in Appendix A how a DPO-style loss can conflict with the behavior cloning objective where our preference loss component of the unified loss does not. Under one player-conditioned reward:

  • the behavior-cloning optimum contains the InfoNCE-optimal critic (Theorem 1);
  • the preference objective between two players reduces, under an imitation target, to Lprefer\mathcal L_{prefer} (Proposition 1, W2b);
  • the two-term loss is equivalent to a MaxEnt goal-conditioned RL problem (Theorem 3).

The unified loss is therefore not a DPO-style loss. It differs from the naive sum in nearly every component, as our proofs and a worked example show:

ComponentNaive sum with LDPO\mathcal L_{DPO}Unified loss with Lprefer\mathcal L_{prefer}
Preference lossBradley-Terry on the policy's log-ratios to πref\pi_{ref}Bradley-Terry on a separate preference head hξ(z)h_\xi(z)
Reference policyRequiredNone
Players' soft-value difference (V=log⁡ZV = \log Z)Dropped, as DPO is derived for two responses to one prompt, whose soft values cancelModeled by hξ(z)h_\xi(z)
Parameters the preference term trainsThe policy's own, entangling preference and behavioral gradientshξh_\xi and, via zpz_p, the player encoder and reference body, not the backbone or policy head
Behavior-cloning optimumCan be moved by LDPO\mathcal L_{DPO}, as by the CPO loss (two-move example, pp. 15-16)Kept for fixed latents
Contrastive weightFixedVanishes as the contrastive gap (how far LCL\mathcal L_{CL} is above its floor) closes (Theorem 2)

Q2 / W2d: The unification in the training loss, and Theorem 2

How does such a unification appear in the training loss used for the experiments?

Theorem 2 (page 5) claims a "unification" where the L_CL loss term is redundant asymptotically under strong simultaneous convergence assumptions [...] and therefore no unification is retained for the experiments.

Directly. The model trains on L(t)=LNLL+Lprefer+λCL(t)LCL\mathcal L^{(t)} = \mathcal L_{NLL}+\mathcal L_{prefer}+\lambda_{CL}^{(t)}\mathcal L_{CL} (Algorithm 1), the training procedure "as implemented":

  • Lprefer\mathcal L_{prefer} on the preference head replaces LDPO\mathcal L_{DPO} and its reference policy;
  • LCL\mathcal L_{CL} enters only with Theorem 2's decaying weight.

We would like to clarify that this decaying aid keeps the unification and that, as we state in the paper, the contrastive optimum meeting the NLL optimum "applies to optima and not optimization dynamics", which motivates our schedule. Separately, we strengthen Theorem 2 by replacing the gradient-flow argument with a standard stochastic-gradient argument. The proof is shorter, and the statement is more realistic: it assumes no convergence of either loss and relies only on standard assumptions. At every step, λCL≤λ0ϵˉCL\lambda_{CL} \le \lambda_0\bar\epsilon_{CL}, where ϵˉCL\bar\epsilon_{CL} is the normalized contrastive gap, and the training loss is within λCLBCL\lambda_{CL}B_{CL} of LNLL+Lprefer\mathcal L_{NLL}+\mathcal L_{prefer}, where BCLB_{CL} bounds LCL\mathcal L_{CL}. It converges to LNLL+Lprefer\mathcal L_{NLL}+\mathcal L_{prefer} as the contrastive gap closes, and in training LCL\mathcal L_{CL} decays toward its floor.

AuthorsOpenReview ↗

Response to Reviewer NT7D (part 2)

(continued from the previous comment)

Q3: Multi-agent

How would your methodology apply to "multi-agent" systems, as mentioned in the title of the work? The current application to chess seems to treat the problem as a single agent problem, by modeling only one player in the game of chess.

"Multi-agent" refers to learning the policies of a population of agents with one model. The theory is not specific to chess: Theorems 1-3 hold for any player-conditioned softmax policy, while Proposition 1 is stated for two players, and other domains are future work. We would like to clarify that UMARM models both players in chess and that the same model models all players we test on. Each game is generated by both players' latent-conditioned policies, the policy is conditioned on both players' latents in place of Maia-3's two ratings (p. 6), and Lprefer\mathcal L_{prefer} models the outcome between them.

W2b: Reduction I

Reduction 1 (page 4) is not a reduction but rather a choice of loss.

We initially motivated our reduction as a reference-free choice; in our revision, we formally derive it in Appendix A through a simple reduction. Between two players, the Bradley-Terry logit of the outcome equals DPO's logit plus β\beta times the players' soft-value difference, which LDPO\mathcal L_{DPO} drops. Assume the reference policy is the behavior-cloned player policy, the soft value is a player-level hξ(z)h_\xi(z), and returns are per-ply averages. Then the outcome's negative log-likelihood is exactly Lprefer\mathcal L_{prefer}, a loss distinct from DPO's (Q1). We delete both CPO sentences the reviewer quoted as they no longer motivate the derivation of Lprefer\mathcal L_{prefer}, and the reduction itself is unchanged in its result from the original submission.

W1 / W2c / W2e: Writing and notation

[...] terms like "InfoNCE", "preference head", "quality-calibrated gradients", "NLL" and "p_data" are used but never defined in the main text. [...] Theorem 1 still seems to be the standard decomposition of a softmax policy with partition Z. [...] the loss considered is even more simple by fixing λDPO=1\lambda_{DPO} = 1.

We address every listed item. We would like to clarify that the preference head, the views and κ\kappa were already defined in the submitted main text (pp. 4, 6), that Theorem 1's result is not the decomposition, and that the trained loss has no DPO term. Item by item:

ItemAnswer
"X; Y" sentencesWe split or remove every one in Secs. 3-5.
NLL, InfoNCENLL is the ubiquitous abbreviation for negative log-likelihood, now expanded at first use. InfoNCE is a common contrastive loss we cited throughout our paper and is our contrastive loss LCL\mathcal L_{CL}, whose formula is on p. 4 where we now name it.
pdatap_{data}The data's move distribution given the player latent, now defined in Theorem 1 (p. 5).
Theorem 1The decomposition is standard. The result is what its two terms are at the NLL optimum: the InfoNCE-optimal critic's difference between the players, when "the views identify their players", and their soft-value difference, which Lprefer\mathcal L_{prefer} models (pp. 5, 16).
Remark on dynamicsMotivation, not a result. The optima are aligned, but LNLL\mathcal L_{NLL} moves the latents only through the move probabilities, while LCL\mathcal L_{CL} acts on them directly (p. 17).
p(i)p(i)The player who makes move ai−1a_{i-1}: White for odd ii, Black for even ii (p. 3).
Quality-calibrated gradientsWe replace it with what LDPO\mathcal L_{DPO}'s gradient does: it raises the winner's ΔR\Delta R relative to the loser's (p. 4).
zz in (τ,z,c)∼D(\tau, z, c) \sim \mathcal DShorthand: zz is computed as zp=gϕ({τp,k})z_p = g_\phi(\lbrace \tau_{p,k} \rbrace) from the player's reference games (p. 4), as p. 3 now says.
λDPO=1\lambda_{DPO} = 1The preference term is Lprefer\mathcal L_{prefer} on hξh_\xi (Q1). For fixed latents, its weight cannot move the behavior-cloning optimum (Proposition 1(c), p. 15).

We hope these answers and the revised paper address the reviewer's concerns; if so, we would appreciate it if the reviewer would consider updating the score.

I thank the authors for their answer and the revision. The concerns about the central claim remain.

On the reduction DPO is designed for a different problem. It substitutes the closed-form solution (softmax policies) of the soft-optimal policies in the standard RLHF reward modeling loss in the Bradley-Terry model, using a reparametrization of the rewards in terms of policies only, comparable for a fixed prompt (or context/latent). Claiming that the new Lprefer\mathcal{L}_{prefer} loss does not use such a modeling through policies and πref\pi_{ref} holds by construction, this is not a reduction. Lprefer\mathcal{L}_{prefer} is a standard maximum likelihood loss for the BT model, where hξh_{\xi} models the score of the players.

Furthermore, the proof of the reduction given in Proposition 1 relies on the assumption πr=πref\pi_r = \pi_{ref} and Vr=hξ(z)V_r = h_\xi(z). By Proposition 1(b), this implies r(s,a,z,c)=hξ(z)r(s, a, z, c) = h_\xi(z) on the support of πref\pi_{ref}, i.e. the rewards are constant per player (they don't depend on the actions (Proposition 1(b)) nor the states (Assumption Vr=hξ(z)V_r = h_\xi(z))). Similarly, the display defining Lprefer\mathcal{L}_{prefer} is wrong in general: it equates G(τ+,z+,c)−G(τ−,z−,c)G(\tau^+, z^+, c) - G(\tau^-, z^-, c) with hξ(z+)−hξ(z−)h_\xi(z^+) - h_\xi(z^-), where the first terms depends on the sampled trajectories but not the second one. The equality requires a per-player constant reward, and rθr_\theta is such a reward only if πθ∝exp⁡(rθ)\pi_\theta \propto \exp(r_\theta) is uniform over legal moves.

Theorem 2 As stated in the revision, Theorem 2 is trivial. It first defines λCL(t)≜λ0,ε‾CL(t)min⁡(1,ε‾CL(t)/ε‾NLL(t))\lambda^{(t)}_{\mathit{CL}} \triangleq \lambda_0 \\, \overline{\varepsilon}^{(t)}_{\mathit{CL}} \min \left( 1, \overline{\varepsilon}^{(t)}_{\mathit{CL}} / \overline{\varepsilon}^{(t)}_{\mathit{NLL}} \right) and L(t)(ω)≜LNLL+Lprefer+λCL(t)LCL\mathcal{L}^{(t)}(\omega) \triangleq \mathcal{L}_{\mathit{NLL}} + \mathcal{L}_{\mathit{prefer}} + \lambda^{(t)}_{\mathit{CL}} \mathcal{L}_{\mathit{CL}}. Then, assuming some BCL≥LCLB_{\mathit{CL}} \geq \mathcal{L}_{\mathit{CL}}, it claims λCL(t)≤λ0,ε‾CL(t)\lambda^{(t)}_{\mathit{CL}} \leq \lambda_0 \\, \overline{\varepsilon}^{(t)}_{\mathit{CL}} and 0≤L(t)−(LNLL+Lprefer)≤λCL(t)BCL0 \leq \mathcal{L}^{(t)} - \left( \mathcal{L}_{\mathit{NLL}} + \mathcal{L}_{\mathit{prefer}} \right) \leq \lambda^{(t)}_{\mathit{CL}} B_{\mathit{CL}}. The final claim is that L(t)→LNLL+Lprefer\mathcal{L}^{(t)} \to \mathcal{L}_{\mathit{NLL}} + \mathcal{L}_{\mathit{prefer}} when ε‾CL(t)→0\overline{\varepsilon}^{(t)}_{\mathit{CL}} \to 0. This holds directly from the two other claims.

Multi-agent Following your remark, I now understand that "multi-agent" is meant to describe single-agent conditional policies, where the opponent acts as an input, with no strategic interaction. This is closer to population imitation learning than multi-agent learning in the usual sense.

I am maintaining my initial score.

AuthorsOpenReview ↗

Response to Reviewer NT7D

We thank the reviewer for the follow-up. We first respond to the points raised in the follow-up, and the table then shows where each concern from the initial review is addressed.

1. The reward in Proposition 1.

Reviewer: The equality requires a per-player constant reward, and rθr_\theta is such a reward only if πθ∝exp⁡(rθ)\pi_\theta \propto \exp(r_\theta) is uniform over legal moves.

This conflates two rewards. Reduction I defines rr relative to the reference, πr=πrefexp⁡(r)/Zr\pi_r=\pi_{ref}\exp(r)/Z_r, so r=rθ−log⁡πrefr=r_\theta-\log\pi_{ref}. A per-player constant rr gives πθ=πref\pi_\theta=\pi_{ref}, the behavior-cloned player policy, not a uniform policy. For any πθ\pi_\theta, rθ=log⁡πθ+hξ(z)r_\theta=\log\pi_\theta+h_\xi(z) has softmax πθ\pi_\theta and gives r=hξ(z)r=h_\xi(z) when πref=πθ\pi_{ref}=\pi_\theta. The reviewer's statement that "the rewards are constant per player" therefore applies only to rr.

The display the reviewer calls "wrong in general" is stated for the imitation target, a player-level soft value and per-ply average returns. The reviewer's own derivation gives r=hξ(z)r=h_\xi(z) under these conditions, so the equality holds. The revision now writes the return of rr as GrG_r.

On "this is not a reduction": Reduction I states the assumptions under which the Bradley-Terry likelihood on returns becomes LpreferL_{prefer}, which the paper calls an ELO-style loss.

2. The full statement of Theorem 2.

Reviewer: As stated in the revision, Theorem 2 is trivial. [...] The final claim is that L(t)→LNLL+LpreferL^{(t)} \to L_{NLL} + L_{prefer} [...]

That is not the theorem's final claim. Section 4 says "Appendix A states Proposition 1 and Theorem 2 in full", and the full statement adds an SGD stationarity bound on LNLL+LpreferL_{NLL}+L_{prefer} when training on the adaptive loss.

3. The bound BCLB_{CL}.

Reviewer: [...] assuming some BCL≥LCLB_{CL} \ge L_{CL} [...]

Theorem 2 needs no such assumption. Appendix A proves 0≤LCL≤2/κmin+log⁡(∣B∣−1)0\le L_{CL}\le 2/\kappa_{min}+\log(|B|-1), so the constant bound in Section 4 always exists.

4. Multi-agent imitation learning.

Reviewer: Following your remark, I now understand that "multi-agent" is meant to describe single-agent conditional policies [...]

Our remark did not say this. It said "learning the policies of a population of agents with one model" and "UMARM models both players in chess". Multi-agent imitation learning learns from "demonstrations of a set of experts interacting with each other within the same environment" [1]. Song et al. propose a framework for multi-agent imitation learning in general Markov games, which include two-player zero-sum games such as chess, and evaluate behavior cloning as a baseline. Each chess game is such a demonstration. This terminology does not affect any theorem or experimental result.

Initial review.

Initial reviewWhere addressed
Writing and undefined termsResponse part 2. The submission defined the preference head, which "maps the player latent zz to a scalar quality score", and a view, "a latent for one player computed from a subset of that player's reference games". The revision defines the remaining terms and splits the "X; Y" sentences in Sections 3-5.
What the unification buys (Q1)Response part 1: the loss "needs no reference policy", keeps "the behavior-cloning optimum for fixed latents", and has a contrastive weight that "vanishes as the contrastive gap closes".
Unification in the training loss (Q2)Response part 1. The experiments train L(t)=LNLL+Lprefer+λCL(t)LCLL^{(t)}=L_{NLL}+L_{prefer}+\lambda_{CL}^{(t)}L_{CL} (Algorithm 1).
Multi-agent setting (Q3)Response part 2 and point 4 above.
Reduction IResponse part 2, Proposition 1 and point 1 above.
Theorem 1Response part 2 and the revised Theorem 1.
Theorem 2Response part 1, Appendix A and points 2-3 above.

Contribution. We restate our contribution from our initial rebuttal for context: UMARM's contribution is "one rating-free model of any player's policy, built from previous games' or the current game's moves". With no rating input, it reaches 56.94% average top-1 accuracy, against 55.64% and 56.72% for the 5M and 23M Maia-3 models. From the current game alone it reaches 56.1%, rising to 62.9% after 40 moves, and ELO is recoverable from its latents (R2=0.79R^2=0.79).

We hope these responses resolve the lingering questions raised in the review. The follow-up does not raise concerns about Theorem 1, Theorem 3, the two-move example, or the experimental results. We would therefore appreciate it if the reviewer would reconsider the score, or identify the specific result about which they have questions.

References:

[1] Jiaming Song, Hongyu Ren, Dorsa Sadigh, Stefano Ermon. Multi-Agent Generative Adversarial Imitation Learning. Advances in Neural Information Processing Systems 31 (NeurIPS 2018).

Reviewer_6jT7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary:This paper introduces Unified Multi-Agent Reward Matching (UMARM), an offline self-supervised framework for modeling heterogeneous human behavior across large agent populations. The authors theoretically connect behavior cloning (via negative log-likelihood, NLL), contrastive representation learning (InfoNCE), and preference optimization (Bradley-Terry). They prove that under a player-conditioned reward rθ(s,a,z,c)r_\theta(s,a,z,c), the NLL optimum aligns with the InfoNCE-optimal player critic up to action-independent terms (Theorem 1), derive an adaptive contrastive schedule with superlinear decay ensuring asymptotic convergence to behavioral and preference objectives (Theorem 2), and demonstrate equivalence to MaxEnt goal-conditioned RL (Theorem 3). In practice, UMARM embeds each player into a continuous latent vector zpz_p via a shared player encoder and conditions a 5M Maia-3 Transformer backbone without requiring scalar skill ratings (ELO), explicit agent IDs, or per-player fine-tuning.

Contributions:

  • Unified Theoretical Framework: Formally bridges behavior cloning, contrastive learning, and preference optimization under a shared player-conditioned reward structure. It provides theoretical proofs showing the alignment of NLL and InfoNCE optima (Theorem 1), derives an adaptive decaying auxiliary schedule with convergence guarantees (Theorem 2), and establishes equivalence to goal-conditioned RL (Theorem 3).
  • Novel Decoupled Multi-Task Architecture: Implements a unified loss LNLL+Lprefer+λCL(t)LCLL_{\text{NLL}} +L_{\text{prefer}} + \lambda_{\text{CL}}^{(t)} L_{\text{CL}} where gradients from all three objectives couple exclusively at the player encoder latent zpz_p. A separate preference head performs quality calibration without directly biasing move-prediction logits.
  • Zero-Shot & Metadata-Free Behavioral Modeling: Eliminates the need for hand-crafted ELO ratings, per-player parameters (e.g., PEFT/LoRA), or meta-learning adaptation steps, supporting zero-shot policy inference for unseen players from current in-game moves.

Validation:

  • Move Prediction Accuracy: Evaluated on ~4.15M positions across 5,000 held-out Lichess players spanning ~4.3M total players from novice to grandmaster. UMARM achieves a 56.94% top-1 move-matching accuracy, outperforming the architecturally identical Maia-3 5M baseline (55.64%) and slightly exceeding the parameter-matched Maia-3 23M model (56.72%), while shuffling the player latent drops accuracy to 53.74%.
  • Skill Recovery via Latent Geometry: Linear ridge regression probes fit from zpz_p to unobserved ELO achieve R2=0.79R^2 = 0.79 (MAE 172), outperforming a non-linear MLP (R2=0.77R^2 = 0.77), confirming that player skill naturally organizes into a clean linear subspace without supervision.
  • Zero-Shot Move Prediction: When constructing zpz_p on the fly using solely within-game moves without historical reference games, top-1 accuracy climbs monotonically from 52.1% (2–5 moves) to 62.9% (41+ moves), surpassing the 5-game few-shot baseline (56.9%) beyond 20 moves.
  • Match Outcome Prediction: Using the decoupled preference head hξ(zp)h_\xi(z_p) to predict outcomes between arbitrary player pairs achieves 61.4% accuracy on validation games, significantly surpassing standard logistic ELO baseline fitting (56.1%).
Critical strengths
  • Unified Theoretical Framework and Decoupled Multi-Task Objective: The paper brilliantly bridges three previously disparate paradigms—behavior cloning, contrastive representation learning, and preference optimization—into a single coherent reward-matching framework. Proving that NLL and InfoNCE optima align (Theorem 1) and establishing equivalence to MaxEnt goal-conditioned RL (Theorem 3) provide deep conceptual insights. This is highly consequential because it replaces heuristic multi-loss combinations with a principled, theoretically grounded objective that simplifies multi-agent behavioral modeling.
  • Metadata-Free Zero-Shot Generalization and Linear Latent Geometry: UMARM successfully eliminates the need for hand-crafted ELO ratings, per-player parameters, or fine-tuning, while achieving state-of-the-art move prediction and enabling zero-shot inference from in-game context. Furthermore, the discovery that player skill emerges cleanly as a linear subspace in the latent space (R2=0.79R^2 = 0.79 via Ridge regression) is remarkable. These results prove that rich behavioral and skill attributes can be effectively learned purely through self-supervised trajectory matching, significantly advancing scalable opponent modeling.
Critical weaknesses
  • Strong Assumptions in Convergence Dynamics (Theorem 2): The theoretical guarantee for the adaptive contrastive decay schedule relies on the simultaneous convergence assumption of both suboptimality gaps and global L-smoothness near the optimum. In practice, behavioral and contrastive objectives can impose conflicting demands on shared encoder parameters, meaning this assumption could fail under different model capacities or domains. This is consequential because it introduces a theoretical vulnerability regarding the robustness of the adaptive decay schedule during early-stage training dynamics.
  • Limited Domain Evaluation: The empirical validation is conducted exclusively on chess (Lichess dataset). While chess is an exceptional and rigorous testbed for multi-agent strategy and human-likeness modeling, the paper lacks empirical demonstration in other multi-agent interactive domains (e.g., multi-player games, economic or robotic trajectory datasets). This restricts the immediate verification of the framework's broad generalizability beyond board games.
Questions for authors
  • Robustness of the Adaptive Contrastive Schedule (Theorem 2): The theoretical guarantees for the adaptive decay schedule rely heavily on the assumption of simultaneous convergence of both suboptimality gaps (ϵNLL\epsilon_{\text{NLL}} and ϵCL\epsilon_{\text{CL}}). Given that behavioral objectives and contrastive objectives can sometimes impose competing or conflicting gradient signals on shared representation parameters, can the authors provide empirical evidence or sensitivity analyses showing how the schedule behaves if one objective plateaus or diverges early? Furthermore, how sensitive is the final model performance to the choice of the initial contrastive weight λ0\lambda_0 and the loss floors (LNLL∗,LCL∗L_{\text{NLL}}^*, L_{\text{CL}}^*)?
  • Impact of the Number of Reference Games (KK) on Zero-Shot vs. Few-Shot Generalization: The paper demonstrates impressive zero-shot move prediction as in-game context accumulates (Table 3), eventually surpassing the 5-game few-shot baseline after 40 moves. Could the authors clarify how varying the number of reference games (KK) during training or inference (e.g., K=1,3,10K=1, 3, 10) affects the geometry of the learned latent space zpz_p and the rate of zero-shot convergence? Additionally, how does the player encoder handle stylistic shifts when a player's reference games span different time controls or historical periods?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6jT7 (part 1)

We thank the reviewer for the careful review and for seeing the framework as a principled replacement for heuristic multi-loss combinations. The method, contributions and results are unchanged: we strengthen the proof of Theorem 2 and add inference-time ablations for KK = 2 to 4.

W1 / Q1a: Theorem 2's assumptions, schedule analyses under different objective scenarios

The theoretical guarantee for the adaptive contrastive decay schedule relies on the simultaneous convergence assumption of both suboptimality gaps and global LL-smoothness near the optimum. In practice, behavioral and contrastive objectives can impose conflicting demands on shared encoder parameters, meaning this assumption could fail under different model capacities or domains.

[...] can the authors provide empirical evidence or sensitivity analyses showing how the schedule behaves if one objective plateaus or diverges early?

We strengthen Theorem 2 by replacing the gradient-flow argument with a standard stochastic-gradient argument. The proof is now shorter and simpler, and the statement is more realistic: it needs no convergence of either loss and relies only on standard optimization assumptions. We include the full statement in Appendix A, page 17, noting that conceptually this proof is stronger and simpler where the conclusion is now per-step bounds and a stationarity bound under weaker assumptions. For any loss floors below the initial losses, it gives two bounds at every step, with no smoothness or capacity assumption:

  • the contrastive weight is at most λ0\lambda_0 times the normalized contrastive gap, λCL≤λ0ϵˉCL\lambda_{CL} \leq \lambda_0 \bar{\epsilon}_{CL}
  • the training loss is within λCLBCL\lambda_{CL} B_{CL} of LNLL+Lprefer\mathcal L_{NLL}+\mathcal L_{prefer}, where BCLB_{CL} bounds LCL\mathcal L_{CL}, and converges to it as the contrastive gap closes.

We included ablations in our original submission in Table 7 that retrained the model with zero (No LCL\mathcal L_{CL}) and λ0=1\lambda_0 = 1 throughout (Fixed λCL=1\lambda_{CL} = 1), which is the weight our schedule keeps if LCL\mathcal L_{CL} never improves. Normal minus Shuffled accuracy is 1.31 and 2.67 points against 3.20 with our schedule. We provide empirical and sensitivity analyses based on our ablations and theoretical results:

  • LNLL\mathcal L_{NLL} plateaus. This is the observed run: the NLL plateaus near 1.3, so the weight is at most quadratic in the contrastive gap, λCL≤(λ0/δ)ϵˉCL2\lambda_{CL} \le (\lambda_0/\delta)\bar\epsilon_{CL}^2 for ϵˉNLL≥δ>0\bar\epsilon_{NLL} \ge \delta > 0.
  • LCL\mathcal L_{CL} stalls, for example because the objectives conflict. The weight stays at most λ0\lambda_0 times the stalled gap, and the SGD bound keeps an extra term of at most (3M2+σ2)λ02(3M^2+\sigma^2)\lambda_0^2 times the average squared gap. In training, LCL\mathcal L_{CL} decays toward its floor.
  • A loss diverges. We assume the contrastive loss does not diverge; if LNLL\mathcal L_{NLL} diverges, its normalized gap is clamped at 1, so the weight is λ0ϵˉCL2≤λ0\lambda_0 \bar\epsilon_{CL}^2 \le \lambda_0.

Q1b: Sensitivity to λ0\lambda_0 and the loss floors

Furthermore, how sensitive is the final model performance to the choice of the initial contrastive weight λ0\lambda_0 and the loss floors (LNLL∗\mathcal L_{NLL}^{\ast}, LCL∗\mathcal L_{CL}^{\ast})?

We are unable to run training hyperparameter sweeps during the rebuttal period, since each training run takes about 17 days. However, we observed previously from runs we conducted before submission (not in the paper) little sensitivity to λ0\lambda_0 and the NLL floor, and more to the contrastive floor:

  • λ0\lambda_0 = 0.5, 1 and 2 gave final Normal accuracies within about 0.15 points, and a final λCL\lambda_{CL} below about 0.05. Only λ0\lambda_0 near 0.1 moved results, toward Table 7's No-LCL\mathcal L_{CL} row.
  • NLL floors from 0.8 to 1.2 made no significant difference.
  • A contrastive floor a little below our 1.43 moved results toward Table 7's fixed-weight row.

Theorem 2 holds for any λ0>0\lambda_0 > 0, and while an error in the NLL floor rescales λCL\lambda_{CL} it does not change that it vanishes. The contrastive floor instead decides whether the contrastive gap LCL−LCL∗\mathcal L_{CL} - \mathcal L_{CL}^{\ast}, and with it the weight, can reach zero.

AuthorsOpenReview ↗

Response to Reviewer 6jT7 (part 2)

(continued from the previous comment)

Q2a: Number of reference games KK

Could the authors clarify how varying the number of reference games (KK) during training or inference (e.g., KK = 1, 3, 10) affects the geometry of the learned latent space zpz_p and the rate of zero-shot convergence?

At inference time, more reference games raise accuracy mostly with the second game, and only for the player's own latent. We add KK = 2 to 4 to Table 7:

KK12345
Normal56.3956.7156.7956.7756.94
Shuffled53.7253.7453.7553.7053.74
Normal minus Shuffled2.672.973.043.073.20

Normal accuracy rises by 0.32 points with the second game, out of 0.55 points from KK = 1 to 5, while Shuffled accuracy stays within 0.05 points.

  • Zero-shot convergence. The zero-shot latent uses only the current game's moves, so KK does not change the zero-shot curve. It changes only the few-shot level the curve is compared with: by move bucket, zero-shot accuracy passes the KK = 1 average at 11-20 moves (56.6%) and the KK = 2 to 5 averages at 21-40 moves (59.1%), not only after 40 moves.
  • Not measured. The latent geometry as a function of KK, training with K≠5K \ne 5, and K>5K > 5; doing so would require retraining.

Q2b: Reference games from different time controls or periods

Additionally, how does the player encoder handle stylistic shifts when a player's reference games span different time controls or historical periods?

The encoder does not model them. It sees only the boards before and after each move and the move's ply index, not the time control or date, and pools the games into one latent that does not depend on their order. LCL\mathcal L_{CL} trains the latents of disjoint subsets of a player's games to agree. Time control and think time enter only the policy, per position. Our data are rated blitz games only, from January 2023 to July 2025 for training and August 2025 for evaluation, and we do not evaluate such shifts. Section 7 suggests that the zero-shot latent, built from the current game alone, can follow a player whose current play differs from their reference games.

W2: Evaluation only on chess

The empirical validation is conducted exclusively on chess (Lichess dataset). [...] the paper lacks empirical demonstration in other multi-agent interactive domains (e.g., multi-player games, economic or robotic trajectory datasets).

We agree: chess is the only domain we evaluate and do not add any new domains during our revision in part because doing so would require months of experiments. Conceptually, our theoretical results and framework generally extend to other multi-agent domains as we do not make any chess-exclusive assumptions in our work: Theorems 1-3 hold for any player-conditioned reward that induces a softmax policy. Lprefer\mathcal L_{prefer} and Proposition 1, however, are stated for two-player games, so they do not cover multi-player games as stated, though we expect such a change to involve a multiplayer objective such as a Plackett-Luce-type extension of our Bradley-Terry-inspired preference loss. We have updated Section 7 to list Go, poker and StarCraft II as future work.

We thank the reviewer again for the constructive questions, which led to the strengthened Theorem 2 and the results over KK. We are happy to answer any follow-up questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewers and Summary of Revisions

We thank the reviewers for their reviews and for finding that the framework bridges three previously disparate paradigms (6jT7), its aim is intellectually interesting and that recovering ELO from the learned latents is a nice property (NT7D), and learning individual behavior from limited reference games is practically relevant (RGnF).

UMARM's contribution is one rating-free model of any player's policy, built from previous games' or the current game's moves, whose losses follow from one player-conditioned reward (Theorems 1-3, Proposition 1). The method, contributions, trained model and all submitted experimental results are unchanged; our revisions made three changes to help answer the reviewers' questions:

  • we strengthened Theorem 2 with a simpler proof and in the process made its assumptions more realistic,
  • we proved a simple derivation for Reduction I to help motivate it, and
  • we ran two inference-time ablations on the number of reference games KK and on rating-matched latent shuffling.

Theorem 2 (6jT7, NT7D, RGnF). We strengthen Theorem 2 by replacing the gradient-flow argument with a standard stochastic-gradient argument. The proof is shorter, and the statement is more realistic: it assumes no convergence of either loss, its floors need not be minima, and its bound now relies only on standard assumptions (LNLL+Lprefer\mathcal L_{NLL}+\mathcal L_{prefer} is smooth, LCL\mathcal L_{CL} has bounded gradients).

Reduction I and the preference loss (NT7D, RGnF). We now derive Lprefer\mathcal L_{prefer}. Between two players, under an imitation target (the behavior-cloned player policy, which is also the reference), the outcome likelihood is a reference-free Bradley-Terry likelihood on the players' soft values (V=log⁡ZV = \log Z). With a player-level soft value hξ(z)h_\xi(z) and per-ply average returns, its negative log-likelihood is Lprefer\mathcal L_{prefer}. Lprefer\mathcal L_{prefer} is therefore an ELO-style loss, not DPO or a DPO-style loss such as CPO, and the unified loss differs from the naive sum with LDPO\mathcal L_{DPO} in nearly every component.

Theorem 1 (RGnF, NT7D). We state density-ratio preservation as a condition of the theorem (the views identify their players) and name the player-dependent offsets (the players' soft values modeled by Lprefer\mathcal L_{prefer}).

Player information beyond the rating (RGnF). We add a rating-matched control (Table 1). Swapping in the latent of a different player rated within 30 ELO points still lowers accuracy by 0.77 points on average (about a quarter of the 3.20-point drop for a random player) and by at least 0.52 points in every bracket, so the latent carries player information beyond the rating.

Number of reference games (6jT7). We add KK = 2 to 4 at inference to Table 7. Normal accuracy rises by 0.32 points with the second game, out of 0.55 points from KK = 1 to 5, while Shuffled accuracy stays within 0.05 points.

Writing (NT7D). We define NLL, p(i)p(i), InfoNCE, pdatap_{data}, and the latents in D\mathcal D in the main text, replace the phrase "quality-calibrated gradients", and split the "X; Y" sentences in Secs. 3-5. Views, κ\kappa and the preference head were already defined.

We hope these changes address the main concerns, and we welcome any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.