acceptodds
Under review as a conference paper at ICLR 2027

MA-UITD: Offline Multi-Agent Reward Inference for Characterizing Continuous Social Behavior

Abstract

Multi-agent inverse reinforcement learning offers a powerful framework not only for imitation learning but also for understanding the neural computations supporting social interactions and decision-making. Inferring rewards from interacting agents is challenging because observed choices reflect both personal preferences and expectations about others. Existing methods often require additional environment interaction, repeated policy optimization as an inner loop, or restrictive assumptions about environment dynamics and action spaces, limiting their applicability to fixed behavioral datasets. We introduce Multi-Agent Unilateral Inverse Temporal Difference Learning (MA-UITD), an offline method for inferring explicit agent-specific rewards from joint demonstrations in continuous-state environments with either discrete or bounded continuous actions. Its core idea is to represent the long-term consequences of each agent’s own actions through belief-conditioned successor features that average over other agents’ behavior. Shared reward features and agent-specific preference vectors connect these unilateral values to the target reward function, while inverse temporal-difference consistency and noise-contrastive behavioral learning enable inference without known dynamics or an inner policy-optimization loop. Across simulated cooperative, competitive, and general-sum tasks, MA-UITD outperforms baselines in discrete-action settings and infers behaviorally compatible reward profiles in continuous-action settings. When applied to experimentally collected social-foraging trajectories, MA-UITD inferred distinct spatial and interaction-dependent reward structures for leaders and followers, providing a quantitative characterization of latent behavioral objectives that may be relevant for understanding the neural computations underlying social behavior.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Multi-agent inverse reinforcement learning, inverse temporal-difference learning, social behavior, computational neuroscience

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uZ2JOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MA-UITD, an offline framework for inferring agent-specific arrival-state rewards from joint demonstrations. It combines shared reward features and individual preference vectors with belief-conditioned successor features that marginalize other agents’ behavior. Learning alternates between behavioral fitting and inverse temporal-difference consistency, with a contrastive objective for continuous actions. Experiments examine policy returns and reward recovery in cooperative, competitive, and general-sum environments. An application to mouse social-foraging trajectories produces different spatial and proximity-related reward components for leaders and followers. The objective is to recover behaviorally compatible rewards rather than uniquely identify underlying preferences.

Critical strengths
  1. The problem is well motivated by behavioral experiments, where fixed trajectories may be available without reward labels, known dynamics, or opportunities for additional interaction. A framework that directly estimates explicit individual rewards from continuous joint behavior addresses a meaningful practical need.
  2. The synthetic evaluation examines both task performance and held-out reward agreement across different interaction types, with substantial improvements over the reported discrete-action baselines. The mouse application illustrates a potentially useful way to characterize individual differences in social behavior. The explicit acknowledgment of reward non-identifiability also appropriately limits the interpretation of the results.
Critical weaknesses
  1. The methodological advance over the closest prior work is insufficiently established. ITD already learns shared reward features, individual preference vectors, and successor features through behavioral fitting and inverse TD consistency. MAMQL already uses opponent-marginalized values for individual action selection. MA-UITD combines these ideas and extends them to continuous actions, but the additional learning capability introduced by the belief-conditioned representation remains unclear. In particular, the implementation calibrates beliefs to demonstrated behavior rather than inferring them separately. The formulation is reasonable, but the current presentation does not convincingly establish a substantial conceptual advance beyond adapting existing mechanisms to this setting.
  2. The evaluation does not sufficiently isolate the benefit of the proposed formulation over its closest methodological foundation. Shared reward features, individual preference vectors, behavioral fitting, and inverse TD consistency are inherited from ITD. Since the implementation calibrates beliefs to demonstrated behavior rather than learning them separately, a matched ITD-style control using the joint state is particularly important. The manuscript provides neither this comparison nor component ablations, and the continuous-action tables report no competing method. Consequently, the results establish promising performance but leave the added value of the proposed multi-agent construction insufficiently demonstrated.

Updated comments: I have decided to lower my score because the author did not respond to any of my comments, especially regarding the difference between the method used in this article and similar prior benchmark methods.

Questions for authors
  1. Clarification and targeted validation: Under the assumption that predictive beliefs match demonstrated opponent behavior, what methodological changes distinguish MA-UITD from ITD adapted to joint-state inputs and agent-specific successor-feature heads? A comparison with matched inputs, capacity, and training budget would materially help assess the contribution.
  2. Targeted validation: Do the inferred leader–follower reward differences remain stable across training seeds and reasonable regularization choices? Does including the distance-dependent reward component improve prediction of held-out mouse actions over an otherwise matched model?
  3. Technical clarification: Appendix B.3 samples negative actions from the expert’s nonuniform marginal action distribution, but Eq. (10) contains no explicit proposal-density correction. How is this objective reconciled with the Boltzmann policy in Eq. (7) and the rollout procedure in Appendix B.4? Is a correction implemented, or is a different base measure intended?
4 comments
AuthorsOpenReview ↗

Response to Weaknesses and Question 1 part 1

We thank the reviewer for the careful reading and constructive and incisive feedback. We apologize for our delayed response; we were completing and validating the additional simulations needed to address the reviewer’s questions thoroughly.

Weaknesses and Question 1

1.The methodological advance over the closest prior work is insufficiently established. ITD already learns shared reward features, individual preference vectors, and successor features through behavioral fitting and inverse TD consistency. MAMQL already uses opponent-marginalized values for individual action selection. MA-UITD combines these ideas and extends them to continuous actions, but the additional learning capability introduced by the belief-conditioned representation remains unclear. In particular, the implementation calibrates beliefs to demonstrated behavior rather than inferring them separately. The formulation is reasonable, but the current presentation does not convincingly establish a substantial conceptual advance beyond adapting existing mechanisms to this setting.

2. The evaluation does not sufficiently isolate the benefit of the proposed formulation over its closest methodological foundation. Shared reward features, individual preference vectors, behavioral fitting, and inverse TD consistency are inherited from ITD. Since the implementation calibrates beliefs to demonstrated behavior rather than learning them separately, a matched ITD-style control using the joint state is particularly important. The manuscript provides neither this comparison nor component ablations, and the continuous-action tables report no competing method. Consequently, the results establish promising performance but leave the added value of the proposed multi-agent construction insufficiently demonstrated.

Q1. Clarification and targeted validation: Under the assumption that predictive beliefs match demonstrated opponent behavior, what methodological changes distinguish MA-UITD from ITD adapted to joint-state inputs and agent-specific successor-feature heads? A comparison with matched inputs, capacity, and training budget would materially help assess the contribution.

We acknowledge that, conceptually, an ITD control using joint state as input and agent-specific successor-feature heads would be similar to the implemented MA-UITD estimator. This similarity reflects the power and generality of the ITD framework, but ITD alone does not specify which value function is behaviorally relevant in a Markov game, how other agents’ behavior should enter that value, or under what conditions it can be estimated from fixed joint demonstrations. MA-UITD addresses these questions through belief-conditioned unilateral values.

This specific structure of MA-UITD, therefore, is not an ad hoc reuse of single-agent ITD; it follows from the logistic quantal response equilibrium (LQRE) formulation of a Markov game. Under LQRE, each agent responds logistically to the expected value of its own action after marginalizing over the simultaneous actions of the other agents. Below, we provide a justification here to show why this form emerges and why any other implementation wouldn't work.

AuthorsOpenReview ↗

Response to Weaknesses and Question 1 part 2

Recall the LQRE constrains in two agents: given precision λi>0\lambda_i>0, define the belief conditioned logit response as:

\triangleq \frac{ \exp\\!\left(\lambda_i\bar Q_i^{\pi}(s,a_i;r_i)\right) }{ \displaystyle \sum_{\tilde a_i\in\mathcal A_i} \exp\\!\left(\lambda_i\bar Q_i^{\pi}(s,\tilde a_i;r_i)\right) }.$$ A joint policy profile $\pi^\star$ is an LQRE under $R=(r_1,r_2)$ if both response equations hold simultaneously: $$\pi_1^\star=\text{LR}_1(\pi_2^\star;r_1), \qquad \pi_2^\star=\text{LR}_2(\pi_1^\star;r_2).$$ Our inverse problem starts from the demonstrated joint policy $\pi^D=(\pi_1^D,\pi_2^D)$ and **searches for rewards $\hat R=(\hat r_1,\hat r_2)$ that make this observed profile an LQRE:** $$\pi_1^D=\text{LR}_1(\pi_2^D;\hat r_1), \qquad \pi_2^D=\text{LR}_2(\pi_1^D;\hat r_2).$$ Consequently, agent 1’s unilateral value must evaluate its learned reward while agent 2 follows its demonstrated policy: $$\bar Q_1^{\pi^D}(s,a_1;\hat r_1) =\mathbb{E}_{\substack{ a_2\sim\pi_2^D(\cdot\mid s),\\ s'\sim P(\cdot\mid s,a_1,a_2),\\ a_1'\sim\pi_1^D(\cdot\mid s') }} \left[ \hat r_1(s') + \gamma\, \bar Q_1^{\pi^D}(s',a_1';\hat r_1) \right],$$ with an analogous recursion for agent 2. After applying the reward-feature factorization and estimating these expectations with demonstrated transitions, this recursion gives exactly the $\mathcal{L}_{\text{UITD}}$ in main text Equation 12. The behavioral objective then enforces that each demonstrated policy is the corresponding logit response. Thus, the two objectives directly impose the inverse LQRE conditions without repeatedly solving a forward equilibrium problem. This derivation explains why MA-UITD coincides computationally with single-ITD on the joint state using agent-specific successor-feature heads. Beyond forwardly deriving MA-UITD from the LQRE constraints, we argue that calibrating predictive beliefs to the demonstrated opponents’ behavior is essential for model-free reward inference in the fully offline setting. Fixed demonstrations do not separately identify beliefs and rewards: different combinations can explain the same observed behavior. Moreover, evaluating alternative beliefs generally require counterfactual transitions that are unavailable without an explicit dynamics model or additional environment interaction. Anchoring beliefs to the demonstrated opponent policies is therefore not arbitrary; Mechanistically, it is the assumption that makes reward inference identifiable and estimable from the available data. **Potential belief inference within the MA-UITD framework** To further illustrate the value of the proposed MA-UITD framework, we also describe a possible inverse-planning application in which MA-UITD can infer an agent’s internal belief when the reward functions and transition dynamics are known. For example, we can parameterize agent $i$’s belief as $\mu_{\theta_i,-i}$ and infer $\theta_i$ from its observed choices. A similar behavioral cloning loss could help the inference: For each candidate belief, the corresponding unilateral value is defined by $$\bar{Q}_i^{\theta_i}(s,a_i) =\mathbb{E}_{ a_{-i}\sim\mu_{\theta_i,-i}(\cdot\mid s),\, s'\sim P(\cdot\mid s,a_i,a_{-i}),\, a_i'\sim\pi_i^D(\cdot\mid s') } \left[ r_i(s') + \gamma\bar{Q}_i^{\theta_i}(s',a_i') \right].$$ The belief parameters can then be estimated by minimizing the negative log-likelihood of the demonstrated actions: $$\mathcal{L}_{\mathrm{belief}}(\theta_i) =- \mathbb{E}_{(s,a_i)\sim\mathcal{D}} \left[ \log \operatorname{LR}_i \left( \mu_{\theta_i,-i};r_i \right) (a_i\mid s) \right].$$ This loss searches for the predictive belief under which the known reward makes the demonstrated actions most likely. It does not require the demonstrated interaction to satisfy an equilibrium condition as individual agent’s subjective belief need not equal the opponents’ actual or demonstrated policies. Only given known rewards and transition dynamics, the loss above can therefore be used to infer an agent’s belief under any form of multi-agent interaction captured by the specified response model and belief class. We do not include this extension in the current manuscript because our focus is the fully offline setting with unknown rewards, unknown dynamics, and no additional environment interaction.
AuthorsOpenReview ↗

Response to Question 2 and Question 3

Question 2

Q2. Do the inferred leader–follower reward differences remain stable across training seeds and reasonable regularization choices? Does including the distance-dependent reward component improve prediction of held-out mouse actions over an otherwise matched model?

Yes, the inferred leader-follower reward differences remained stable across training seeds. Appendix C.4 Figure 9 reports inferred-reward correlations across seeds for each mouse pair.

Adding an explicit inter-animal-distance component produced a role-specific improvement in held-out action prediction. Relative to the capacity-matched additive position model, the distance-augmented model consistently reduced held-out NCE for followers across all five mouse pairs and all three training seeds (mean ΔNCE=−0.0167\Delta\mathrm{NCE}=-0.0167), whereas the improvement for leaders was small and inconsistent (mean ΔNCE=−0.0019\Delta\mathrm{NCE}=-0.0019). This result suggests that follower actions contain social-distance-dependent structure beyond additive spatial effects(Appendix C.4 Figure 10).

Question 3

Q3. Appendix B.3 samples negative actions from the expert’s nonuniform marginal action distribution, but Eq. (10) contains no explicit proposal-density correction. How is this objective reconciled with the Boltzmann policy in Eq. (7) and the rollout procedure in Appendix B.4? Is a correction implemented, or is a different base measure intended?

Thank you for identifying this ambiguity. The original manuscript described only the empirical-marginal negatives used for the animal data and did not state that uniform negatives were used in the simulation experiments. We have corrected this omission.

For animal data, no proposal-density correction is implemented because the negative-action distribution is an intentional model choice that defines the base measure. We use a uniform base measure for simulated agents, where no action tendency is privileged by default. For the animal data, we use each animal’s empirical marginal action distribution, assuming that it reasonably approximates the animal’s baseline action biases. However, we also note that this is a domain-motivated modeling assumption rather than a universally required choice; other base measures would represent different assumptions about baseline behavior.

Rollout candidates are drawn from the same base measure used during training, ensuring consistency between training and evaluation. A density correction would be required only if the empirical marginal were used merely as a sampling proposal while targeting a uniform-base policy. We also added a sensitivity analysis of the negative-action distribution in Appendix D, Figure 12.

AuthorsOpenReview ↗

A reminder

We apologize again for our delayed response while we completed and validated the additional experiments. We have now posted a detailed point-by-point response to all of your comments, including a substantive clarification of MA-UITD’s relationship to ITD, as well as the requested baseline comparisons and ablations. Since your updated score was explicitly based on the absence of a response, we would be grateful if you could review our response and revised manuscript and reconsider your assessment in light of this new information.

Reviewer_SrTaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multi-Agent Unilateral Inverse TemporalDifferenceLearning (MA-UITD), which is an offline multi-agent inverse reinforcement learning framework for inferring agents own reward functions. They use belief-conditioned successor features to represent the long-term consequences of each agent’s own actions. These features average over the possible behaviors of other agents. The method is evaluated in both simulated environments and on experimentally collected real behavioral data.

Critical strengths

The work applies multi-agent IRL to an interesting real-world dataset from experiments with mice. It develops a well-designed RL framework to address this problem.

Critical weaknesses

The methodological contribution seems like too engineering. The main algorithmic components like marginalizing over others’ actions to obtain unilateral values, inverse temporal-difference learning, and NCE to handle the normalizer for continuous actions have precedents in prior work. The paper also lacks theoretical justification for combining these components in the proposed way. I am particularly concerned about the simplification in Section 4.2. Although it seems reasonable for offline data, it introduces a gap between the theoretical formulation and the implemented method. It is unclear whether this simplification preserves the earlier theoretical framework or effectively departs from it.

Questions for authors
  1. Under the simplification in Section 4.2, what role does the previously introduced multi-agent framework still play?
  2. Do the reward features need to be manually specified? If so, how does this affect the method? Defining different feature dimensions for each environment based on human understanding may introduce subjective biases?
  3. Does full cooperation yield the highest payoff in Gems? Given that full cooperation yields substantially higher returns in Gems, why do the authors include suboptimal competitive behavior in half of the demonstrations? Furthermore, MA-UITD does not match the expert returns in either version. It underperforms the experts in the discrete setting but outperforms them in the continuous setting. What explains these discrepancies? For example, does the continuous version cooperate more frequently than the demonstrators?
2 comments
AuthorsOpenReview ↗

Response to Weaknesses and Question 1

We thank the reviewer for the careful reading and constructive feedback. We address the concerns and questions below.

Weaknesses and Question 1

The methodological contribution seems like too engineering. The main algorithmic components like marginalizing over others’ actions to obtain unilateral values, inverse temporal-difference learning, and NCE to handle the normalizer for continuous actions have precedents in prior work. The paper also lacks theoretical justification for combining these components in the proposed way. I am particularly concerned about the simplification in Section 4.2. Although it seems reasonable for offline data, it introduces a gap between the theoretical formulation and the implemented method. It is unclear whether this simplification preserves the earlier theoretical framework or effectively departs from it.

Questions 1: Under the simplification in Section 4.2, what role does the previously introduced multi-agent framework still play?

We provide a theoretical justification showing that, under the stated conditions, satisfying the two loss conditions in Section 4.2 is equivalent to requiring that the demonstrated joint policy constitute a logistic quantal response equilibrium (LQRE) under the recovered agent-specific rewards.

LQRE constrains.

Let π=(π1,π2)\pi=(\pi_1,\pi_2) be a stationary joint policy. For a candidate reward rir_i, define agent ii's joint-action value by

\triangleq \mathbb E_{\pi} \left[ \left. \sum_{k=0}^{\infty}\gamma^k r_i(s_{t+k+1}) \\,\right|\\, s_t=s,\\ a_{1,t}=a_1,\\ a_{2,t}=a_2 \right].$$ In a simultaneous-move game, agent 1 chooses $a_1$ before observing agent 2's current action. The relevant unilateral value therefore averages over agent 1's belief ($\mu_{1,2}$) about $a_2$: $$\begin{aligned} \bar Q_1^{\pi}(s,a_1;r_1) &\triangleq \sum_{a_2\in\mathcal A_2} \mu_{1,2}(a_2\mid s)Q_1^{\pi}(s,a_1,a_2;r_1), \end{aligned}$$ Given precision $\lambda_i>0$, define the belief conditioned logit response as: $$\text{LR}(\mu_{i,-i};r_i)(a_i\mid s) \triangleq \frac{ \exp\\!\left(\lambda_i\bar Q_i^{\pi}(s,a_i;r_i)\right) }{ \displaystyle \sum_{\tilde a_i\in\mathcal A_i} \exp\\!\left(\lambda_i\bar Q_i^{\pi}(s,\tilde a_i;r_i)\right) }.$$ A joint policy profile $\pi^\star$ is an LQRE under $R=(r_1,r_2)$ if both response equations hold simultaneously: $$\pi_1^\star=\text{LR}_1(\pi_2^\star;r_1), \qquad \pi_2^\star=\text{LR}_2(\pi_1^\star;r_2).$$ Thus, LQRE is not merely a pair of separate softmax policies. It is a coupled fixed point: each agent responds to the policy that the other agent actually uses. **Anchoring belief to observed marginal policies yields LQRE.** MA-UITD anchors each agent’s belief about the other agent’s action to the corresponding demonstrated marginal policy: $$\mu_{1,2} = \pi^D_2, \quad \mu_{2,1} = \pi^D_1$$ Given inferred $\hat{r_1}(s)$ and $\hat{r_2}(s)$, we automatically have the following because $\mathcal L_\text{UITD}$ enforces Bellman consistency, linking $\hat{r}$ to $\bar{Q}$ and their corresponding logit policies are given by: $$\pi_1 = \text{LR}(\pi_2^D; \hat{r_1}), \quad \pi_2 = \text{LR}(\pi_1^D; \hat{r_2})$$ To satisfy the constraints imposed by LQRE, we only need to satisify: $$\pi_1 = \pi_1^D, \quad \pi_2 = \pi_2^D$$ which is exactly given by behavioral cloning enforced through $\mathcal L_\text{NCE}$. Therefore, MA-UITD does not repeatedly solve a forward equilibrium problem. Instead, it treats the demonstrated policy as a candidate equilibrium and searches for rewards that make that candidate self-consistent.
AuthorsOpenReview ↗

Response to Question 2 and Question 3

Question 2

Do the reward features need to be manually specified? If so, how does this affect the method? Defining different feature dimensions for each environment based on human understanding may introduce subjective biases?

No, the reward features are learned from demonstrations as part of joint training with the behavioral and UITD objectives. We do not manually assign meanings to individual feature dimensions. The number of reward features is a model-capacity hyperparameter, and its choice may affect the inferred rewards. To assess this dependence, we have added a sensitivity analysis over reward-feature dimensions in the Appendix D, Figure 15. The results show that small feature dimensions can lead to variable reward recovery. At larger dimensions, reward recovery becomes relatively stable.

Question 3

Does full cooperation yield the highest payoff in Gems? Given that full cooperation yields substantially higher returns in Gems, why do the authors include suboptimal competitive behavior in half of the demonstrations? Furthermore, MA-UITD does not match the expert returns in either version. It underperforms the experts in the discrete setting but outperforms them in the continuous setting. What explains these discrepancies? For example, does the continuous version cooperate more frequently than the demonstrators?

Yes. Jointly collecting the purple gem gives each agent a reward of 66, whereas collecting a private red or blue gem gives the corresponding agent only 11. Full cooperation is therefore the payoff-dominant equilibrium. However, the private-gem strategy remains a second, lower-payoff competitive equilibrium. We deliberately constructed the Gems demonstrations as an equal mixture of policies specialized to these two equilibria. Thus, “expert” refers to proficiency within the demonstrated interaction mode, rather than to a dataset that globally maximizes return. This balanced, multimodal dataset tests whether a reward-inference method can represent both cooperative and competitive behavior in a mixed-motive game instead of collapsing to the higher-return cooperative mode.

Given this intentionally multimodal expert dataset, higher return does not necessarily indicate better imitation in Gems: a policy can obtain a higher return simply by favoring the cooperative equilibrium more often than the demonstrations. The upward arrows in the Gems columns of Table1/2 were therefore misleading. We evaluate Gems policies by the absolute difference between their returns and the expert reference, with a smaller gap indicating better reproduction of the demonstrated mixture. We also report MAMQL in the discrete setting and behavioral cloning as reference methods. Accordingly, we have removed the upward arrows from the Gems policy-return tables and clarified that both underperforming and outperforming the expert represent deviations from the target behavior rather than worse and better performance, respectively.

Reviewer_e5mJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MA-UITD, an offline multi-agent inverse RL method that recovers a separate reward function for each agent from fixed joint demonstrations, with no environment interaction and no inner policy-optimization loop. It extends inverse temporal-difference learning (Filos et al., 2021) to Markov games. Each agent's reward is an arrival-state reward r_i(s') = Phi(s')^T w_i, with shared features Phi and agent-specific weights w_i. Agent i's behavior is modeled by belief-conditioned successor features that average over the other agents' actions, using the empirical conditional behavior of the other agents as the belief. Training alternates between a noise-contrastive loss on the agent's own actions (which handles continuous actions) and a TD-consistency loss on the successor features. The method is tested in three simulated games (Simple Spread, Simple Push, Gems) against MA-AIRL and MAMQL with discrete actions, in continuous-action versions of the same games, and on social-foraging trajectories from pairs of mice, where it finds that followers' rewards depend more strongly on proximity to the partner than leaders' rewards do.

Critical strengths
  1. The problem setting is well motivated and practically useful: a fully offline method that gives explicit per-agent rewards and works with continuous states and bounded continuous actions, which is what behavioral neuroscience data look like. The combination of belief-marginalized successor features with an NCE loss that avoids the continuous-action normalizer is a clean and sensible design, and the paper is careful to say it targets "reward compatibility" rather than claiming to identify a unique ground-truth reward.

  2. The empirical results in the discrete simulations are encouraging. MA-UITD gets higher policy returns than both baselines in Simple Spread and Simple Push, recovers both cooperative and competitive behavior in Gems (where MA-AIRL collapses to competitive behavior), and gets better reward correlation (Pearson and Spearman) than the baselines in all three environments. The mouse application shows that the method can be run on real data and gives an interpretable hypothesis.

Critical weaknesses
  1. The evidence for the continuous-action setting, which is the main selling point, is weak. There are no baselines at all in Table 2 and Table 3, since the baselines only handle discrete actions, so it is hard to judge whether the numbers are good. The results themselves are mixed: in Simple Push the adversary's return is -2.29 against an expert -0.06, reward correlation is only 0.34 (Pearson) for the adversary, and rank correlations are around 0.3 to 0.4 everywhere. In Gems the learned policy's return (3.98) is higher than the expert's (3.34), which suggests that the policy match is not tight in the way the text implies. A natural comparison would be a discretized version of the baselines, or the single-agent ITD / an offline behavior-cloning baseline.

  2. The conclusions about reward structure depend heavily on modeling choices that are not stress-tested. The belief of each agent is set to the empirical behavior of the others, so the method assumes the demonstrators play a stationary policy profile and that agents' beliefs match what they actually do. Rewards are restricted to arrival-state features, and in the mouse analysis they are further restricted to self position, other position and inter-animal distance, so the finding that followers value proximity is partly built into the representation. The claim rests on n = 5 mouse pairs (p = 0.026, no correction mentioned) and there is no validation that the inferred rewards predict held-out behavior or beat a simple alternative model. In addition, in simulation only three seeds are used, and I did not see sensitivity analyses for the NCE proposal distribution, temperature, feature dimension or the role of the l1 penalty, even though the paper itself says the reward is non-identifiable (so different Phi and w could give the same behavior).

Questions for authors
  1. How sensitive are the recovered rewards to the choice of proposal distribution nu_i, tau_NCE and the number of negatives K? Do different seeds or different feature dimensions give rewards that agree with each other, or only policies that agree?

  2. What is the effect of replacing the empirical belief with a wrong belief (for example, when the demonstrators are not playing a stationary equilibrium, or when the data come from mixed experts)? Do the rewards degrade gracefully?

  3. For the mouse data, do the results hold if the reward features are richer or different (for example, adding velocity or reward-zone features), and does the model predict held-out trajectories better than a simple single-agent IRL or behavior-cloning baseline?

  4. Is there an explanation for why the learned continuous policy beats the expert in continuous Gems, and for the weak correlation in continuous Simple Push? Is that a failure of reward recovery or of the policy extraction step?

  5. Could you add at least one baseline for the continuous-action experiments, for example a discretized MAMQL or a single-agent offline method applied per agent? This is a suggestion for strengthening the paper and not a requirement for my assessment.

  6. Could you include more mouse pairs, or a permutation test, to support the leader-follower statistical claim? Again, a suggestion only.

2 comments
AuthorsOpenReview ↗

Response to Question 1,2,3,4

We appreciate the reviewer for the careful reading and constructive feedback. We supplemented relevant experiments. We address the concerns and questions below.

Question 1

Q1. How sensitive are the recovered rewards to the choice of proposal distribution nu_i, tau_NCE and the number of negatives K? Do different seeds or different feature dimensions give rewards that agree with each other, or only policies that agree?

We added comprehensive sensitivity analyses for the negative-action proposal distribution νi\nu_i, NCE temperature τNCE\tau_{\mathrm{NCE}}, number of negative actions KK, and reward-feature dimension (Appendix D, Figs. 13--16). Across the three tasks, reward recovery and closed-loop performance are broadly robust over a range of non-extreme settings, although the best configuration is task-dependent. The clearest capacity effect occurs for the reward-feature dimension: very small dimensions can produce poor or seed-sensitive reward recovery, whereas moderate-to-large dimensions generally yield more consistent correlations. Extreme NCE temperatures can also reduce stability, while the effects of KK and the proposal distribution are less systematic across tasks.

Question 2

Q2. What is the effect of replacing the empirical belief with a wrong belief (for example, when the demonstrators are not playing a stationary equilibrium, or when the data come from mixed experts)? Do the rewards degrade gracefully?

We added a uniform-belief implementation(Appendix C.5) in discrete Gems and found that reward recovery degrades sharply under belief misspecification. Compared with the calibrated-belief reference, the uniform-belief variant substantially reduce reward correlations, produce negative correlations for Agent 1, and increase variability across seeds(Appendix C.5 Table 11). Thus, the degradation is not just graceful under severe misspecification.

Question 3

Q3. For the mouse data, do the results hold if the reward features are richer or different (for example, adding velocity or reward-zone features), and does the model predict held-out trajectories better than a simple single-agent IRL or behavior-cloning baseline?

Yes, the leader–follower reward difference persists under alternative reward representations. We repeated the analysis using each animal’s distance to the reward zone together with inter-animal distance. The follower-greater-than-leader proximity modulation remained consistent across all five mouse pairs and under both reward representations (Appendix C.4, Fig. 12), showing that the result is not specific to using raw position features. For predictive validation, we compared MA-UITD with a supervised Gaussian behavioral-cloning baseline on held-out action ranking. At each held-out state, both methods ranked the demonstrated action against the same 16 fixed negative actions. MA-UITD achieved higher top-1 accuracy for both leaders and followers across all five mouse pairs (Appendix C.4, Fig. 11).

Question 4

Q4. Is there an explanation for why the learned continuous policy beats the expert in continuous Gems, and for the weak correlation in continuous Simple Push? Is that a failure of reward recovery or of the policy extraction step?

In continuous Gems, a return higher than the expert return should not be interpreted as better policy imitation or failure. The demonstration dataset contains a mixture of cooperative and competitive behaviors; therefore, the objective is to reproduce this behavioral mixture rather than to maximize a single scalar return. A learned policy can achieve a higher return by shifting the balance between these behaviors while becoming less faithful to the demonstrations. We therefore treat closeness to the expert return as the relevant return-based criterion and have removed the misleading upward arrows from the policy-imitation tables.

For continuous Simple Push, we stratified reward recovery by demonstration-data coverage. The adversary's recovered reward agrees most strongly with the ground-truth reward in well-sampled regions, and the correlation progressively decreases as sparsely sampled tail states are included. The result indicates that the weak overall correlation primarily reflects reward-recovery error in poorly covered regions. Because the competitive dynamics concentrate most demonstrations within a relatively narrow region of the state space, large portions of the evaluation space receive limited support. This also exposes an important limitation of MA-UITD as a fully offline method: reward recovery is reliable where the demonstration distribution provides sufficient support, but degrades in sparsely covered regions.

AuthorsOpenReview ↗

Response to Question 5 and 6

Question 5

Q5. Could you add at least one baseline for the continuous-action experiments, for example a discretized MAMQL or a single-agent offline method applied per agent? This is a suggestion for strengthening the paper and not a requirement for my assessment.

Thank you for the suggestion. We added Dis-MAMQL, trained on discretized continuous demonstrations, and an input- and capacity-matched Gaussian behavioral-cloning baseline to the continuous policy-imitation comparison (Table 2). We also report the held-out reward-recovery results for Dis-MAMQL in Appendix C.1, Table 8.

Question 6

Q6. Could you include more mouse pairs, or a permutation test, to support the leader-follower statistical claim? Again, a suggestion only.

We expanded the analysis from five to twelve mouse pairs. The leader–follower difference in proximity modulation remained significant across the full cohort(two-sided paired tt-test at d=0.07d=0.07, P=5.57×10−4P=5.57\times10^{-4}, n=12n=12 pairs; Figure 3d,e).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.