- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MA-UITD, an offline framework for inferring agent-specific arrival-state rewards from joint demonstrations. It combines shared reward features and individual preference vectors with belief-conditioned successor features that marginalize other agents’ behavior. Learning alternates between behavioral fitting and inverse temporal-difference consistency, with a contrastive objective for continuous actions. Experiments examine policy returns and reward recovery in cooperative, competitive, and general-sum environments. An application to mouse social-foraging trajectories produces different spatial and proximity-related reward components for leaders and followers. The objective is to recover behaviorally compatible rewards rather than uniquely identify underlying preferences.
Critical strengths
- The problem is well motivated by behavioral experiments, where fixed trajectories may be available without reward labels, known dynamics, or opportunities for additional interaction. A framework that directly estimates explicit individual rewards from continuous joint behavior addresses a meaningful practical need.
- The synthetic evaluation examines both task performance and held-out reward agreement across different interaction types, with substantial improvements over the reported discrete-action baselines. The mouse application illustrates a potentially useful way to characterize individual differences in social behavior. The explicit acknowledgment of reward non-identifiability also appropriately limits the interpretation of the results.
Critical weaknesses
- The methodological advance over the closest prior work is insufficiently established. ITD already learns shared reward features, individual preference vectors, and successor features through behavioral fitting and inverse TD consistency. MAMQL already uses opponent-marginalized values for individual action selection. MA-UITD combines these ideas and extends them to continuous actions, but the additional learning capability introduced by the belief-conditioned representation remains unclear. In particular, the implementation calibrates beliefs to demonstrated behavior rather than inferring them separately. The formulation is reasonable, but the current presentation does not convincingly establish a substantial conceptual advance beyond adapting existing mechanisms to this setting.
- The evaluation does not sufficiently isolate the benefit of the proposed formulation over its closest methodological foundation. Shared reward features, individual preference vectors, behavioral fitting, and inverse TD consistency are inherited from ITD. Since the implementation calibrates beliefs to demonstrated behavior rather than learning them separately, a matched ITD-style control using the joint state is particularly important. The manuscript provides neither this comparison nor component ablations, and the continuous-action tables report no competing method. Consequently, the results establish promising performance but leave the added value of the proposed multi-agent construction insufficiently demonstrated.
Updated comments: I have decided to lower my score because the author did not respond to any of my comments, especially regarding the difference between the method used in this article and similar prior benchmark methods.
Questions for authors
- Clarification and targeted validation: Under the assumption that predictive beliefs match demonstrated opponent behavior, what methodological changes distinguish MA-UITD from ITD adapted to joint-state inputs and agent-specific successor-feature heads? A comparison with matched inputs, capacity, and training budget would materially help assess the contribution.
- Targeted validation: Do the inferred leader–follower reward differences remain stable across training seeds and reasonable regularization choices? Does including the distance-dependent reward component improve prediction of held-out mouse actions over an otherwise matched model?
- Technical clarification: Appendix B.3 samples negative actions from the expert’s nonuniform marginal action distribution, but Eq. (10) contains no explicit proposal-density correction. How is this objective reconciled with the Boltzmann policy in Eq. (7) and the rollout procedure in Appendix B.4? Is a correction implemented, or is a different base measure intended?
4 comments
Response to Weaknesses and Question 1 part 2
Recall the LQRE constrains in two agents: given precision , define the belief conditioned logit response as:
\triangleq \frac{ \exp\\!\left(\lambda_i\bar Q_i^{\pi}(s,a_i;r_i)\right) }{ \displaystyle \sum_{\tilde a_i\in\mathcal A_i} \exp\\!\left(\lambda_i\bar Q_i^{\pi}(s,\tilde a_i;r_i)\right) }.$$ A joint policy profile $\pi^\star$ is an LQRE under $R=(r_1,r_2)$ if both response equations hold simultaneously: $$\pi_1^\star=\text{LR}_1(\pi_2^\star;r_1), \qquad \pi_2^\star=\text{LR}_2(\pi_1^\star;r_2).$$ Our inverse problem starts from the demonstrated joint policy $\pi^D=(\pi_1^D,\pi_2^D)$ and **searches for rewards $\hat R=(\hat r_1,\hat r_2)$ that make this observed profile an LQRE:** $$\pi_1^D=\text{LR}_1(\pi_2^D;\hat r_1), \qquad \pi_2^D=\text{LR}_2(\pi_1^D;\hat r_2).$$ Consequently, agent 1’s unilateral value must evaluate its learned reward while agent 2 follows its demonstrated policy: $$\bar Q_1^{\pi^D}(s,a_1;\hat r_1) =\mathbb{E}_{\substack{ a_2\sim\pi_2^D(\cdot\mid s),\\ s'\sim P(\cdot\mid s,a_1,a_2),\\ a_1'\sim\pi_1^D(\cdot\mid s') }} \left[ \hat r_1(s') + \gamma\, \bar Q_1^{\pi^D}(s',a_1';\hat r_1) \right],$$ with an analogous recursion for agent 2. After applying the reward-feature factorization and estimating these expectations with demonstrated transitions, this recursion gives exactly the $\mathcal{L}_{\text{UITD}}$ in main text Equation 12. The behavioral objective then enforces that each demonstrated policy is the corresponding logit response. Thus, the two objectives directly impose the inverse LQRE conditions without repeatedly solving a forward equilibrium problem. This derivation explains why MA-UITD coincides computationally with single-ITD on the joint state using agent-specific successor-feature heads. Beyond forwardly deriving MA-UITD from the LQRE constraints, we argue that calibrating predictive beliefs to the demonstrated opponents’ behavior is essential for model-free reward inference in the fully offline setting. Fixed demonstrations do not separately identify beliefs and rewards: different combinations can explain the same observed behavior. Moreover, evaluating alternative beliefs generally require counterfactual transitions that are unavailable without an explicit dynamics model or additional environment interaction. Anchoring beliefs to the demonstrated opponent policies is therefore not arbitrary; Mechanistically, it is the assumption that makes reward inference identifiable and estimable from the available data. **Potential belief inference within the MA-UITD framework** To further illustrate the value of the proposed MA-UITD framework, we also describe a possible inverse-planning application in which MA-UITD can infer an agent’s internal belief when the reward functions and transition dynamics are known. For example, we can parameterize agent $i$’s belief as $\mu_{\theta_i,-i}$ and infer $\theta_i$ from its observed choices. A similar behavioral cloning loss could help the inference: For each candidate belief, the corresponding unilateral value is defined by $$\bar{Q}_i^{\theta_i}(s,a_i) =\mathbb{E}_{ a_{-i}\sim\mu_{\theta_i,-i}(\cdot\mid s),\, s'\sim P(\cdot\mid s,a_i,a_{-i}),\, a_i'\sim\pi_i^D(\cdot\mid s') } \left[ r_i(s') + \gamma\bar{Q}_i^{\theta_i}(s',a_i') \right].$$ The belief parameters can then be estimated by minimizing the negative log-likelihood of the demonstrated actions: $$\mathcal{L}_{\mathrm{belief}}(\theta_i) =- \mathbb{E}_{(s,a_i)\sim\mathcal{D}} \left[ \log \operatorname{LR}_i \left( \mu_{\theta_i,-i};r_i \right) (a_i\mid s) \right].$$ This loss searches for the predictive belief under which the known reward makes the demonstrated actions most likely. It does not require the demonstrated interaction to satisfy an equilibrium condition as individual agent’s subjective belief need not equal the opponents’ actual or demonstrated policies. Only given known rewards and transition dynamics, the loss above can therefore be used to infer an agent’s belief under any form of multi-agent interaction captured by the specified response model and belief class. We do not include this extension in the current manuscript because our focus is the fully offline setting with unknown rewards, unknown dynamics, and no additional environment interaction.Response to Question 2 and Question 3
Question 2
Q2. Do the inferred leader–follower reward differences remain stable across training seeds and reasonable regularization choices? Does including the distance-dependent reward component improve prediction of held-out mouse actions over an otherwise matched model?
Yes, the inferred leader-follower reward differences remained stable across training seeds. Appendix C.4 Figure 9 reports inferred-reward correlations across seeds for each mouse pair.
Adding an explicit inter-animal-distance component produced a role-specific improvement in held-out action prediction. Relative to the capacity-matched additive position model, the distance-augmented model consistently reduced held-out NCE for followers across all five mouse pairs and all three training seeds (mean ), whereas the improvement for leaders was small and inconsistent (mean ). This result suggests that follower actions contain social-distance-dependent structure beyond additive spatial effects(Appendix C.4 Figure 10).
Question 3
Q3. Appendix B.3 samples negative actions from the expert’s nonuniform marginal action distribution, but Eq. (10) contains no explicit proposal-density correction. How is this objective reconciled with the Boltzmann policy in Eq. (7) and the rollout procedure in Appendix B.4? Is a correction implemented, or is a different base measure intended?
Thank you for identifying this ambiguity. The original manuscript described only the empirical-marginal negatives used for the animal data and did not state that uniform negatives were used in the simulation experiments. We have corrected this omission.
For animal data, no proposal-density correction is implemented because the negative-action distribution is an intentional model choice that defines the base measure. We use a uniform base measure for simulated agents, where no action tendency is privileged by default. For the animal data, we use each animal’s empirical marginal action distribution, assuming that it reasonably approximates the animal’s baseline action biases. However, we also note that this is a domain-motivated modeling assumption rather than a universally required choice; other base measures would represent different assumptions about baseline behavior.
Rollout candidates are drawn from the same base measure used during training, ensuring consistency between training and evaluation. A density correction would be required only if the empirical marginal were used merely as a sampling proposal while targeting a uniform-base policy. We also added a sensitivity analysis of the negative-action distribution in Appendix D, Figure 12.
A reminder
We apologize again for our delayed response while we completed and validated the additional experiments. We have now posted a detailed point-by-point response to all of your comments, including a substantive clarification of MA-UITD’s relationship to ITD, as well as the requested baseline comparisons and ablations. Since your updated score was explicitly based on the absence of a response, we would be grateful if you could review our response and revised manuscript and reconsider your assessment in light of this new information.
Response to Weaknesses and Question 1 part 1
We thank the reviewer for the careful reading and constructive and incisive feedback. We apologize for our delayed response; we were completing and validating the additional simulations needed to address the reviewer’s questions thoroughly.
Weaknesses and Question 1
We acknowledge that, conceptually, an ITD control using joint state as input and agent-specific successor-feature heads would be similar to the implemented MA-UITD estimator. This similarity reflects the power and generality of the ITD framework, but ITD alone does not specify which value function is behaviorally relevant in a Markov game, how other agents’ behavior should enter that value, or under what conditions it can be estimated from fixed joint demonstrations. MA-UITD addresses these questions through belief-conditioned unilateral values.
This specific structure of MA-UITD, therefore, is not an ad hoc reuse of single-agent ITD; it follows from the logistic quantal response equilibrium (LQRE) formulation of a Markov game. Under LQRE, each agent responds logistically to the expected value of its own action after marginalizing over the simultaneous actions of the other agents. Below, we provide a justification here to show why this form emerges and why any other implementation wouldn't work.