- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates teammate‑prediction auxiliary losses in cooperative MARL using value‑of‑information analysis. It finds prediction objectives may increase representation decodability but often fail to boost task performance. It also proposes an evaluation protocol for partner-modelling claims.
Critical strengths
- This paper conducts extensive experiments to explore the practical effects of teammate modelling in cooperative multi‑agent reinforcement learning.
- The proposed evaluation checklist contains several good practical advices for empirical MARL research.
Critical weaknesses
- The main contributions of this paper lie in the proposed evaluation protocol and a set of empirical observations. But I think the main experimental conclusions are over‑generalization, since most experiments are conducted on a simple environments, mainly a 5×5 Cooperative Reaching environment.
- The manuscript is difficult to follow. The overall structure and writing need improvement for better readability.
Questions for authors
- Most of the main empirical findings are established in a simplified gridworld. To what extent do the authors expect these findings to generalize to more complex cooperative tasks?
- What happens if we test against totally new teammates the agent has never seen before?
Response to Reviewer PaqL
Thank you for your detailed feedback and for recognizing the extensive experiments and practical value of the evaluation protocol.
Experiments scope: Our contribution addresses a general evaluation problem in cooperative RL determining whether auxiliary teammate prediction contributes to better coordination. The questions of whether partner information matters to a decision, whether the agent represents it, and whether prediction training improves return can be asked in any cooperative task, whatever its size or complexity.
The controlled tasks allow us to independently vary available information and relevant rewards, while the commitment task provides the reference. These experiments demonstrate why increased decodability or a return gain over an unmatched baseline can leave the contribution of predition training unresolved. The emperical effects of individual objectives may differ in larger teams and more complex tasks.
Questions for Authors
Q1. To what extent do the findings generalize to more complex cooperative tasks?
We expect the distinction between predictability and decision value, and the need to control for architectural and training-budget effects, to remain relevant in more complex tasks. Establishing the magnitude and direction of a particular prediction objective’s effect requires experiments in those settings.
Appendix F provides one such evaluation in OvercookedV2, using one layout, a frozen partner, and 20 seeds per arm. Training with a privileged recipe-prediction target increases probe accuracy from 0.38 to 0.81, while task return remains equivalent within the prespecified five-point margin (+0.74, 95% CI [−2.22, +3.60]). The next-action head also raises probe accuracy, from 0.38 to 0.51 (+0.130 [0.113, 0.146]), and its return comparison, like the feature-reading head’s, is inconclusive.
These results provide a further instance of improved decodability without an established return benefit. The additional policy branch’s benefit in Cooperative Reaching was not established in this setting, illustrating why the effects of particular interventions should be evaluated separately across tasks.
Q2. What happens with teammates the agent has never seen before?
The details of two complementary evaluations are in Appendix C
In Table 13 Followers train against a population of frozen leaders and encounter different leaders only at evaluation. For followers trained without a prediction objective, the results are as follows.
Followers of unrewarded leaders drop from 0.637 to 0.522 with new leaders, while followers of rewarded leaders show little change in mean return. Relative to the unrewarded-leader condition, held-out return is higher by +0.185 [0.112, 0.262] with progress-reward leaders and +0.193 [0.121, 0.270] with legibility-reward leaders. These comparisons reflect both leader quality and transfer across leaders.
In Table 14, Independently trained agents are paired without further training. No-objective agents exceed the 0.5 return threshold in 22 of 60 cross-seed pairings, compared with 45 of 60 original training pairs. This indicates that coordination is partly specific to the training pair.
Both evaluations involve previously unseen policies produced under the same task and training conditions. Teammates with deliberately different conventions or learning procedures would test a broader form of generalization.