acceptodds
Under review as a conference paper at ICLR 2027

The Decision Value of Partner Information in Cooperative Reinforcement Learning

Abstract

Cooperative agents often need information that is available only to their partners. Auxiliary objectives that predict a teammate’s behavior can improve internal representations, but prediction accuracy alone does not establish whether those representations improve coordination. We study this distinction by applying classical value of information to cooperative decisions and independently varying the availability and reward relevance of a partner’s private information. Experiments in an exactly solvable commitment task and modified Cooperative Reaching environments separate the value of observing a partner from the benefit of training an agent to predict it. With memory trained through time, recurrent policies without prediction objectives recover the available history value in the toy task and match a scripted-partner reference in the continuously observed reaching task. Auxiliary prediction can substantially increase the decodability of a reward-irrelevant partner variable while leaving return equivalent within a prespecified margin. When partner observations are restricted to an initial window, however, some co-learning runs remain below a prespecified task-return threshold at training budget. In this setting, a reward for behavior that makes the partner’s goal inferable increases the fraction of training runs exceeding the threshold from 42/60 to 60/60 and improves mean task return by 0.131 (95% confidence interval [0.085, 0.181]), excluding the auxiliary reward; a matched reward for progress toward the goal does the same, and given four times the budget the unrewarded runs cross the threshold too, so the gain is one of speed. Teammate prediction objectives yield no statistically resolved improvement; an extra input branch read by the policy helps, whether or not it is trained to predict the partner. These results distinguish representational gains from behavioral gains and show that, under restricted observation, encouraging informative partner behavior can speed up coordination where prediction objectives on their own do not improve coordination.

Primary area:
reinforcement learning
Keywords:
Multi-agent reinforcement learning, Cooperative decision-making, Teammate modeling, Value of information, Implicit communication

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PaqLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates teammate‑prediction auxiliary losses in cooperative MARL using value‑of‑information analysis. It finds prediction objectives may increase representation decodability but often fail to boost task performance. It also proposes an evaluation protocol for partner-modelling claims.

Critical strengths
  1. This paper conducts extensive experiments to explore the practical effects of teammate modelling in cooperative multi‑agent reinforcement learning.
  2. The proposed evaluation checklist contains several good practical advices for empirical MARL research.
Critical weaknesses
  1. The main contributions of this paper lie in the proposed evaluation protocol and a set of empirical observations. But I think the main experimental conclusions are over‑generalization, since most experiments are conducted on a simple environments, mainly a 5×5 Cooperative Reaching environment.
  2. The manuscript is difficult to follow. The overall structure and writing need improvement for better readability.
Questions for authors
  1. Most of the main empirical findings are established in a simplified gridworld. To what extent do the authors expect these findings to generalize to more complex cooperative tasks?
  2. What happens if we test against totally new teammates the agent has never seen before?
1 comment
AuthorsOpenReview ↗

Response to Reviewer PaqL

Thank you for your detailed feedback and for recognizing the extensive experiments and practical value of the evaluation protocol.

  • Experiments scope: Our contribution addresses a general evaluation problem in cooperative RL determining whether auxiliary teammate prediction contributes to better coordination. The questions of whether partner information matters to a decision, whether the agent represents it, and whether prediction training improves return can be asked in any cooperative task, whatever its size or complexity.

  • The controlled tasks allow us to independently vary available information and relevant rewards, while the commitment task provides the reference. These experiments demonstrate why increased decodability or a return gain over an unmatched baseline can leave the contribution of predition training unresolved. The emperical effects of individual objectives may differ in larger teams and more complex tasks.

Questions for Authors

Q1. To what extent do the findings generalize to more complex cooperative tasks?

We expect the distinction between predictability and decision value, and the need to control for architectural and training-budget effects, to remain relevant in more complex tasks. Establishing the magnitude and direction of a particular prediction objective’s effect requires experiments in those settings.

Appendix F provides one such evaluation in OvercookedV2, using one layout, a frozen partner, and 20 seeds per arm. Training with a privileged recipe-prediction target increases probe accuracy from 0.38 to 0.81, while task return remains equivalent within the prespecified five-point margin (+0.74, 95% CI [−2.22, +3.60]). The next-action head also raises probe accuracy, from 0.38 to 0.51 (+0.130 [0.113, 0.146]), and its return comparison, like the feature-reading head’s, is inconclusive.

These results provide a further instance of improved decodability without an established return benefit. The additional policy branch’s benefit in Cooperative Reaching was not established in this setting, illustrating why the effects of particular interventions should be evaluated separately across tasks.

Q2. What happens with teammates the agent has never seen before?

The details of two complementary evaluations are in Appendix C

In Table 13 Followers train against a population of frozen leaders and encounter different leaders only at evaluation. For followers trained without a prediction objective, the results are as follows.

Leader populationTraining-pool returnHeld-out return
No leader reward0.6370.522
Progress reward0.7170.707
Legibility reward0.7150.714

Followers of unrewarded leaders drop from 0.637 to 0.522 with new leaders, while followers of rewarded leaders show little change in mean return. Relative to the unrewarded-leader condition, held-out return is higher by +0.185 [0.112, 0.262] with progress-reward leaders and +0.193 [0.121, 0.270] with legibility-reward leaders. These comparisons reflect both leader quality and transfer across leaders.

In Table 14, Independently trained agents are paired without further training. No-objective agents exceed the 0.5 return threshold in 22 of 60 cross-seed pairings, compared with 45 of 60 original training pairs. This indicates that coordination is partly specific to the training pair.

Both evaluations involve previously unseen policies produced under the same task and training conditions. Teammates with deliberately different conventions or learning procedures would test a broader form of generalization.

Reviewer_Ef1SOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tries to propose an analysis based on the "value of partner information" in cooperative agents. The goal seems to be to study the distinction between using and not using auxiliary objectives that predict teammate's behaviours. The experiments try to quantify the difference between the value of observing a partner from the value of predicting it. I couldn't find specific conclusions but only analysis of experimental results showing that "auxiliary prediction can increase the decodability of a reward-irrelevant partner variable while leaving return equivalent within a prespecified margin".

Critical strengths
  • Declared goal: The goal of the paper could be interesting, namely to provide an empirical framework for separating task-level decision value, what is represented, and what affects behavior.
  • Relevance to the conference: The paper addresses cooperative multi-agent learning, partial observability, representation learning, and evaluation methodology, all relevant to the ICLR community.
  • Bibliography: The references seem to be quite complete and up-to-date
  • Reproducibility: authors attach the code which should allow the reader to reproduce the results
Critical weaknesses
  • Writing quality and paper organization: the paper has a very low-level writing quality and organization in my opinion. For instance, in the introduction there is no motivation statement, no problem definition/formalization, no clear comparison with the literature to identify the methodological contributions, no clear explanation of the novelty in the contributions. In the tenth line of the paper a test is already introduced without specific motivation. Results charts are provided already un the second page (see Fig 1) where the reader does not not know anything about the problem, the method and the experimental setting. This is completely incomprehensible in my opinion. The rest of the introduction provides a description of the contributions that are too related to specific tests and do not draw any general conclusion. Also the rest of the paper is badly organized and written in my opinion. This makes the paper impossible to be evaluted w.r.t. several standard review dimensions, such as, novelty/originality, technical soundness, etc.
  • significance/impact: The paper presents a set of experiments, but offers no methodological guidance or general principles that the research community can apply beyond the tested settings. It therefore provides little basis for understanding what broader lessons follow from the results. In my view, this limits the paper’s significance and likely impact.
  • novelty/originality: I cannot see specific novelty in the proposed contribution. A clear methodological comparison w.r.t the state of the art is missing
  • technical soundness: it is very difficult to understand the message of the paper in the current form, so I cannot evaluate the technical soundness.
Questions for authors
  1. Can you please summarize what the reader can learn as a general principle from the paper and how it can be use this in future research developments of cooperative RL? 2. Can you summarize in a few sentences i) the problem you tackle; ii) the motivation why it is useful to solve it; iii) the methodological improvement you provide? 3. Can you please identify the closer paper in the literature to your work? What is the specific methodological improvement you provide w.r.t. this work?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Ef1S

Thank you for your time in writing this detailed feedback. We agree that the introduction should explain the research question and its purpose more clearly.

  • Presentation: In the revised manuscript, We introduce the cooperative decision problem first, define the information available to each agent, and explain the evaluation procedure before presenting Figure 1. The introduction now summarises the evaluation protocol of Appendix G.

  • Significance: Our paper helps researchers determine whether training an agent to predict its teammate actually improves coordination. In our experiments, Adding a component trained to predict teammate improved performance. Keeping that component while removing its prediction training also improved the performance but keeping that component while removing its prediction training improved it equally. Without these comparisons, we would have credited teammate prediction for a gain achievable without it. See Q1 for the general principles.

  • Technical Soundness: We hope the summary in Q2 makes the paper assessable All the numbers are reported with confidence intervals over multiple seeds with controls that match added architecture.

  • Novelty and Originality: A partner that waits in the same corner regardless of the task's goal may be easy to predict while revealing little to no useful information whereas a partner that moves toward the required destination provides that changes the teammates decision. Our framework makes this distinction measurable and tests whether an auxiliary prediction objective helps the learner obtain the benefit. Please refer Q3 for closest worj and our specific improvement.

Questions for Authors

Q1. General principles. (1) Before adding a teammate-prediction objective, check whether the teammate’s information can change any rewarded decision; if it cannot, no objective can improve return. (2) Probe or prediction accuracy is not evidence of coordination: in our experiments a next-action head raised decodability of a reward-irrelevant partner variable from 0.42 to 0.98 with return unchanged. Report decodability and return separately, against a control with the same added architecture. (3) Compare at a longer training budget before claiming a better policy; both gains we found were gains in learning speed.

Q2. Problem, motivation, improvement. (i) Teammate-prediction objectives are added to cooperative agents on the assumption that predicting a partner improves coordination, but published gains report return only. (ii) Without separating what the objective adds, a gain from extra capacity or faster learning is credited to prediction: in our experiments the gain came from an added input branch that helped equally with random labels or no prediction loss, and it disappeared at a longer budget. (iii) We provide a test that computes the value of the partner’s information for the task, then compares the objective against matched architectural controls, a partner swap and a longer budget.

Q3. Closest work. Closest in aim is Tessera et al. (2026), who characterise learned cooperative policies with diagnostics defined under the converged policy; they measure statistical dependence on the partner’s private information, where our partner swap measures it by intervention, against a ceiling computed from the task. Closest in method are the auxiliary agent-modelling architectures of Hernández-Leal et al. (2019), whose AMS and AMF designs we test directly, and LIAM (Papoudakis et al., 2021), which we run as a variant. Our improvement over both is the control set: varying whether the partner’s variable is observable and whether it affects reward, and testing the objective against controls that keep its architecture but remove its prediction content. The revised related work states both relationships explicitly.

I thank the authors for their explanation and answers. I'll consider them in the final discussion with the other reviewers.

Reviewer_UtLiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether learning to predict a teammate actually helps an agent make better decisions. Its experiments show that an agent can become much better at recognizing its teammate’s goal without earning more reward. Where coordination improves, the benefit can come from extra processing capacity or faster training rather than predicting the teammate.

Critical strengths
  1. Good experimental design: The paper uses ranodm prediction targets and extra layers without prediction training to test what causes the improvements.
  2. Well supported experiments: Repeated experiments, longer training, and distinction between negligible and uncertain results, raise the paper's credibility.
Critical weaknesses
  1. Figure 1 has overlapping text with the title.
  2. Most findings come from two agent decision problem. More complicated teamwork may behave differently.
  3. Previous work, like InfoPG (https://arxiv.org/abs/2201.08484) or Social Influence... (https://arxiv.org/abs/1810.08647) already touch this topic. Contrasting to them would be useful.
Questions for authors

Can the method estimate decision benefits when teammate behavior is unknown?

1 comment
AuthorsOpenReview ↗

Response to Reviewer UtLi

Thank you for your detailed feedback and for recognizing the experimental controls and repeated evaluations.

  • The overlapping text in Figure 1 is corrected in the revised manuscript, and the revised introduction explains the cooperative decision problem and evaluation procedure before the figure appears.

  • We agree that larger teams may behave differently. Our controlled tasks allow us to independently vary whether partner information is observable and whether it affects reward. This helps identify why an auxiliary prediction objective does or does not improve coordination. With additional teammates, information may be redundant or complementary, so its value must be assessed alongside the other information available to the agent. Our empirical conclusions concern the two-agent settings evaluated.

  • InfoPG (Konan et al., 2022) introduces iterated reasoning about neighboring agents and connects its policy gradient formulation to a mutual information lower bound. Social Influence (Jaques et al., 2019) rewards agents for the counterfactual effect of their actions on other agents’ behavior. Our analysis asks how much revealing a partner variable can improve a rewarded decision, then tests whether auxiliary prediction training helps obtain that benefit. We will expand the related work discussion to explain these connections.

Our leader-legibility reward also relates to prior work on informative behavior. In our experiments, a matched progress reward produces a similar finite budget improvement. This limits what we can credit specifically to encouraging informative behavior.

Questions for Authors

  1. The learner can interact with a teammate without knowing its policy parameters. Exact calculation of the information value, however, requires a model of teammate behavior and the relevant action values.

When repeated interaction and controlled information access are available, the empirical protocol can evaluate an unknown teammate. Holding that teammate fixed, we can compare policies trained with restricted observations, observation history, and privileged access to the relevant partner variable. These comparisons estimate benefits achieved by the tested policies and may also include optimization effects. For a population of teammates, the same comparisons can be made over a specified evaluation distribution. Establishing an exact optimal value or a general online estimator for unseen teammates requires additional modeling and validation. We will also clarify this distinction in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.