- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper propose PCMA that learns coordinated agent preferences to enable complementary tradeoffs. The authors formulate it as a team-optimal equilibrium problem, and show that preference diversity yields a first-order improvement in the team objective. Experiments on multiple cooperative environments show that PCMA improves both performance and trade-off coordination.
Critical strengths
- The motivation of this paper is very clear and practical.
- Most parts is presented clearly and the whole article is easy to follow.
- The method designed is valid intuitively.
Critical weaknesses
-
In partially observable domains, as agents cannot directly observe the state, they need to rely on a local history to infer state information [1]. However, in Equation 1 and 2, they formulate the planner and actor as depending solely on the current local observation and . Unless an additional memory mechanism is introduced, the policies derived from this research is theoretically limited.
-
There is a big gap between what the experiments trying to validate and the actual solution concept of their method, PCMA. The paper models the problem as a MOPOSG, where each agent has its own individual reward . Even if we could get such a team-level equilibrium under the ideal condition, it does not necessarily imply global optimality, so overall performance is not guaranteed to improve technically. However, the experiments are trying to show that those evaluation metrics at the team level are indeed improved by PCMA, e.g., success rate, which is a significant mismatch.
-
Using average reward as an evaluation metric is weird, because the objective in RL optimization is typically expected return (-discounted cumulative reward following the target policy [2]). I can understand this given that the authors use a MOPOSG framework, meaning each agent has a different reward scale, so averaging is needed. Even so, the metric should still be "average return". This paper never explicitly clarifies whether "average reward" refers to the average across agents, or the average across both agents and steps. If it is the latter one, the evaluation would be uninformative, since RL is not going to optimize average rewards per step.
References:
[1] Planning and acting in partially observable stochastic domains. Kaelbling et. al.
[2] Reinforcement Learning: An Introduction. Sutton et. al.
[3] SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. Ellis et. al.
Questions for authors
-
This paper mainly discusses formulation of MOPOSG, whereas cooperative multi-agent reinforcement learning is more broadly formulated as Dec-POMDPs. While I understand that Dec-POMDPs are only suited for fully cooperative settings, many of the testbeds used in these experiments, such as StarCraft, are more commonly modeled as Dec-POMDPs. Is there a specific reason for modeling this on a POSG?
-
As the SMAC v2 paper notes, the SMAC v1 tested in this paper actually don't require on-loop control [3]. So I'm wondering how can we show that agents are indeed deciding their preferences and roles based on the input?
7 comments
W1: The planner and the actor depend only on the current observation.
We agree that agents with memory can mitigate partial observability. However, our research focus is not partial observability but the coordination among different objectives. Moreover, PCMA extends to memory directly: replacing the observation with the action-observation history , i.e., and , requires no other change. We report the results of replacing the simple feed-forward policy of the actor and the planner with an RNN on SMACv2. Recurrent PCMA achieves a higher mean success rate than recurrent MAPPO and IPPO on both maps.
| SMACv2 success rate | Terran RNN | Zerg RNN |
|---|---|---|
| PCMA (Ours) | 0.69±0.05 | 0.47±0.03 |
| MAPPO | 0.64±0.06 | 0.38±0.04 |
| IPPO | 0.63±0.07 | 0.36±0.05 |
W2: Gap between the solution concept and what the experiments validate.
Section 3.2 defines the solution concept in terms of the team return at equilibrium.
First, the solution is a team-optimal equilibrium. The mo-POSG in Section 3.1 specifies what each agent observes and receives, including its own vector reward. An equilibrium only means that no agent can improve its own utility by unilaterally changing its policy; as the reviewer notes, it therefore need not be optimal for the team. As is pointed out in Section 3.2, this is exactly why the preferences must be coordinated. Different preference profiles induce different equilibria , and since the agents share a common interest, the team objective is the natural criterion for selecting among them. Therefore, we do not take an arbitrary equilibrium as the solution, but coordinate so that the induced equilibrium maximizes the team objective, .
Second, both our theory and our evaluation target this team objective. On the theory side, Theorem 4.2 gives the first-order team improvement under a preference profile; accordingly, the planner that sets is trained with the team advantage. For tasks with a success criterion, we report how often the team completes the task. Since task completion contributes positively to the team reward, we expect a higher success rate to be associated with higher . Other reward components, such as collision penalties in Predator-Prey, also affect the team return.
W3: "Average reward" as an evaluation metric.
We apologize for the imprecise terminology. The correct term is “average return,” which we compute by averaging episodic returns across agents, objectives, and evaluation episodes. We will use this term and clarify its definition in the paper.
References
- Eric A. Hansen, Daniel S. Bernstein, and Shlomo Zilberstein. Dynamic Programming for Partially Observable Stochastic Games. AAAI, 2004.
- Florian Felten et al. MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learning. arXiv:2407.16312, 2024.
Thanks for your response, after carefully reading the author responses, the reviewer still feels unconvinced by most of the main complaints. So, the reviewer decided to maintain the original evaluation.
We thank the reviewer for the careful review. Before answering point by point, we would like to clarify what the paper studies. PCMA addresses an under-explored problem, multi-objective multi-agent RL, where each agent faces its own trade-off among multiple objectives while also sharing a common interest as a system. And our focus is preference coordination: how agents with different trade-offs coordinate their preferences toward this shared interest. We respond to the questions and concerns below.
Q1: Why model the problem as an MO-POSG rather than a Dec-POMDP?
We went through a careful discussion before choosing this formulation. Note that a Dec-POMDP is the special case of a POSG in which all agents share the same reward function (Hansen et al., 2004), and we choose to go beyond this case, for two reasons.
First, one shared preference defines a team-level trade-off. If all agents scalarize a common reward vector with the same preference, the resulting objective expresses one trade-off for the team. Therefore, we take the mo-POSG of Felten et al. (2024, Definition 1) as our formulation. This formulation is also practically meaningful. Take a basketball team as an example: every player wants to win, but a center and a guard weigh scoring, defending and ball protection differently. Furthermore, the mo-POSG captures a modeling case that is common but largely overlooked in practice: agents share a common interest but face different trade-offs, which also arises in multi-LLM-agent systems, drone swarms, etc.
Second, shared feedback makes role-specific preferences harder to learn. PCMA aims to let each agent learn a preference suited to its role. A shared reward vector records the team outcome on each objective. From that feedback alone, it is difficult to tell which agent should give more weight to a particular objective. In our formulation, each agent additionally receives its own vector reward and maximizes Ui(θ;pi)=Jteam(θ)+pi⊤Ji(θ), so each agent trades off its own objectives Ji, with Jteam capturing the team's shared interest. Hence, the problem changes from a cooperative game with a common payoff into a preference-conditioned general-sum game G(p), and the question we focus on is how agents coordinate their preferences, rather than multi-agent credit assignment.
Q2: SMACv1 does not require closed-loop control. Do agents decide their preferences and roles from their input?
First, agents can decide their preferences from their observations, and this ability is learned in training. The planner maps each agent's local observation, including its unit type, health and the relative positions of allies and enemies, to its preference. For example, under our two-dimensional SMAC reward (damage dealt and damage taken), a unit that observes that it is on the front line can lower its weight on avoiding damage and hold its position. To further validate this, we evaluate trained PCMA policies with each agent's preference replaced by a constant, namely that agent's planner output averaged over all time steps of the evaluation episodes, so that each agent keeps its own preference but can no longer adjust it to the current observation. This lowers the success rate:
Second, in most cases, the SMACv1 learning curves already start at high success rates. To test PCMA under greater difficulty, we added the Terran and Zerg 5 vs. 4 tasks from SMACv2 with extended partial observability. Both tasks randomize unit types and starting positions each episode. PCMA maintains the highest success rate on both maps: