acceptodds
Under review as a conference paper at ICLR 2027

Learning Coordinated Preference for Multi Objective Multi Agent Reinforcement Learning

Abstract

Cooperative multi-objective multi-agent reinforcement learning (MOMARL) models team decision making under multiple, potentially conflicting objectives. In this setting, conflicts arise not only across objectives but also across agents with different observations, roles, and contributions. We propose Preference Coordinated Multi-agent Policy Optimization (PCMA), which learns coordinated agent-specific preferences to enable complementary trade-offs among agents. Theoretically, we formulate cooperative MOMARL as a team-optimal equilibrium problem, and show that, under suitable conditions, preference diversity yields a first-order improvement in the team objective. Experiments on multiple cooperative MOMA environments and a practical traffic-control scenario show that PCMA improves both performance and trade-off coordination.

Primary area:
reinforcement learning
Keywords:
MOMARL, Team-optimal Equilibrium, Preference Coordination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xhftOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper propose PCMA that learns coordinated agent preferences to enable complementary tradeoffs. The authors formulate it as a team-optimal equilibrium problem, and show that preference diversity yields a first-order improvement in the team objective. Experiments on multiple cooperative environments show that PCMA improves both performance and trade-off coordination.

Critical strengths
  1. The motivation of this paper is very clear and practical.
  2. Most parts is presented clearly and the whole article is easy to follow.
  3. The method designed is valid intuitively.
Critical weaknesses
  1. In partially observable domains, as agents cannot directly observe the state, they need to rely on a local history to infer state information [1]. However, in Equation 1 and 2, they formulate the planner and actor πi\pi_i as depending solely on the current local observation oio_i and pip_i. Unless an additional memory mechanism is introduced, the policies derived from this research is theoretically limited.

  2. There is a big gap between what the experiments trying to validate and the actual solution concept of their method, PCMA. The paper models the problem as a MOPOSG, where each agent has its own individual reward RiR_i. Even if we could get such a team-level equilibrium under the ideal condition, it does not necessarily imply global optimality, so overall performance is not guaranteed to improve technically. However, the experiments are trying to show that those evaluation metrics at the team level are indeed improved by PCMA, e.g., success rate, which is a significant mismatch.

  3. Using average reward as an evaluation metric is weird, because the objective in RL optimization is typically expected return (γ\gamma-discounted cumulative reward following the target policy [2]). I can understand this given that the authors use a MOPOSG framework, meaning each agent has a different reward scale, so averaging is needed. Even so, the metric should still be "average return". This paper never explicitly clarifies whether "average reward" refers to the average across agents, or the average across both agents and steps. If it is the latter one, the evaluation would be uninformative, since RL is not going to optimize average rewards per step.

References:

[1] Planning and acting in partially observable stochastic domains. Kaelbling et. al.

[2] Reinforcement Learning: An Introduction. Sutton et. al.

[3] SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. Ellis et. al.

Questions for authors
  1. This paper mainly discusses formulation of MOPOSG, whereas cooperative multi-agent reinforcement learning is more broadly formulated as Dec-POMDPs. While I understand that Dec-POMDPs are only suited for fully cooperative settings, many of the testbeds used in these experiments, such as StarCraft, are more commonly modeled as Dec-POMDPs. Is there a specific reason for modeling this on a POSG?

  2. As the SMAC v2 paper notes, the SMAC v1 tested in this paper actually don't require on-loop control [3]. So I'm wondering how can we show that agents are indeed deciding their preferences pip_i and roles based on the input?

7 comments

We thank the reviewer for the careful review. Before answering point by point, we would like to clarify what the paper studies. PCMA addresses an under-explored problem, multi-objective multi-agent RL, where each agent faces its own trade-off among multiple objectives while also sharing a common interest as a system. And our focus is preference coordination: how agents with different trade-offs coordinate their preferences toward this shared interest. We respond to the questions and concerns below.


Q1: Why model the problem as an MO-POSG rather than a Dec-POMDP?

We went through a careful discussion before choosing this formulation. Note that a Dec-POMDP is the special case of a POSG in which all agents share the same reward function (Hansen et al., 2004), and we choose to go beyond this case, for two reasons.

First, one shared preference defines a team-level trade-off. If all agents scalarize a common reward vector with the same preference, the resulting objective expresses one trade-off for the team. Therefore, we take the mo-POSG of Felten et al. (2024, Definition 1) as our formulation. This formulation is also practically meaningful. Take a basketball team as an example: every player wants to win, but a center and a guard weigh scoring, defending and ball protection differently. Furthermore, the mo-POSG captures a modeling case that is common but largely overlooked in practice: agents share a common interest but face different trade-offs, which also arises in multi-LLM-agent systems, drone swarms, etc.

Second, shared feedback makes role-specific preferences harder to learn. PCMA aims to let each agent learn a preference suited to its role. A shared reward vector records the team outcome on each objective. From that feedback alone, it is difficult to tell which agent should give more weight to a particular objective. In our formulation, each agent additionally receives its own vector reward and maximizes Ui(θ;pi)=Jteam(θ)+pi⊤Ji(θ)U_i(\theta;p_i)=J_{\text{team}}(\theta)+p_i^\top J_i(\theta), so each agent trades off its own objectives JiJ_i, with JteamJ_{\text{team}} capturing the team's shared interest. Hence, the problem changes from a cooperative game with a common payoff into a preference-conditioned general-sum game G(p)\mathcal{G}(p), and the question we focus on is how agents coordinate their preferences, rather than multi-agent credit assignment.


Q2: SMACv1 does not require closed-loop control. Do agents decide their preferences and roles from their input?

First, agents can decide their preferences from their observations, and this ability is learned in training. The planner maps each agent's local observation, including its unit type, health and the relative positions of allies and enemies, to its preference. For example, under our two-dimensional SMAC reward (damage dealt and damage taken), a unit that observes that it is on the front line can lower its weight on avoiding damage and hold its position. To further validate this, we evaluate trained PCMA policies with each agent's preference replaced by a constant, namely that agent's planner output averaged over all time steps of the evaluation episodes, so that each agent keeps its own preference but can no longer adjust it to the current observation. This lowers the success rate:

Success ratePredator-PreyCatchSMAC 2s3z
Learned planner (observation-dependent)0.96±0.020.94±0.030.89±0.10
Fixed per-agent preference0.84±0.060.86±0.040.80±0.14

Second, in most cases, the SMACv1 learning curves already start at high success rates. To test PCMA under greater difficulty, we added the Terran and Zerg 5 vs. 4 tasks from SMACv2 with extended partial observability. Both tasks randomize unit types and starting positions each episode. PCMA maintains the highest success rate on both maps:

SMACv2 success rateTerran FFZerg FF
PCMA (Ours)0.61±0.040.44±0.05
MAPPO0.57±0.060.34±0.05
IPPO0.55±0.050.32±0.05

W1: The planner and the actor depend only on the current observation.

We agree that agents with memory can mitigate partial observability. However, our research focus is not partial observability but the coordination among different objectives. Moreover, PCMA extends to memory directly: replacing the observation oio_i with the action-observation history hih_i, i.e., pi=ϕψ(hi)p_i=\phi_\psi(h_i) and πθ(ai∣hi,pi)\pi_\theta(a_i \mid h_i, p_i), requires no other change. We report the results of replacing the simple feed-forward policy of the actor and the planner with an RNN on SMACv2. Recurrent PCMA achieves a higher mean success rate than recurrent MAPPO and IPPO on both maps.

SMACv2 success rateTerran RNNZerg RNN
PCMA (Ours)0.69±0.050.47±0.03
MAPPO0.64±0.060.38±0.04
IPPO0.63±0.070.36±0.05

W2: Gap between the solution concept and what the experiments validate.

Section 3.2 defines the solution concept in terms of the team return at equilibrium.

First, the solution is a team-optimal equilibrium. The mo-POSG in Section 3.1 specifies what each agent observes and receives, including its own vector reward. An equilibrium only means that no agent can improve its own utility by unilaterally changing its policy; as the reviewer notes, it therefore need not be optimal for the team. As is pointed out in Section 3.2, this is exactly why the preferences must be coordinated. Different preference profiles pp induce different equilibria θ∗(p)\theta^{\ast}(p), and since the agents share a common interest, the team objective is the natural criterion for selecting among them. Therefore, we do not take an arbitrary equilibrium as the solution, but coordinate pp so that the induced equilibrium maximizes the team objective, max⁡pJteam(θ∗(p))\max_{p} J_{\text{team}}(\theta^{\ast}(p)).

Second, both our theory and our evaluation target this team objective. On the theory side, Theorem 4.2 gives the first-order team improvement ΔJteam\Delta J_{\text{team}} under a preference profile; accordingly, the planner that sets pp is trained with the team advantage. For tasks with a success criterion, we report how often the team completes the task. Since task completion contributes positively to the team reward, we expect a higher success rate to be associated with higher JteamJ_{\text{team}}. Other reward components, such as collision penalties in Predator-Prey, also affect the team return.


W3: "Average reward" as an evaluation metric.

We apologize for the imprecise terminology. The correct term is “average return,” which we compute by averaging episodic returns across agents, objectives, and evaluation episodes. We will use this term and clarify its definition in the paper.


References

  • Eric A. Hansen, Daniel S. Bernstein, and Shlomo Zilberstein. Dynamic Programming for Partially Observable Stochastic Games. AAAI, 2004.
  • Florian Felten et al. MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learning. arXiv:2407.16312, 2024.

Thanks for your response, after carefully reading the author responses, the reviewer still feels unconvinced by most of the main complaints. So, the reviewer decided to maintain the original evaluation.

Reviewer_4KJ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a cooperative multi-objective multi-agent RL (MOMARL) algorithm, where a team of agents must balance multiple objectives. The key idea is to let each agent learn their preference over a vector of individual rewards and coordinate at the team level. In the proposed method, each agent has a preference planner and a preference-conditioned actor. The training framework uses CTDE with a team critic, per-agent vector critics, and a pairwise preference-diversity regularizer. Empirically, the method is tested on MPE, MOMAland tasks, MOMAwalker, and SMAC, compared against MADDPG, IPPO, and MAPPO baselines.

Critical strengths
  • The paper tackles an important and underexplored problem in cooperative multi-objective MARL in which conflicts arise both across objectives and across agents, making a single scalar reward inadequate. The core motivation, that enforcing one shared preference vector forces all agents to compete along the same reward dimensions is a compelling insight. The preference based approach is an interesting direction in which each agent learning its own preference and coordinating at the team level. Agents can occupy distinct, complementary regions of the Pareto front instead of collapsing onto a single trade-off.

  • The empirical evaluation is thorough, spanning a diverse range of cooperative multi-agent environments --- MPE, walker locomotion, SMAC, and a practical traffic-control scenario CARLA (in appendix). PCMA is compared against MADDPG, IPPO and MAPPO baselines and it achieves competetive performance.

Critical weaknesses
  • Multi-objective: The proposed objective formulation in Subsection 3.2 is more towards individual reward decomposition rather than a genuine mult-objective MARL problem. The team objective JteamJ_{team} is common across agents and is independent of individual preferences. The multi-objective component is the agent-specific JiJ_{i}, making the formulation a shared reward plus per-agent reward decomposition rather than a genuine multi-objective MARL problem with K conflicting team-level objectives. Experimental design also reflects the same structure, i.e., team reward + vector of individual rewards. It appears that the multi-objective element has been tailored to fit the method, rather than the method being designed to address a genuine multi-objective team setting.

  • Given that the problem is cooperative, the evaluation compares only against MADDPG, IPPO, and MAPPO, excluding the standard cooperative MARL baselines like QMIX/QPLEX/COMA. Without these baseline comparisons, it is difficult to assess PCMA's advantage of using preference based coordination in cooperative MARL problems.

  • The traffic-control scenario domain is mentioned in the abstract and introduction's motivating example; however, the evaluation results are presented in the Appendix.

Questions for authors
  • How are the theoretical results on Nash equilibrium/team optimal equilibrium analyzed in the experimental results?

  • What are the team reward and individual reward vectors in the cooperative spread domain?

  • How is PPO applied to the Dirichlet planner? The Dirichlet distribution parameter αi=ϕψ(oi)\alpha_i = \phi_\psi(o_i) is unbounded (αi≥0)(\alpha_i \geq 0).

  • How is the multi-agent credit assignment problem addressed, as it is the key challenge for coordination in any cooperative MARL setting?

4 comments

We thank the reviewer for recognizing the importance of the problem and our core insight. We respond to the questions and concerns below.


Q1: How are the equilibrium results reflected in the experiments?

Because exploitability estimates are noisy in large games with DNN policies (Timbers et al., 2022), we examine two empirical implications of our equilibrium analysis. First, we use the learning curve to assess whether the agents approach an equilibrium during training. In Figure 5(a), team performance levels off as training progresses, consistent with this interpretation. Second, we use final team performance to assess whether coordinating preferences leads to a better outcome for the team. In the ablation, the team performs better with PCMA’s learned preferences than with random agent-specific preferences (RAND) or a shared mean preference (SAME).


Q2: What are the team reward and the individual reward vectors in Cooperative Spread?

The reward definitions of all tasks are listed in Table 2 (Appendix). In Cooperative Spread, two agents must cover two landmarks, L1L_1 and L2L_2.

  • Individual vector reward of agent ii: ri=(−0.01⋅d(i,L1),−0.01⋅d(i,L2))r_i=(-0.01 \cdot d(i,L_1), -0.01 \cdot d(i,L_2)), i.e., the negative distance to each landmark.
  • Team reward: +1 when the two landmarks are covered by different agents, and −1 when the agents collide; both end the episode.

Q3: How is PPO applied to the Dirichlet planner, given that its concentration parameters are unbounded?

As in Section 5.2, the planner is a stochastic policy over preferences: it outputs αi=ϕψ(oi)\alpha_i=\phi_\psi(o_i) and samples pi∼Dir(αi)p_i \sim \mathrm{Dir}(\alpha_i). PPO only needs the log-density of the sampled preference, log⁡Dir(pi∣αi)\log \mathrm{Dir}(p_i \mid \alpha_i). The unbounded range of αi\alpha_i is therefore not a problem in itself, just as for Gaussian policies in PPO, whose mean and variance are also unbounded. The PPO surrogate uses the ratio of the new and old Dirichlet densities evaluated at the sampled pip_i.


Q4: How is the multi-agent credit assignment problem addressed?

Credit assignment is indeed a key challenge in cooperative MARL, but it is not the focus of this paper, and the reason lies in the problem formulation. Credit assignment concerns how to distribute a single team reward among agents, whereas in our formulation each agent receives its own individual vector reward explicitly. Therefore, the core problem we study is how to coordinate the agents' trade-offs over these individual vector rewards, rather than how to split the team reward; for the latter, PCMA uses a standard CTDE framework, where a centralized critic provides the team advantage and per-agent vector critics provide the individual advantages. That said, preference learning also acts as an indirect form of credit assignment: through their learned preferences, agents take different roles in the team, e.g., favoring different landmarks or prey (Figure 3), as discussed in Section 6.1.

W1: Why individual vector rewards instead of K team-level objectives?

We gave careful consideration to the choice of individual vector rewards with a shared team objective.

First, one shared preference defines a team-level trade-off. If all agents scalarize the same K team objectives with one shared preference, the resulting objective expresses one trade-off for the team. From a shared reward vector alone, it is difficult to tell which agent should prioritize which objective. We model this setting as a multi-objective partially observable stochastic game (mo-POSG). In our formulation, each agent maximizes Ui(θ;pi)=Jteam(θ)+pi⊤Ji(θ)U_i(\theta;p_i)=J_{\text{team}}(\theta)+p_i^\top J_i(\theta), where JteamJ_{\text{team}} captures the shared team objective and pip_i weights the individual objectives in JiJ_i.

Second, the environment design reflects a practical coordination problem. Agents working toward a shared goal may weigh their own objectives differently depending on their roles. On a basketball team, for example, all players want to win, yet some prioritize offense while others focus on defense. Similar trade-offs arise in drone swarms and multi-LLM systems. Such settings are common but largely overlooked, and few benchmarks capture them. We therefore adapted existing tasks to represent both aspects of the problem through a shared reward for the team's goal and a vector reward for each agent's own objectives. We expect these tasks to generalize to other settings in which agents share a goal but face different trade-offs.


W2: Comparison with cooperative MARL baselines.

Since this setting is under-explored, it lacks well-established baselines, so we compare with strong cooperative MARL methods under a shared scalarization. We added QMIX and QPLEX on Cooperative Spread and SMAC, with the same code base and reward as PCMA. Under the same number of update steps, both achieve lower success rates than PCMA on both tasks, suggesting that value decomposition alone may not address the central challenge of coordinating agent preferences.

Success rateSpreadSMAC 2s3z
PCMA (Ours)0.98±0.020.89±0.10
QMIX0.06±0.040.19±0.24
QPLEX0.07±0.100.44±0.37

W3: The traffic-control results are in the appendix.

We thank the reviewer for this suggestion. We originally placed the CARLA traffic-control pilot in the appendix due to the page limit. We will revise the layout to make room for the main CARLA results and move them into the main text.


References

  • Florian Felten et al. MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learning. arXiv:2407.16312, 2024.
  • Finbarr Timbers, Nolan Bard, Edward Lockhart, Marc Lanctot, Martin Schmid, Neil Burch, Julian Schrittwieser, Thomas Hubert, and Michael Bowling. Approximate Exploitability: Learning a Best Response. IJCAI, 2022.

I thank the authors for their response. However, my concerns regarding the objective formulation and the lack of comparison with standard cooperative MARL baselines remain. As a result, I will maintain my current score.

We sincerely thank you for reading our response and for sharing your remaining concerns, and we would like to further clarify the two points below.

First, on the objective formulation. MOMARL is an under-explored problem without a standard formulation. Previous works such as MOMAland [1] consider team-level objectives, but this setting cannot represent different agents holding different preferences. Our work takes this problem as its focus and studies how to coordinate the preferences of different agents, with a shared team goal guiding this coordination, and a scalar team goal makes this guidance clear. Besides, when all agents receive the same reward vector, our formulation recovers the team-level one.

Second, on the comparison with cooperative MARL baselines. We provide a more complete baseline comparison under W1 in our response to Reviewer E2SP. Beyond MAPPO, IPPO, and MADDPG in the paper, it adds the trust-region method HAPPO and the value-based methods QMIX and QPLEX. HAPPO improves once our preference coordination is added, and the value-based methods do not carry over to this setting. Across these families, standard methods have no mechanism for resolving the conflict among agents' objectives, which is the problem our work addresses.

Thank you again for your time and consideration.

Reviewer_E2SPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies cooperative multi-objective multi-agent reinforcement learning (MOMARL), where agents share a sparse team reward and additionally receive agent-specific vector rewards over K objectives. Rather than scalarizing all agents with one shared preference vector, the authors propose PCMA: a per-agent stochastic planner maps local observations to Dirichlet parameters, a preference is sampled at each environment step, and a preference-conditioned actor is trained with PPO under CTDE, combining a centralized team critic with per-agent vector critics; a pairwise diversity bonus on the sampled preferences is added to the planner objective (Eq. 3). Theoretically, the paper gives a first-order decomposition of team improvement into team-gradient, average-alignment, and preference-diversity terms (Thm 4.2), and a local equilibrium-tracking bound for slowly varying preference profiles (Thm 4.6). Experiments on modified MPE, CrazyRL, Multiwalker, and SMAC tasks, plus a CARLA intersection scenario, report gains over MADDPG/IPPO/MAPPO and show interpretable preference specialization among agents.

Critical strengths
  1. Well-motivated problem and formulation. The conflict argument against shared-preference scalarization (the unsignalized-intersection example, L. 45-49, Fig. 1) is intuitive, and the team-optimal equilibrium objective is a sensible solution concept for cooperative MOMARL. The mo-POSG with partially aligned interests (shared sparse team reward + agent-specific vector guidance, L. 132-137) is a reasonable abstraction of many practical cooperative tasks.

  2. Simple and modular method. The planner + preference-conditioned actor design is lightweight and CTDE-compatible (the planner depends only on local observations, so execution remains decentralized), and it ports across backbones (PPO in the main suite, SAC in the CARLA study).

  3. Interpretable emergent specialization. Figs. 3-4 provide qualitative evidence that learned preferences spread agents along the return front and induce complementary roles (landmark coverage, prey/safety split, damage-dealt vs. damage-taken in SMAC). This is arguably the paper's most convincing result.

Critical weaknesses
  1. Outdated and incomplete baselines (L. 401-403). The only comparisons are MADDPG (2017), IPPO, and MAPPO with uniform scalarization. (a) MoMix is criticized for its IGM assumption and continuous-action limitation (L. 109-113) but never evaluated -- Spread, Predator-Prey, and SMAC are all discrete-action, where that limitation is irrelevant. (b) No value-decomposition (QMIX/QPLEX), trust-region (HAPPO/HATRPO), or transformer-based (MAT) baselines. (c) None of the cited preference-conditioned MORL methods (Abels et al., 2019; Yang et al., 2019; Basaklar et al., 2023) are adapted as multi-agent baselines. With only 3 seeds (L. 425), several headline gains are within noise (Safe Predator-Prey 0.96 vs 0.91; SMAC-3m tied at 0.97; SMAC-8m 0.87+-0.05 vs 0.80+-0.16; MOMAwalker std +-20; Table 1). The empirical bar is too low to support the improvement claims.

  2. The theory does not justify the diversity regularizer (Thm 4.2 vs. Eq. 3). The positive diversity term in Thm 4.2 exists only under Assumption 4.1 (L. 188-191), which directly assumes that preference deviations align with team-improvement contributions -- i.e., it assumes the coordination problem is already solved. The planner objective (L. 320), however, rewards raw pairwise diversity D_alpha regardless of alignment, and App. C.2 itself constructs a same-mean, same-diversity but misaligned preference profile that stalls learning strictly below the best shared preference (L. 1161-1163). The abstract's claim that "preference diversity yields a first-order improvement in the team objective" (L. 18-20) is therefore overstated: diversity helps only when aligned, and nothing in the algorithm enforces or even checks alignment.

  3. Thm 4.6 does not describe the proposed algorithm. (a) Tracking requires slowly varying preferences (||p_{t+1} - p_t|| <= delta, L. 251-254), but Alg. 1 (L. 329) and Alg. 2 (L. 985) resample p_i from a Dirichlet at every environment step, so the slow-variation premise fails by construction. (b) Assumption 4.4 (L. 231-235) -- contraction of simultaneous gradient ascent around a strict local NE with rate rho < 1 -- is not implied by the second-order conditions cited from Ratliff et al. (L. 216-217): simultaneous gradient play is known to fail to converge to strict local Nash equilibria in general continuous games, and the dependence of rho on eta, as well as uniformity of the attraction neighborhood along the preference path, are unaddressed. (c) Lemma 4.3's nonsingular-Jacobian requirement (L. 224) is itself restrictive in symmetric team games, where permutation symmetry yields non-isolated equilibria. As written, Sec. 4.2 is a conditional exercise with unclear bearing on PCMA.

  4. The evaluation does not test the paper's own claims. (a) No multi-objective metrics (hypervolume, expected utility, Pareto coverage) although trade-off coordination is the stated motivation. (b) The central practical argument against retrain-per-preference methods (L. 107-109) is never evaluated: there is no test of conditioning the trained actors on externally specified preferences at execution time. (c) Ablations (lambda_1, lambda_2, RAND/SAME) are conducted only on Cooperative Spread (L. 430, Fig. 6). (d) All environments use author-designed reward decompositions and success criteria (Table 2), and SMAC uses difficulty 5 instead of the standard 7 (L. 861), limiting comparability with the literature. (e) The CARLA study (Table 3) does not use the proposed Dirichlet planner at all (L. 914-917) -- preferences are hand-assigned per approach direction with a learned shift (L. 923-924) -- so it validates preference conditioning rather than PCMA. (f) lambda_1/lambda_2 are tuned per environment (Table 5) without a matched tuning budget for baselines, and Table 1's "average reward" is never defined (L. 349, L. 434: how objectives are aggregated, and whether it is comparable across methods).

Questions for authors
  1. Can you log p̃_i^T b̃_i (equivalently kappa in Assumption 4.1) during training to test whether preference-improvement alignment actually holds? Is there any environment in which lambda_1 > 0 with misaligned diversity actively hurts performance, as App. C.2 (L. 1161-1163) would predict?

  2. Why regularize raw pairwise diversity Dp (L. 316-318) instead of an alignment-aware variant? How does such a variant compare empirically?

  3. Can the trained actors be conditioned on user-specified preferences at execution time (zero-shot trade-off control)? Please evaluate this -- it is the central claimed advantage over retraining per preference vector (L. 107-109), and it is currently untested.

  4. Alg. 2 (L. 1004-1011) omits the planner's importance ratio over Dirichlet samples, and the same team advantage A_team updates both the actor and the planner. How is preference-level credit assigned, and did you observe interference between the two updates?

  5. Why is SMAC difficulty set to 5 rather than the standard 7 (L. 861), and how do all methods perform on the original, unmodified rewards? How exactly is Table 1's "average reward" computed for each method (L. 349)?

5 comments

We thank the reviewer for the detailed review, and for finding the formulation a reasonable abstraction of practical cooperative tasks, the method simple and modular, and the emergent specialization interpretable. We respond to the questions and concerns below.


Q1: Does preference-improvement alignment hold in practice, and can misaligned diversity hurt?

Obtaining reliable estimates of p~i⊤b~i\tilde p_i^\top\tilde b_i or κ\kappa is difficult, because each entry of bib_i is an inner product of the policy gradients of the team objective and of one individual objective, whose estimates are very noisy in deep MARL. At evaluation time, we permuted the learned preferences across agents while preserving pˉ\bar p and DpD_p. Mean success fell on all three tasks, consistent with the alignment hypothesis.

Success ratePredator-PreyCatchSMAC 2s3z
Learned preferences0.96±0.020.94±0.030.89±0.10
Permuted across agents0.81±0.080.87±0.040.82±0.17

Theorem 4.2 specifies when diversity contributes to team improvement. Its contribution to the first-order decomposition is positive under the alignment condition. PCMA uses diversity as a regularizer rather than as an objective in itself. Intuitively, the regularizer encourages agents to explore policies under different preferences, which helps them avoid conflicts over shared resources that arise when all agents follow the same preference (Figure 1). The Predator-Prey ablation in W4(c) shows that mean success at λ1=0.1\lambda_1=0.1 was lower than at λ1=0\lambda_1=0. Whether misalignment caused this drop remains open. We will revise this claim to state the alignment condition explicitly.


Q2: Why regularize raw diversity instead of an alignment-aware variant?

Estimating bib_i at every training update would require K+1K+1 additional policy-gradient estimates per agent and the corresponding inner products. These estimates are noisy, so an alignment-aware term would be both costly to compute and unreliable as a training signal. It is therefore poorly suited to regularizing exploration. We use raw diversity DpD_p with a small weight because it provides a cheap, stable exploration signal. The team advantage guides alignment when training the planner. We did not implement an alignment-aware variant, and we see a low-variance estimator of bib_i as an interesting direction for future work.


Q3: Can the trained actors be conditioned on user-specified preferences at execution time?

We acknowledge the reviewer's interesting question about conditioning on user-specified preferences. First, the limitation we identify in MORL concerns the cost of an exhaustive preference sweep in a multi-agent setting. Even if each of nn agents has only kk candidate preference vectors, such a sweep must cover knk^n joint combinations, which quickly becomes impractical as the team grows. Second, to avoid this sweep, PCMA lets each agent learn to select its own preference from its current observation. Training the planner on the shared team objective encourages agents to coordinate these choices at execution time. We leave zero-shot adaptation to user-specified preferences for future work.


Q4: The planner's importance ratio, and credit between the planner and the actor.

Algorithm 2 omits the planner's importance ratio for brevity. As in Section 5.2, the planner is a stochastic policy over preferences: it outputs αi=ϕψ(oi)\alpha_i=\phi_\psi(o_i) and samples pi∼Dir(αi)p_i \sim \mathrm{Dir}(\alpha_i). Its PPO update uses the ratio ρi\rho_i between the new and old Dirichlet densities of the sampled pip_i, clipped to [1−ϵ,1+ϵ][1-\epsilon,1+\epsilon].

As for credit and interference, unlike for the finer-grained actions, we do not perform explicit credit assignment at the preference level; instead, the centralized team critic evaluates the overall outcome, and its team advantage updates the planner. The actor and the planner are therefore co-optimized toward the same team objective: the actor learns to act well under the current preferences, and the planner learns preferences under which the team performs well. We did not observe instability from the coupled updates in practice.

Q5: SMAC difficulty and the "average reward".

The original SMAC experiments used difficulty 5. All SMAC results in this response use the standard difficulty 7. This harder setting requires more training steps to converge and yields the same qualitative conclusions. On both SMAC maps, MAPPO achieved a higher mean success rate with the vector reward than with the original scalar reward. We apologize for the imprecise terminology in Table 1. The correct term is “average return,” which we compute for every method by averaging episodic returns across agents, objectives, and evaluation episodes. We will use this term and clarify its definition in the paper.

Success rateSMAC 3mSMAC 2s3z
PCMA (Ours), vector reward0.96±0.040.89±0.10
MAPPO, vector reward0.91±0.030.83±0.06
MAPPO, original reward0.80±0.040.72±0.05

W1: Outdated and incomplete baselines.

We agree that more seeds can provide a more precise estimate of mean performance. The original comparisons used three random seeds, and all newly added experiments use five. We also expanded the baseline comparisons as described below.

(a) MoMix. The key difference is the problem setting: MoMix assumes one team vector reward and one preference shared by all agents, which is system-level MORL, whereas in our setting each agent receives its own vector reward. Hence, MoMix does not apply directly.

(b) More baselines. For value decomposition methods, we added QMIX and QPLEX. Since they require discrete actions, we applied them to Spread, Predator-Prey and SMAC, where both achieved lower mean success than PCMA, suggesting that value decomposition alone may not address the central challenge of coordinating agent preferences. We evaluated HAPPO both with and without our preference coordination mechanism. PCMA-HAPPO achieved a higher mean success rate than HAPPO on all four tasks in the table.

(c) Preference-conditioned MORL. We want to note that preference conditioning is not the focus of this paper. With MM candidate preferences per agent and NN agents, an exhaustive search would evaluate MNM^N joint preference profiles. Therefore, we adapted the conditioned network of Abels et al. (2019) as PC-MAPPO, in which all agents in the team share an identical preference w∼Dir(1)w\sim\mathrm{Dir}(\mathbf{1}), sampled per episode during training. PC-MAPPO does not close the gap to PCMA, suggesting that coordinating agent-specific preferences may contribute to the gain.

Success rateSpreadCatchPredator-PreySMAC 2s3z
PCMA (Ours)0.98±0.020.94±0.030.96±0.020.89±0.10
PCMA-HAPPO (Ours)0.99±0.010.94±0.020.97±0.020.97±0.02
HAPPO0.19±0.010.90±0.030.85±0.050.83±0.06
PC-MAPPO0.11±0.100.82±0.020.66±0.050.84±0.06
QMIX0.06±0.04–0.12±0.150.19±0.24
QPLEX0.07±0.10–0.14±0.070.44±0.37

W2: The theory does not justify the diversity regularizer.

Please see Q1 and Q2.

W3: Theorem 4.6 and the algorithm.

We admit that our analysis relies on several assumptions, due to the complexity of learning dynamics in deep multi-agent systems. Below, we explain how each assumption enters the theoretical analysis in Section 4.2.

(a) Slow variation of the preference profile. In Theorem 4.6, tt indexes policy updates. The planner is fixed during each rollout, and PPO clipping discourages large changes to its preference distribution at a given observation between updates. This motivates studying a slowly changing preference profile. Individual Dirichlet draws can still differ substantially even at the same observation, so slow variation of the profile remains an assumption of the analysis.

(b) Assumption 4.4. The second-order conditions enter Lemma 4.3. Theorem 4.6 separately assumes local contraction of the policy update. The shared team return motivates studying this regime because it gives each agent an incentive to improve the same objective. Whether the joint update contracts also depends on how the individual objectives interact near the equilibrium.

(c) Lemma 4.3. Lemma 4.3 assumes a nonsingular joint-gradient Jacobian at the equilibrium under study (Ratliff et al., 2014). Permuting a finite set of agents produces only finitely many relabelings of a given equilibrium. Permutation symmetry is therefore compatible with a locally isolated equilibrium, although nonsingularity still requires a separate assumption.


W4: Evaluation.

(a) Multi-objective metrics. Standard multi-objective metrics, such as hypervolume, expected utility and Pareto coverage, measure how well a set of policies covers a Pareto front in one shared return space, whereas in our setting each agent has its own reward vector, so there is no single return space in which a team-level Pareto front is well defined. We also deliberately do not define K team-level objectives, because we want to measure the trade-offs each agent faces in its own return space. Instead, the team shares a scalar objective JteamJ_{\text{team}}, such as winning the game, and the individual objectives shape how each agent contributes to it. In tasks with a success criterion, completing the task contributes positively to the team reward, so we expect higher success rates to be associated with higher JteamJ_{\text{team}}. We report success rate as a measure of task completion and use the return fronts in Figure 3 to show per-agent trade-offs.

(b) Zero-shot preference control. Please see Q3.

(c) Ablations beyond Spread. We varied λ1 and λ2 on Predator-Prey and λ2 on SMAC. RAND uses random agent-specific preferences without the planner, and SAME makes all agents share the mean preference.

λ100.020.050.1
Predator-Prey success rate0.91±0.020.93±0.010.96±0.020.88±0.06
Predator-PreySuccess rate
PCMA (λ2 = 0.2)0.96±0.02
λ2 = 00.86±0.07
λ2 = 0.10.93±0.03
λ2 = 0.40.59±0.08
RAND0.93±0.01
SAME0.91±0.01
PCMA on SMAC3m8m2s3z
λ2 = 0.10.96±0.040.88±0.030.89±0.10
λ2 = 00.88±0.010.84±0.020.71±0.10

Including the individual advantage at its default weight improves mean success on Predator-Prey and all three SMAC maps. The largest gain is on 2s3z.

(d) SMAC difficulty and comparability. Please see Q5.

(e) CARLA. The CARLA pilot adapts PCMA to traffic control. Each approach direction has a prior preference, and the planner learns an observation-dependent shift from that prior. We will describe this adaptation explicitly in the paper.

(f) Tuning budget and "average reward". PCMA and MAPPO use the same PPO hyperparameters (Table 4). We set the PCMA-specific coefficients including λ1 and λ2 (Table 5). For "average reward", please see Q5.


References

  • Lillian J. Ratliff, Samuel A. Burden, and S. Shankar Sastry. On the Characterization of Local Nash Equilibria in Continuous Games. arXiv:1411.2168, 2014.
  • Axel Abels, Diederik M. Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic Weights in Multi-Objective Deep Reinforcement Learning. ICML, 2019.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.