acceptodds
Withdrawn from ICLR 2027

Is Per-Agent Policy Composition Safe? Rethinking Successor-Feature Transfer in Cooperative Multi-Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many reinforcement learning systems, from fleet management to traffic signal control, must serve an objective that changes dynamically after deployment, and retraining a policy for each new objective is prohibitively expensive. For a single agent, this problem is well understood: successor features with generalized policy improvement, together with their universal extension, recombine a library of learned policies into a policy for any new objective, with a guarantee that the result is never worse than any policy in the library. However, multi-agent transfer has received far less attention, and the common practice of letting each agent recombine its own library independently inherits the recipe but not the guarantee. We prove that this independent composition can produce joint behavior strictly worse than every policy in the library, because recombining teammates changes the environment each agent faces and invalidates the values it relies on, a failure with no single-agent counterpart. We further show that the only unconditionally safe fixed rule is synchronized composition, which moves the whole team to one jointly trained policy but cannot serve objectives that assign different goals to different agents. To attain safety and flexibility at once, we propose multi-agent universal successor feature approximators (MA-USFA), a hierarchical method with two layers: a lower layer of universal successor feature approximators that predicts each agent's successor features while conditioned on its teammates' objectives, and an upper composer that selects, across agents, which library entry each agent should follow and supplies the cross-agent correction a per-agent value cannot represent. Trained once over the distribution of objectives, it is applied at deployment with no per-task adaptation. On a controlled grid world and a real-world city-scale traffic signal control problem, MA-USFA matches or exceeds every fixed composition rule and recovers the performance of policies retrained from scratch. The code of this paper is provided at anonymous repository https://anonymous.4open.science/r/MA-USFA-1566 .

Primary area:
reinforcement learning
Keywords:
Transfer Learning, Multi-Agent Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_go9tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether successor-feature generalized policy improvement (GPI) is still safe when each agent in a cooperative team composes its policy on its own. It gives a counterexample to the best-library guarantee, analyzes synchronized policy switching, and proposes sufficient conditions for independent composition. It then introduces MA-USFA: a successor-feature model conditioned on teammate context, together with a learned composer. The intended protocol trains both layers once and then freezes them for new objectives. The experiments use a controlled collision gridworld and a simulated traffic network with 196 intersections. They report gains over fixed composition rules and performance close to task-specific retraining.

Critical strengths

S1. A relevant failure mode, with an explicit example: In Appendix C.3, independent composition returns 268, while the best library policy returns 287 and joint GPI reaches 321. I checked these numbers exactly. The example shows well how values computed against one teammate policy can mislead agents that recompose independently. This narrower result, falling below the best library policy, is still meaningful despite my concerns below.

S2. A controlled test and a large application: Varying the collision coupling gives a setting that is easy to interpret, and the 196-intersection traffic experiment makes the question practically relevant. I also like that the paper reports throughput, waiting time, queue length and speed, not only task return. This helps show the practical effect of coordination.

Critical weaknesses

W1. The supplied code trains a composer for each target task, not once: The abstract, §4.3 and Algorithms 1–2 promise one composer, trained over objectives and then frozen for new ones. But in the supplied code, both the gridworld and the traffic experiments train a new composer on each target task. This shows per-task adaptation, not the advertised transfer. The code also differs from Eq. (11). The successor features take no policy-encoding or teammate-context input. The gridworld composer also ignores its task input: when I changed it, the actions stayed exactly the same at all 625 two-agent position pairs. So the frozen, context-conditioned selector is not yet validated. Finally, the gridworld baselines are at a disadvantage. Their library ignores the collision coupling, and their features leave out the collision penalty (§5.2, Appendix E.1), while the composer learns with the full reward. So the composer's gain is mixed up with the baselines' model errors.

W2. Proposition 4 is false as stated, and the claim that the learned composer never gets worse is not justified: Proposition 4 says that if the joint library envelope is supermodular (with consistent tie-breaking), then independent composition is no worse than synchronized composition. I found a counterexample with a small change to the paper's own Table 2. I changed one feature of agent 2 (second stage, state 0, action 0) from (7,7)(7,7) to (8,7)(8,7), kept everything else, and used a discount of 0.95. The rewards are still separable, and the envelope is still supermodular. But independent composition now gets 267.6, while synchronized composition gets 279. There are no ties, and I checked these numbers by exact enumeration. I also think the proof in Appendix C.4 has two steps that do not follow: the sublattice argument does not make the per-agent scores line up, and the claimed equality between the best library value and the synchronized value does not follow from Proposition 1. Separately, §4.1, §4.3 and Appendix G claim that the learned composer never reduces performance on any objective, because it starts from the independent rule and is then trained with TD updates. But zero initialization only fixes the starting policy. Minimizing each agent's TD loss does not guarantee that the team return improves monotonically on every objective. The paper gives no such guarantee, and no mechanism that enforces it. These issues affect the paper's central safety claim, not just how it is written.

Questions for authors

Q1. Which implementation did you use? Is there another implementation or checkpoint that implements Eq. (11) and a single composer trained over objectives, then frozen before testing on held-out tasks? If so, please point to it, and explain how the reported results relate to the per-target training I described above.

Q2. Which explicit assumption rules out the modified Table 2 example above, and what corrected guarantee holds for Eq. (5)? What shows non-regression for the composer that is actually trained with TD, as opposed to its initialization?

Q3. Could you evaluate a single frozen composer on held-out objectives, against fixed rules whose successor features model the full reward under the same transition kernel, and report uncertainty across training seeds? This would separate the proposed transfer mechanism from task-specific adaptation and from model misspecification.

Reviewer_PEK4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies policy composition based on successor-feature (SF) in cooperative MARL. It shows independent policy composition can fail, analyzes the conditions for safe composition. The paper proposes MA-USFA, a hierarchical framework that combines SF with a learned composer.

Critical strengths

The problem is well motivated. The paper identifies a limitation of extending single-agent GPI to multi-agent settings. The counterexample and traffic experiments demonstrate the theoretical and practical relevance of the problem.

Critical weaknesses
  1. Incorrect Theoretical Guarantees

    Propositions 4 and 6 seem to be incorrect under the stated assumptions. Supermodularity only guarantees that the set of joint maximizers forms a sublattice. This does not mean that independently greedy decisions by each agent can achieve the joint optimum.

    I would like to ask the authors to check the following counterexample.

    Consider a two-agent MDP with discount factor γ=0.9\gamma = 0.9. Agent 2 has independent dynamics and always receives zero reward. Agent 1 has three states, s0s_0, xx, and yy. At s0s_0, action 0 gives reward 8 and terminates, while action 1 moves to xx. At xx, action 1 gives reward 3 and moves to yy. At yy, action 1 gives reward 10 and terminates. All other actions give zero reward and terminate. The policy library contains π0=(1,1,0)\pi^0 = (1, 1, 0) and π1=(1,0,1)\pi^1 = (1, 0, 1), where each entry indicates the action at (s0,x,y)(s_0, x, y).

    This counterexample seems to satisfy all the assumptions in Propositions 4 and 6. However, we have

    Vind(s0)=8<10.8=Vsync(s0)V_{\mathrm{ind}}(s_0) = 8 < 10.8 = V_{\mathrm{sync}}(s_0),

    which contradicts the claimed performance guarantee. The authors should explain which assumption is violated in this example, or revise the propositions if the counterexample is valid.

    In addition, Proposition 6 claims that SF are independent of the library policy under factored dynamics. However, SFs still depend on the future actions selected by the policy, even when the dynamics are factored. Also, Appendix C.4 assumes Vsync=max⁡kVπkV_{\mathrm{sync}} = \max_k V^{\pi^k}, but this is not generally true because switching between library policies at different states can achieve a higher value than any individual library policy.


  1. Significant Implementation Mismatch

    The released implementation appears to train the composer separately for each target task before evaluation, while only the successor-feature backbone is frozen. Therefore, the reported performance seems to rely on task-specific adaptation, rather than zero-shot transfer using a single pretrained composer without additional gradient updates. This is inconsistent with the claimed "zero per-task adaptation," which is mentioned three times in the main text.

    Moreover, the implementation does not seem to explicitly include teammate-conditioned successor features and library-entry selection as described in the paper. Therefore, the experiments do not clearly validate the proposed MA-USFA framework.


Minor Weaknesses

  • The related work section is limited and does not discuss several relevant studies, especially on multi-agent successor features and coordination [1, 2], as well as multi-objective MARL. Discussing these works would help clarify the novelty of the paper.

  • Although the paper reports the number of training episodes, it does not provide enough details about computational resources, hyperparameter tuning budgets, and model selection criteria. In particular, the paper reports the best result among different composer architectures, but it is unclear how the best model was selected. This raises concerns about reproducibility and fair comparison.

[1] Tarun Gupta, Akshat Kumar, and Praveen Paruchuri. (2019). Successor Features Based Multi-Agent RL for Event-Based Decentralized MDPs. AAAI.

[2] Seung Hyun Kim, Neale Van Stralen, Girish Chowdhary, and Huy T. Tran. (2022). Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning. AAMAS.

Questions for authors
  1. Please provide responses to Weaknesses 1 and 2.

  2. Can the authors demonstrate zero-shot performance using a single frozen composer on unseen objectives?

  3. How does the released implementation correspond to the proposed teammate-conditioned USFA and policy-library composer?

  4. What formally guarantees that the learned composer never performs worse than the initial transfer policy?

  5. How can MA-USFA outperform synchronized composition when the library contains only one policy (Table 6)?

  6. Can the authors provide stronger comparisons with existing multi-agent SF methods and baselines under matched training budgets?

Reviewer_Q8j4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the safety of successor-feature-based policy composition in cooperative MARL. It analyzes synchronized and independent composition and proposes MA-USFA, combining teammate-context-conditioned successor features with a learned composer. Experiments use a controlled grid world and a 196-agent traffic-control network.

Critical strengths
  1. It is interesting to explore the potential failure of independently composing policies.

  2. The controlled coupling experiments and large-scale traffic application provide useful complementary settings.

Critical weaknesses
  1. The overall presentation obscures important distinctions. Dense sentences and repeated metaphors such as “pricing” should be replaced with clearer RL terminology. Proven results, architectural motivations, and empirical observations should be explicitly separated. Key traffic results should appear in the main paper.

  2. Proposition 4 does not hold as stated. Consider a one-step, two-agent game with unit weights, exact successor features, and a single library policy choosing (0,0). Let the individual reward pairs for joint actions (0,0), (0,1), (1,0), and (1,1) be (0,2), (0,1), (1,0), and (1,2), respectively. The joint value is supermodular (2+32+3\geq1+11+1), but Eq. (5) uniquely selects (1,0), yielding a team reward of 1, whereas synchronized composition yields 2. Thus, supermodularity alone is insufficient for the claimed guarantee.

  3. The learned method’s safety guarantee is unsupported. Sections 4.1–4.3 infer improvement on every objective from initialization at the independent policy and freezing the lower layer. However, the TD updates in Algorithm 1 do not guarantee non-degradation for every task and state. Moreover, improving over independent composition does not imply matching synchronized composition, since the former can already be unsafe.

  4. The empirical evidence needs stronger controls. Held-out objective splits and independent training seeds are insufficiently documented. Traffic results lack uncertainty estimates and comparisons with retraining or strong task-conditioned baselines. Ablations should isolate teammate conditioning and the composer.

  5. The positioning against prior work is incomplete. Multi-objective reinforcement learning algorithms (such as PGMORL, GPI-PD) are very related to what the authors studied, which are missing in the whole paper.

Questions for authors

Please refer to Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.