- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether successor-feature generalized policy improvement (GPI) is still safe when each agent in a cooperative team composes its policy on its own. It gives a counterexample to the best-library guarantee, analyzes synchronized policy switching, and proposes sufficient conditions for independent composition. It then introduces MA-USFA: a successor-feature model conditioned on teammate context, together with a learned composer. The intended protocol trains both layers once and then freezes them for new objectives. The experiments use a controlled collision gridworld and a simulated traffic network with 196 intersections. They report gains over fixed composition rules and performance close to task-specific retraining.
Critical strengths
S1. A relevant failure mode, with an explicit example: In Appendix C.3, independent composition returns 268, while the best library policy returns 287 and joint GPI reaches 321. I checked these numbers exactly. The example shows well how values computed against one teammate policy can mislead agents that recompose independently. This narrower result, falling below the best library policy, is still meaningful despite my concerns below.
S2. A controlled test and a large application: Varying the collision coupling gives a setting that is easy to interpret, and the 196-intersection traffic experiment makes the question practically relevant. I also like that the paper reports throughput, waiting time, queue length and speed, not only task return. This helps show the practical effect of coordination.
Critical weaknesses
W1. The supplied code trains a composer for each target task, not once: The abstract, §4.3 and Algorithms 1–2 promise one composer, trained over objectives and then frozen for new ones. But in the supplied code, both the gridworld and the traffic experiments train a new composer on each target task. This shows per-task adaptation, not the advertised transfer. The code also differs from Eq. (11). The successor features take no policy-encoding or teammate-context input. The gridworld composer also ignores its task input: when I changed it, the actions stayed exactly the same at all 625 two-agent position pairs. So the frozen, context-conditioned selector is not yet validated. Finally, the gridworld baselines are at a disadvantage. Their library ignores the collision coupling, and their features leave out the collision penalty (§5.2, Appendix E.1), while the composer learns with the full reward. So the composer's gain is mixed up with the baselines' model errors.
W2. Proposition 4 is false as stated, and the claim that the learned composer never gets worse is not justified: Proposition 4 says that if the joint library envelope is supermodular (with consistent tie-breaking), then independent composition is no worse than synchronized composition. I found a counterexample with a small change to the paper's own Table 2. I changed one feature of agent 2 (second stage, state 0, action 0) from to , kept everything else, and used a discount of 0.95. The rewards are still separable, and the envelope is still supermodular. But independent composition now gets 267.6, while synchronized composition gets 279. There are no ties, and I checked these numbers by exact enumeration. I also think the proof in Appendix C.4 has two steps that do not follow: the sublattice argument does not make the per-agent scores line up, and the claimed equality between the best library value and the synchronized value does not follow from Proposition 1. Separately, §4.1, §4.3 and Appendix G claim that the learned composer never reduces performance on any objective, because it starts from the independent rule and is then trained with TD updates. But zero initialization only fixes the starting policy. Minimizing each agent's TD loss does not guarantee that the team return improves monotonically on every objective. The paper gives no such guarantee, and no mechanism that enforces it. These issues affect the paper's central safety claim, not just how it is written.
Questions for authors
Q1. Which implementation did you use? Is there another implementation or checkpoint that implements Eq. (11) and a single composer trained over objectives, then frozen before testing on held-out tasks? If so, please point to it, and explain how the reported results relate to the per-target training I described above.
Q2. Which explicit assumption rules out the modified Table 2 example above, and what corrected guarantee holds for Eq. (5)? What shows non-regression for the composer that is actually trained with TD, as opposed to its initialization?
Q3. Could you evaluate a single frozen composer on held-out objectives, against fixed rules whose successor features model the full reward under the same transition kernel, and report uncertainty across training seeds? This would separate the proposed transfer mechanism from task-specific adaptation and from model misspecification.