- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ConventionPlay groups policies by their ability to cooperate, then trains adaptive partners restricted to subsets of these conventions. Training with both fixed and adaptive partners teaches the final agent when to guide or accommodate unfamiliar teammates. Results across repeated games, spatial coordination and Task-Relay show gains against adaptive test partners.
Critical strengths
-
The method trains partners that can use only some conventions. This gives the agent practice with teammates that cannot fully adapt. The clustering and rebalancing experiments test convention discovery with different numbers of conventions and partner policies.
-
ConventionPlay substantially improves Task-Relay performance when production paths offer different rewards. The gains hold with both learned and hand-coded adaptive partners.
Critical weaknesses
-
The comparisons do not show whether grouping policies by convention improves training. The missing comparison is with partners built from policy subsets that are not grouped by convention. SyKLRBR allows unrestricted adaptation and does not provide this comparison.
Adding adaptive partners also increases how often the agent trains with them. The gains therefore cannot be attributed to partner diversity alone. A fair comparison must hold these sampling proportions fixed and match the total cost of all training stages and cross-play collection.
-
The paper does not test whether the agent can work with partners that try to lead it toward a different convention. The learned test partners are built from convention subsets, as in training. The Task-Relay heuristics let the agent choose the path, even when they vary when or how they respond.
-
The paper does not test the trained agent when convention discovery misses or merges groups, or when the groups have overlapping compatibility patterns. Varying the number of conventions tests only the rebalancing step, not the complete method.
-
The evaluation uses only a repeated matrix game, Point Mass Rendezvous and Task-Relay. It lacks tests on richer coordination benchmarks such as Overcooked, used by FCP and CoMeDi. The experiments do not show whether the gains hold beyond these three settings.
-
Deciding whether to lead or follow a partner is not new. Nikolaidis et al., HRI 2016 already make this choice based on how much the partner can adapt. The new contribution is how ConventionPlay trains partners with limited sets of conventions.
Questions for authors
- Do gains persist over unstructured policy subsets with partner count, sampling mixture and full training budget matched? Which conventions can those partners support?
- Do gains persist against partners that actively steer the team toward another convention?
- How does final-agent performance change when discovery misses or merges conventions, or their compatibility patterns overlap?
- Do the gains persist on an established benchmark such as Overcooked?
- What does ConventionPlay add beyond prior work that chooses whether to lead or follow based on how much a partner can adapt?
We appreciate the reviewer's assessment and have extended our experimental method with the following:
The results and discussion of these experiments are found in Appendix E of our updated submission. Our primary claims hold with the updated experiment. In the fully differentiated case, where we expect the largest gains, ConventionPlay improves on the best baseline against PAHT(2) in PMR and Task-Relay and is competitive in the Matrix Game, showing strong performance against a wide range of adaptive partners. ConventionPlay (Rand) also performs strongly, suggesting the primary value comes not from convention modeling but from training with limited-capability partners.
How much an error in convention partitioning affects final performance depends on the specific task, the capabilities of the partner, and the relative value of the conventions. In general, if a convention is missed or merged with another, the K1 population may not accurately cover the range of capability-limited partners, which can leave the K2 agent better at probing for some conventions than others.
We present new experimental results in Figure 14 of the updated submission, which measures the efficiency of the ad-hoc agent when interacting with the Psampled(1) population across a range of estimated conventions q. We see that underestimating q is most harmful, with a 30 point efficiency drop compared to the peak, which lies near or slightly above the true q. This suggests that under-segmenting conventions, so that convention sets are merged, is more detrimental than over-segmenting. A limitation of our ConventionPlay implementation (Algorithm 1) is that it requires assigning each policy to a single convention. In future work, we would like to explore leveraging probabilistic convention representation (e.g. as done by our matrix factorization method) in tasks where joint strategies are more likely to overlap.
Preliminary experimentation in Overcooked revealed little differentiation in rewards across the various conventions that form during population training. We do not expect ConventionPlay to have an advantage in such scenarios, similar to the reported results in the non-differentiated variants of the Matrix Game, PMR, and Task Relay.
We notice two significant contributions beyond Nikolaidis et al.’s work on partner adaptability: First, our method of defining and discovering conventions is based on the compatibility of policies, and therefore we can discover conventions in a range of environments purely using a population based training method. Second, BAM represents “adaptability” explicitly as a scalar and has a pre-programmed preference for the “best” convention. In our setting a partner can support a subset of many conventions (5-8 in our tasks), and the best one is not explicitly defined in advance. Our recurrent policies implicitly build a belief over which conventions a partner supports. We reason that this latent representation is more flexible than a scalar, as it allows an RL-optimized policy to switch from a probing strategy to a committing strategy that maximizes return over the episode. We will include a discussion of this work in revisions of this text.