acceptodds
Under review as a conference paper at ICLR 2027

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Abstract

Ad-hoc collaboration often requires agents to identify and adhere to some shared convention within a cooperative task. Existing work on reinforcement learning (RL) for ad-hoc collaboration focuses on training agents that adapt to the conventions established by their partners. These methods fail to consider the possibility that while some partners might follow only a single fixed convention, others may themselves be capable of adapting to multiple conventions. Here we present ConventionPlay, an RL-based approach that teaches agents to discover their partner's optimal convention by training against a learned population of partners that exhibit different degrees of adaptability across conventions. Some of these partners follow a single, fixed convention, while others are able to adapt to a subset of the possible conventions for the task. The existence of partners that support a limited subset of conventions forces agents trained against this population to actively probe their partner's capabilities, and steer their partner towards the most effective joint strategy that they are capable of following. Our experimental results demonstrate that agents trained via ConventionPlay achieve superior performance to existing ad-hoc collaboration methods against test populations of partners that are compatible with multiple conventions.

Primary area:
reinforcement learning
Keywords:
Ad-hoc teamwork, ad-hoc collaboration, multi-agent reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_fkCMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ConventionPlay groups policies by their ability to cooperate, then trains adaptive partners restricted to subsets of these conventions. Training with both fixed and adaptive partners teaches the final agent when to guide or accommodate unfamiliar teammates. Results across repeated games, spatial coordination and Task-Relay show gains against adaptive test partners.

Critical strengths
  • The method trains partners that can use only some conventions. This gives the agent practice with teammates that cannot fully adapt. The clustering and rebalancing experiments test convention discovery with different numbers of conventions and partner policies.

  • ConventionPlay substantially improves Task-Relay performance when production paths offer different rewards. The gains hold with both learned and hand-coded adaptive partners.

Critical weaknesses
  • The comparisons do not show whether grouping policies by convention improves training. The missing comparison is with partners built from policy subsets that are not grouped by convention. SyKLRBR allows unrestricted adaptation and does not provide this comparison.

    Adding adaptive partners also increases how often the agent trains with them. The gains therefore cannot be attributed to partner diversity alone. A fair comparison must hold these sampling proportions fixed and match the total cost of all training stages and cross-play collection.

  • The paper does not test whether the agent can work with partners that try to lead it toward a different convention. The learned test partners are built from convention subsets, as in training. The Task-Relay heuristics let the agent choose the path, even when they vary when or how they respond.

  • The paper does not test the trained agent when convention discovery misses or merges groups, or when the groups have overlapping compatibility patterns. Varying the number of conventions tests only the rebalancing step, not the complete method.

  • The evaluation uses only a repeated matrix game, Point Mass Rendezvous and Task-Relay. It lacks tests on richer coordination benchmarks such as Overcooked, used by FCP and CoMeDi. The experiments do not show whether the gains hold beyond these three settings.

  • Deciding whether to lead or follow a partner is not new. Nikolaidis et al., HRI 2016 already make this choice based on how much the partner can adapt. The new contribution is how ConventionPlay trains partners with limited sets of conventions.

Questions for authors
  1. Do gains persist over unstructured policy subsets with partner count, sampling mixture and full training budget matched? Which conventions can those partners support?
  2. Do gains persist against partners that actively steer the team toward another convention?
  3. How does final-agent performance change when discovery misses or merges conventions, or their compatibility patterns overlap?
  4. Do the gains persist on an established benchmark such as Overcooked?
  5. What does ConventionPlay add beyond prior work that chooses whether to lead or follow based on how much a partner can adapt?
1 comment

Do gains persist over unstructured policy subsets with partner count, sampling mixture and full training budget matched? Which conventions can those partners support? Do gains persist against partners that actively steer the team toward another convention?

We appreciate the reviewer's assessment and have extended our experimental method with the following:

  • An ablation where ConventionPlay's K1K_1 subsets are sampled randomly instead of by convention, ConventionPlay (Rand).
  • Modified end-to-end training budgets such that all ad-hoc methods have a total environment-frame budget at least that of ConventionPlay (Table 14).
  • A cross-play evaluation of the ego-policies from each ad-hoc method paired with one another, PAHT(2)\mathcal{P}^{(2)}_{\text{AHT}}, to test performance against partners whose strategies may adapt or steer the team towards different conventions.

The results and discussion of these experiments are found in Appendix E of our updated submission. Our primary claims hold with the updated experiment. In the fully differentiated case, where we expect the largest gains, ConventionPlay improves on the best baseline against PAHT(2)\mathcal{P}^{(2)}_{\text{AHT}} in PMR and Task-Relay and is competitive in the Matrix Game, showing strong performance against a wide range of adaptive partners. ConventionPlay (Rand) also performs strongly, suggesting the primary value comes not from convention modeling but from training with limited-capability partners.

How does final-agent performance change when discovery misses or merges conventions, or their compatibility patterns overlap?

How much an error in convention partitioning affects final performance depends on the specific task, the capabilities of the partner, and the relative value of the conventions. In general, if a convention is missed or merged with another, the K1K_1 population may not accurately cover the range of capability-limited partners, which can leave the K2K_2 agent better at probing for some conventions than others.

We present new experimental results in Figure 14 of the updated submission, which measures the efficiency of the ad-hoc agent when interacting with the Psampled(1)\mathcal{P}^{(1)}_{\text{sampled}} population across a range of estimated conventions qq. We see that underestimating qq is most harmful, with a 30 point efficiency drop compared to the peak, which lies near or slightly above the true qq. This suggests that under-segmenting conventions, so that convention sets are merged, is more detrimental than over-segmenting. A limitation of our ConventionPlay implementation (Algorithm 1) is that it requires assigning each policy to a single convention. In future work, we would like to explore leveraging probabilistic convention representation (e.g. as done by our matrix factorization method) in tasks where joint strategies are more likely to overlap.

Do the gains persist on an established benchmark such as Overcooked?

Preliminary experimentation in Overcooked revealed little differentiation in rewards across the various conventions that form during population training. We do not expect ConventionPlay to have an advantage in such scenarios, similar to the reported results in the non-differentiated variants of the Matrix Game, PMR, and Task Relay.

What does ConventionPlay add beyond prior work that chooses whether to lead or follow based on how much a partner can adapt?

We notice two significant contributions beyond Nikolaidis et al.’s work on partner adaptability: First, our method of defining and discovering conventions is based on the compatibility of policies, and therefore we can discover conventions in a range of environments purely using a population based training method. Second, BAM represents “adaptability” explicitly as a scalar and has a pre-programmed preference for the “best” convention. In our setting a partner can support a subset of many conventions (5-8 in our tasks), and the best one is not explicitly defined in advance. Our recurrent policies implicitly build a belief over which conventions a partner supports. We reason that this latent representation is more flexible than a scalar, as it allows an RL-optimized policy to switch from a probing strategy to a committing strategy that maximizes return over the episode. We will include a discussion of this work in revisions of this text.

Reviewer_DNA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ConventionPlay for ad-hoc collaboration with partners of different adaptive capabilities. It partitions fixed K0 policies into conventions using cross-play returns, trains K1 partners that support only subsets of these conventions, and then trains a K2 agent to probe a partner’s capabilities and steer it towards the best convention it can support. Experiments cover the Repeated Matrix Game, Point Mass Rendezvous, and Task-Relay.

Critical strengths

The method explicitly models capability-limited adaptive partners. Training with K1 partners that support only subsets of conventions encourages the K2 agent to probe and lead when beneficial, rather than always following the partner. The method performs strongly on the constructed adaptive-partner populations. Under full payoff differentiation The spectral variant obtains normalized returns of 52.63 on Matrix Game, 66.11 on PMR, and 53.66 on Task-Relay, the highest values in the corresponding rows.

Critical weaknesses

The evaluation distributions closely match the method’s assumptions. P(sample) directly uses the capability-set construction, while P (heur)1 is manually designed to exhibit limited-convention adaptation. This does not establish robustness to partners produced by substantially different adaptation mechanisms.

Training cost and convention discovery are not sufficiently controlled. ConventionPlay requires training multiple additional K1 policies, but the default K1 training-population size and total interaction cost are not clearly reported. Moreover, the main experiments set q1 from domain knowledge to 8, 6, and 5 for Matrix Game, PMR, and Task-Relay, respectively, without oracle-partition, random-partition, or automatic-q comparisons on these tasks.

Questions for authors

Assessment-critical experiment: Does ConventionPlay still outperform the baselines against adaptive partners trained independently with different objectives or learning algorithms, rather than through the same capability-set construction?

Assessment-critical experiment: Can the authors compare discovered, oracle, and random convention partitions on all three main tasks, and report results with q, q selected automatically as well as agreement with the ground-truth convention labels?

Clarification: How many K1 policies are trained in the default configuration? What are the total environment interactions and computational costs of ConventionPlay compared with each main baseline?

2 comments

Assessment-critical experiment: Does ConventionPlay still outperform the baselines against adaptive partners trained independently with different objectives or learning algorithms, rather than through the same capability-set construction?

We have extended our evaluation to include adaptive partners trained independently with different objectives and learning algorithms, namely the ad-hoc agents of every method in our experiments, which together form P(2)AHT\mathcal{P}^{(2)}{\text{AHT}}. Please see Appendix E of the updated submission for results and discussion. In the fully differentiated case, where we expect the largest gains, we see that ConventionPlay improves on the best baseline in PMR (by 18 points) and Task-Relay (by 20 points), and is also competitive in the Matrix Game. This, along with its performance against P(1)sampled\mathcal{P}^{(1)}{\text{sampled}} and Pheur(1)\mathcal{P}^{(1)}_{\text{heur}}, shows that ConventionPlay coordinates well with a wide range of adaptive partners, not only those built through our capability-set construction.

Assessment-critical experiment: Can the authors compare discovered, oracle, and random convention partitions on all three main tasks, and report results with q, q selected automatically as well as agreement with the ground-truth convention labels?

We introduce a new ablation to isolate the effect of an accurate convention partition. ConvPlay (Rand) builds the subsets for its intermediate K1K_1 agents by randomly sampling the K0K_0 population, simulating the suggested random-partition experiment. End-to-end results for ConvPlay (Rand) on all three tasks are shown in Appendix E of the updated submission (Figure 12, Tables 15-18), trained with the same frame budget as ConventionPlay (Table 14). Rand also performs strongly in several settings, and we expect its benefit varies with the disparity among the conventions discovered in the K0K_0 population. As the hyperparameter qq is the main determinant of partition accuracy, we have also extended our analysis with a sensitivity test to qq for the full ConventionPlay pipeline. Since there is no clean ground truth for the conventions of a learned K0K_0 population, where policies may implement several conventions partially or exhibit non-transitive compatibility, we test how the full pipeline responds to a range of partitions. Figure 14 of the updated submission shows the efficiency of the ad-hoc agent when interacting with the Psampled(1)\mathcal{P}^{(1)}_{\text{sampled}} population in PMR across a range of estimated conventions qq, which peaks near or slightly above the true qq and degrades most when qq is underestimated.

Clarification: How many K1 policies are trained in the default configuration? What are the total environment interactions and computational costs of ConventionPlay compared with each main baseline?

Our default configuration trains 20, 15 and 12 K1K_1 policies for the Matrix Game, PMR and Task-Relay respectively (Appendix C.3 of our updated submission).

Total environment frames for ConventionPlay and each main baseline, summed across all levels of the hierarchy, are reported in Table 14 (Appendix E). ConventionPlay uses a budget at most that of any other method.

Reviewer_DNA4OpenReview ↗

Feedback to the authors

Many thanks for your answers, which make me understand the problem better. I will keep the ovrerall rating considering the novelty and contributions of the techniques.

Reviewer_7BXKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ConventionPlay for ad-hoc collaboration with partners that differ in their ability to adapt across conventions. It partitions a self-play population into conventions using cross-play returns, trains K1 partners as best responses to restricted convention subsets, and trains a final K2 agent against the union of K0 and K1 partners. The intended behavior is to follow fixed partners while probing and steering adaptive partners toward higher-value conventions. Experiments on three environments report improvements on capability-limited adaptive partners, sometimes with reduced performance on fixed partners.

Critical strengths

The paper identifies a clear limitation of training only against fixed-convention partners. The proposed population construction is simple and can be combined with existing diversity-generation methods. The distinction between following rigid partners and leading adaptive ones is intuitive.

The evaluation covers three environments, learned and hand-coded partners, and several payoff structures. The fixed-partner evaluation usefully exposes the cost of probing, while the K1-size and convention-partitioning studies examine important design choices. Improvements on the constructed adaptive partners are often substantial.

Critical weaknesses

The central notion of capability is not adequately established. In Section 5, a K1 partner's capability set is implemented only by restricting the conventions seen during its best-response training. There is no guarantee or measurement that the resulting policy supports every convention inside this set, or cannot adapt to conventions outside it. The paper therefore conflates training exposure with execution-time capability, making the claimed capability inference difficult to interpret.

The main adaptive evaluations closely reproduce the training assumptions. P1_sampled uses the same capability-set sampling and best-response construction, with ground-truth convention labels. P1_heur directly hand-codes partners that switch when the ego proposes a convention in their repertoire. These are useful controlled tests, but they do not substantiate the claimed out-of-distribution robustness to generally adaptive partners. The current evidence mainly demonstrates specialization to the authors' partner model.

The largest Task-Relay gains are not cleanly attributable to selecting better conventions. Even with non-differentiated payoffs, several baselines score approximately 2--12 against adaptive partners, compared with 70--83 for ConventionPlay, while FCP scores approximately 77--83. This suggests baseline-specific initiation or deadlock failures rather than a general advantage in capability inference. Without a deadlock analysis, the source of the main improvement remains unclear.

The contribution is narrower than the motivation suggests. K2 is a standard recurrent population best response; the main addition is the restricted K1 population. Prior work already studies leading adaptive teammates and probing partner adaptability, including Nikolaidis et al. (2016), Stone et al. (2010), and Xie et al. (2020). Moreover, Table 5 uses substantially different population sizes across methods, and there is no matched comparison against unrestricted adaptive K1 partners. Thus the specific benefit of capability limitation is not isolated from population and training-budget differences.

Questions for authors

Is there any guarantee or empirical verification that a K1 policy's actual behavior matches its designated capability set, rather than merely reflecting restricted training exposure? Which aspect of P1_sampled is intended to be out of distribution, given that it uses the same partner-generation mechanism and ground-truth convention labels? What causes the near-zero non-differentiated Task-Relay results for several baselines, and what fraction of ConventionPlay's gain comes from avoiding deadlock rather than convention selection? What are the K1 population sizes and total training budgets for each method, and is there an existing matched control using unrestricted adaptive K1 partners?

3 comments

Is there any guarantee or empirical verification that a K1 policy's actual behavior matches its designated capability set, rather than merely reflecting restricted training exposure?

We present empirical evidence that training exposure to partners of a certain convention allow K1K_1 agents to coordinate on only said conventions. This is done by individually testing K1K_1 agents on Ptest0\mathcal{P}^0_{test} agents for which ground-truth convention labels are defined. Table 7 in Appendix C.3 shows the number of conventions each capability-limited K1K_1 agent was discovered to be competent in by thresholding its coordination efficiency with the entire Ptest0\mathcal{P}^0_{test}. Most agents are found to be competent in 2-3 conventions, which matching our hyperparameters for generating K1K_1 subsets.

Which aspect of P1_sampled is intended to be out of distribution, given that it uses the same partner-generation mechanism and ground-truth convention labels?

We consider P^(1){sampled} out of distribution because of the underlying K0K_0 population used to build this test set. Partners are sampled from P^(0){test} which is generated per task in a domain-specific way for which we have convention labels by design. In the Matrix Game and PMR these are hard-coded policies (a single fixed action, or driving to a fixed landmark). In Task-Relay they are self-play populations trained under environment variants that each bias agents toward a particular manufacturing path. The K1K_1 agents in P^(1){sampled} are therefore best responses to subsets of policies that no K2K_2 method saw during training. We have additionally added a new out-of-distribution population P^(2){AHT} which is simply the various AHT benchmarks we are using in our experiment. This test set allows measuring ConventionPlay with richer types of adaptive partner behavior. Results are reported in Appendix E.

What causes the near-zero non-differentiated Task-Relay results for several baselines, and what fraction of ConventionPlay's gain comes from avoiding deadlock rather than convention selection?

We thank the reviewer for their attention to detail in our choice of benchmark. In the original Task-Relay, the near-zero baseline results stem from a co-adaptation deadlock, in which each agent repeats non-committal actions while waiting for the other to commit first. The ability to avoid this deadlock is a feature of ConventionPlay that was revealed to us through experimentation. It arises from pairing a best-response objective with a population of "follower" agents (P(1)\mathcal{P}^{(1)}), which teaches the policy to "lead" in order to break the deadlock. We acknowledge that deadlock avoidance is entangled with convention selection in this environment, which makes it difficult to isolate the steering behavior we intend to measure. We have therefore simplified the environment by removing the "HANDOFF" action and allowing either agent to "APPLY" a tool when the right prerequisites are matched. The resulting policies must still negotiate their approach to the task, but are considerably less likely to get stuck in a deadlock. This modified environment is used for all new end-to-end results in Appendix E. In PsampledP_{sampled} and PAHTP_{AHT}, where the leading baseline, SyKLRBR, deadlocks in fewer than 10% of episodes, ConventionPlay (MatFact) retains an advantage of roughly 20 points, indicating that its gains do not stem from deadlock avoidance alone.

Additionally, our new probing analysis in Task-Relay (Figure 13) indicates that ConventionPlay's gains reflect convention selection and not only deadlock avoidance. When the partner holds the lowest-value line for 10 to 20 steps, ConventionPlay (MatFact) still finishes on the best line in roughly 60% of episodes, compared to under 10% for the baselines. ConventionPlay (Spectral) does not exhibit this behavior in Task-Relay. We plan to include both variants of the environment, along with a detailed discussion, in the final submission.

What are the K1 population sizes and total training budgets for each method, and is there an existing matched control using unrestricted adaptive K1 partners?

Please see Appendix E of the updated submission, where we control for training budget. ConventionPlay is trained with a total environment-frame budget, summed across all levels of the hierarchy, that is at most that of any other method; budgets for each method are reported in Table 14. ConventionPlay uses K1K_1 training populations of 20, 15 and 12 agents in the Matrix Game, PMR and Task-Relay respectively.
We additionally introduce SyKLRBR (Matched), which mirrors ConventionPlay in using a level-2 hierarchy and the same number of K1K_1 agents, but trains each K1K_1 agent as a best response to the full K0K_0 population rather than a capability-limited subset. This provides the matched control with unrestricted adaptive K1K_1 partners, and isolates the effect of capability-limited training from that of population size. Our primary results hold under this more conservative comparison.

. Prior work already studies leading adaptive teammates and probing partner adaptability, including Nikolaidis et al. (2016), Stone et al. (2010), and Xie et al. (2020).

We wanted to extend our discussion of prior work as this a point other reviewers have brought up as well:

Stone [2] and Nikolaidis [3] do consider adaptive partners, but their solutions make strong assumptions about adaptivity. In [2], the teammate is a single best-responder whose rule and parameters are known to the ego agent, which reduces leading it to a planning problem. In [3], adaptivity is captured by a scalar which cannot represent partners adapting to some conventions but not others, and is integrated into the robot’s policy through a hand-designed decision model. Using only interactions within the game, we discover a convention structure that a task admits and derive a diverse set of adaptive agents from them. The resulting ad-hoc agent implicitly probes, infers, and steers partners to a convention as a consequence of maximising its return against this population.

Xie [4] focuses on opponent shaping, similar to M-FOS covered in our related works: they shape non-stationary partners over repeated episodes by learning how a partner’s latent strategy evolves in response to the ego agent’s behavior. The key differentiator is that ConventionPlay learns to shape a test-time partner’s behavior within a single episode, without requiring repeated interactions with that partner.

[2] Stone, Kaminka, and Rosenschein (2010). Leading a Best-Response Teammate in an Ad Hoc Team.

[3] Nikolaidis et al, (2016). Formalizing Human-Robot Mutual Adaptation: A Bounded Memory Model

[4] Xie et al. (2020). Learning Latent Representations to Influence Multi-Agent Interaction

Reviewer_ZX2LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ConventionPlay, an ad-hoc collaboration method that trains agents with partners capable of adapting to restricted subsets of coordination conventions. This training encourages agents to probe partner capabilities and balance leading and following. Experiments across three cooperative tasks show improved coordination with adaptive partners, particularly when conventions differ in reward.

Critical strengths

S1. Interesting problem formulation.

The paper addresses partner adaptability, an important aspect of ad-hoc teamwork. Considering partners that can adapt to multiple, but not all conventions is a meaningful extension of fixed-convention settings. Learning when to lead or follow could improve collaboration by allowing agents to coordinate on better joint strategies.

Critical weaknesses

W1. Insufficient quantitative evidence for the claimed probing-and-steering behavior.

Figure 4 (right) shows improved returns as partner adaptability increases, which is consistent with steering but does not directly measure whether partners switch to higher-value conventions in response to the agent's actions. While Appendix D.4 provides useful qualitative evidence through PMR trajectories, I would like to see a systematic evaluation of how frequently successful steering occurs across partners and environments, particularly in Task-Relay.

W2. Insufficient control for training population diversity and computational resources.

Table 5 shows that the K0 population sizes differ across methods. Furthermore, ConventionPlay trains additional K1 agents, potentially benefiting from greater partner diversity and training resources. Comparisons under matched baseline computational budgets, would help isolate the proposed contribution.

Questions for authors
  1. Do the probing and steering behaviors arise from capability-limited (K_1) partner training, or from a larger training population? Compare all methods with matched total training population sizes to control overall training budgets would help assess whether the gains can be attributed to the proposed training structure.

  2. I would also like to see whether similar probing-and-steering behavior emerges in Task-Relay, where the largest performance improvements are reported. Such analysis would help establish whether the proposed mechanism consistently explains the observed gains across environments.

  3. How representative are the PMR trajectories in Figure 10 across the experiments? Quantifying successful switches to higher-value supported conventions and fallback behavior with fixed partners across partners and seeds would help assess whether these examples reflect a reliable strategy underlying the reported gains.

3 comments

Do the probing and steering behaviors arise from capability-limited (K_1) partner training, or from a larger training population? Compare all methods with matched total training population sizes to control overall training budgets would help assess whether the gains can be attributed to the proposed training structure.

Please see Appendix E of the updated submission, where we control for training budget. ConventionPlay is trained with a total environment-frame budget, summed across all levels of the hierarchy, that is no larger than that of any other method (Table 14). We additionally introduce SyKLRBR (Matched), which mirrors ConventionPlay in using a level-2 hierarchy and the same number of K1K_1 agents, but trains each K1K_1 agent against the full K0K_0 population rather than a capability-limited subset. This isolates the effect of training capability-limited adaptive partners from that of simply training against more partners.

Our primary results hold in this controlled comparison, indicating that the gains stem from the capability-limited K1K_1 structure rather than from a larger training population.

I would also like to see whether similar probing-and-steering behavior emerges in Task-Relay, where the largest performance improvements are reported. Such analysis would help establish whether the proposed mechanism consistently explains the observed gains across environments.

Please see Figure 13 of the updated submission for a probing analysis in Task-Relay similar to those completed for the Matrix Game and PMR. When paired with tit-for-tat-like partners that hold a low-value convention for 10--20 steps before following, the best-performing variant of ConventionPlay (MatFact) still induces a switch to the best convention in roughly 60% of episodes, compared to under 10% for the baselines.

How representative are the PMR trajectories in Figure 10 across the experiments? Quantifying successful switches to higher-value supported conventions and fallback behavior with fixed partners across partners and seeds would help assess whether these examples reflect a reliable strategy underlying the reported gains.

We appreciate the suggestion to extend the analysis in Figure 10. Table 13 (Appendix D.5) of the updated submission reports, per partner and averaged across K2K_2 seeds, the percentage of episodes in which each method induces a tit-for-tat partner to switch to its higher-value target. ConventionPlay is the most effective on average (22.2% for MatFact and 17.2% for Spectral, versus 12.6% for the strongest baseline), with the largest gains on high-value targets (landmarks 3--5). With fixed-convention partners (Ptest0\mathcal{P}^0_{test}), ConventionPlay falls back to the partner's convention, with efficiency within $\approx$4 points of the best baseline or above it in the differentiated PMR variants (Table 15).

Thank you for the clarifications. I find the responses to Q2 and Q3 convincing, and they strengthen the paper.

For Q1, I appreciate the budget controls and SyKLRBR (Matched). However, my remaining concern is the comparison with the other baselines: their performance may also depend on population size. Comparisons matching both total population size and environment-frame budget would help clarify how much of the advantage over those methods remains under these controls.

I also share reviewer 7BXK’s concern about the contribution. Prior work already studies probing partner strategies [1] and leading adaptive teammates toward higher team rewards [2]. The main advance here is convention discovery and restricted K1 population construction, and I still find this contribution somewhat limited.

These remaining concerns leave me inclined to maintain my score.

[1] Ma et al. (2024). Fast Peer Adaptation with Context-aware Exploration.

[2] Stone, Kaminka, and Rosenschein (2010). Leading a Best-Response Teammate in an Ad Hoc Team.

For Q1, I appreciate the budget controls and SyKLRBR (Matched). However, my remaining concern is the comparison with the other baselines: their performance may also depend on population size. Comparisons matching both total population size and environment-frame budget would help clarify how much of the advantage over those methods remains under these controls.

We acknowledge that some of the remaining baselines vary in population size. However, these methods also differ in their objectives and their use of the cognitive hierarchy. CoMeDi, MEP, and LIPO use explicit diversity objectives for K0K_0, and so scale differently with population size than SyKLRBR, FCP, or ConventionPlay. For this reason, we matched them on environment interactions rather than population count.

We are happy to include population-size sweeps for these baselines, or comparisons matched on total population and frames, in a final revision.

I also share reviewer 7BXK’s concern about the contribution. Prior work already studies probing partner strategies [1] and leading adaptive teammates toward higher team rewards [2]. The main advance here is convention discovery and restricted K1 population construction, and I still find this contribution somewhat limited.

Many ad-hoc methods can result in “probing” strategies that reveal a partner’s hidden type (similar to [1] or the baselines in our experiments), but are ultimately intended for a test-time population of fixed-convention agents. Our adaptive populations, P(1)\mathcal{P}^{(1)} sampled and heuristic, expose the limitations of existing probing behavior and represent important settings for developing ad-hoc agents.

Stone [2] and Nikolaidis [3] do consider adaptive partners, but their solutions make strong assumptions about adaptivity. In [2], the teammate is a single best-responder whose rule and parameters are known to the ego agent, which reduces leading it to a planning problem. In [3], adaptivity is captured by a scalar which cannot represent partners adapting to some conventions but not others, and is integrated into the robot’s policy through a hand-designed decision model. Using only interactions within the game, we discover a convention structure that a task admits and derive a diverse set of adaptive agents from them. By maximizing return against this population, the resulting ad-hoc agent implicitly probes, infers, and steers partners to a convention.

Xie [4] focuses on opponent shaping, similar to M-FOS covered in our related works: they shape non-stationary partners over repeated episodes by learning how a partner’s latent strategy evolves in response to the ego agent’s behavior. The key differentiator is that ConventionPlay learns to shape a test-time partner’s behavior within a single episode, and without requiring repeated interactions with that partner.

We will include sharper contrast with these works in subsequent revisions of our text.

[1] Ma et al. (2024). Fast Peer Adaptation with Context-aware Exploration.

[2] Stone, Kaminka, and Rosenschein (2010). Leading a Best-Response Teammate in an Ad Hoc Team.

[3] Nikolaidis et al, (2016). Formalizing Human-Robot Mutual Adaptation: A Bounded Memory Model

[4] Xie et al. (2020). Learning Latent Representations to Influence Multi-Agent Interaction

Comments on the paper

1 comment
Area_Chair_Jhh4OpenReview ↗

Please review the authors’ rebuttal and update your assessment

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.