acceptodds
Under review as a conference paper at ICLR 2027

Zero-Shot Evaluation of Strategies by Playing Imitation Policies

Abstract

How a strategy handles an opponent it has never met, or a condition it has never seen, is not settled by running it against the opponents at hand, since the one that defeats it need not be among them. The strategies that most need an answer often cannot be run at will, leaving only a short record of past matches. Ratings and outcome predictors fit the results of those matches but take no condition as input, while a policy cloned from the record knows only the situations the record reached. The match can still be played, with a policy inferred from the record in place of the original. The simulator supplies the opponent, the condition and the dynamics, so the only unknown is how the strategy responds. We introduce PlayRead, an in-context imitation policy with a retrieval memory, trained on a population of runnable strategies. At each step of a new match, it retrieves recorded moments that resemble the current situation and the responses that followed them, while the match in progress is kept in its context. Training pairs the record of one strategy with its actions against an opponent absent from that record, and a new strategy is evaluated zero-shot by placing its record in the memory, whose values may then be tuned on that record. We evaluate PlayRead on a continuous-control game, Leduc poker, Overcooked and the iterated prisoner's dilemma. In the continuous-control game, playing the inferred policy is substantially more accurate than predicting payoff from the same memory when the record is short or the condition is unseen. In an unseen condition, PlayRead with one recorded episode predicts matchups more accurately than behavioral cloning fitted to eighteen. For most strategies the inferred policy also names the condition in which the strategy is weakest, and it yields an opponent that defeats it. With six open language models in the prisoner's dilemma, PlayRead predicted how they would play one another before they met.

Primary area:
reinforcement learning
Keywords:
zero-shot evaluation, multi-agent evaluation, in-context learning, amortized policy inference, imitation learning, opponent modeling, language model agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_v5i4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies zero-shot evaluation of a strategy that cannot be run, using only a short record of its past matches. The goal is to predict its payoffs against unseen opponents and under unseen game conditions, and to find opponents that exploit it. To do this, the authors propose PLAYREAD, an in-context imitation policy with a retrieval memory, trained on a population of runnable strategies. PLAYREAD infers a runnable stand-in for the target strategy and plays it in a simulator, which supplies the rules, the opponent, and the condition, so the only unknown is how the strategy responds. At each step of a new match, it retrieves recorded moments that resemble the current situation and the responses that followed them, while the match in progress is kept in its context. Therefore, a new strategy is evaluated zero-shot by replacing its record in the memory. The authors conduct PLAYREADS on four games and these results show that with short records or unseen conditions, the advantage comes from playing the inferred policy rather than from a readout of the same memory

Critical strengths
  1. The problem studied in this paper is important for real-world applications and the proposed method that achieving high zero-shot prediction accuracy against new opponents by inferring a runnable stand-in for the target strategy and playing it in a simulator, is novelty and useful.

  2. This paper provides sufficient and carefully controlled empirical study. In the experimental part, the method is evaluated across four domains with different payoff structures. These controlled comparisons make the main claims credible and clarify when the approach works and when it does not.

Critical weaknesses

One concern is the applicability of the method in the real-world setting. Specifically, the method heavily rely on a population of runnable strategies. However, in some motivating scenarios, such as a rival bot, a population of similar runnable strategies may not be available. The claim of zero-shot evaluation of unavailable strategies is well supported mainly when the target resembles the training population, which may not hold in real applications.

Questions for authors
  1. Could the authors explain the applicability issue? In other words, for the motivating scenarios, such as a rival bot, how would one obtain a population of runnable strategies similar to the target? If such a population is not available, are there alternative solutions?

  2. This work is closely related to in-context opponent modelling. Could the authors discuss more explicitly how your formulation and method differ from these works?

3 comments
AuthorsOpenReview ↗

Response to Reviewer v5i4

Thank you for the review and for recognizing the practical importance of the problem. We answer each point below.

  1. Applicability when the target differs from the population

Every strategy we evaluate is held out from training, and PlayRead stays accurate when the target differs from the training population and its record is short. A reader trained only on scripted strategies evaluates agents trained by reinforcement learning from one recorded episode, reaching 0.824 in an unseen condition against 0.785 for the best baseline (Table 11).

This follows from how PlayRead is built (Section 3.2). The shared network holds what is common to all strategies. The memory holds what is specific to the target, its own record. Since no input says which strategy it imitates, the network learns to take each strategy's behavior from the memory. A held-out strategy therefore needs no change to the network, only its record in the memory.

Accuracy improves further as the population comes closer to the target. Adding 270 other reinforcement-learning agents to the scripted training population raises the one-episode correlation on the 90 held-out agents from 0.838 to 0.890 in a seen condition (Table 12). The population can also be built from players that cannot be run. For a rival bot, replays of other players and bots can be turned into strategies by behavioral cloning, as our Overcooked population already does with 712 clones of recorded human play.

  1. Difference from in-context opponent modeling

The two address different problems. In-context opponent modeling, as in TAO, OMIS and OEOM, aims to control an agent. It builds one that adapts to the opponent it faces. Our aim is evaluation, from the standpoint of an evaluator rather than a player. We predict how a strategy fares against others before it is met.

The objectives differ accordingly. In these methods the agent's return is maximized, so the opponent model only has to capture how the opponent plays against that agent. PlayRead maximizes the likelihood of the strategy's own actions in matches against opponents absent from its record (Eq. 4). Its stand-in must reproduce the strategy's payoffs against any opponent and in unseen conditions, playing whole matches alone in the simulator. Because its own mistakes carry forward over a match, the method trains it with interactive imitation on matches it plays itself (Section 3.2).

Because each stand-in can play any opponent, matches between strategies that have never met can be simulated, a question that the objective of opponent modeling does not target. In a round robin among 200 held-out strategies in the seen condition, both sides of every match are stand-ins built from 18 recorded episodes. The predicted outcomes correlate with the true ones at 0.935, against 0.952 when one side is the real strategy (Appendix L).

Changes to the paper

The revision, marked in blue, adds the comparison with in-context opponent modeling to the related work in Section 5 and states in Section 3.2 how a population can be assembled from replays.

Thanks for the authors' responses. The discussion about in-context opponent modeling and the explanation of applicability address most of my concerns, and I would like to keep my score, which is originally positive.

AuthorsOpenReview ↗

Thank you for the follow-up

Thank you for your follow-up and positive assessment. We are glad our responses addressed your concerns, and the clarifications are now reflected in the revised paper.

Reviewer_Nv8uOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a system that learns to fast mimic a player by combining pretraining on large amount of strategies and retrieval based fast adaption. It also include a memory tuning step for fast learning with a few trajectories of a new, unknown policy.

Critical strengths

The paper have good empirical contributions. It combines a few techniques in a well executed package, with interesting problem formulation and good evaluation. Specifically,

  • Training with per-policy memory pool is interesting, and the paper shows that having such memory pool is useful, and tuning the memory value vectors further improves performance.
  • The evaluation uses a large pool of policies covering both scripted and learned policies.
  • It contains ablations (Table 2) demonstrating the effect of different components. It is also helpful to have the ablation in two scales in terms of the number of strategies.
  • It is tested on a few games, hill game, Overcooked, Leduc and iterated prisoner’s dilemma.
Critical weaknesses
  • The hill game is relatively simple, and I am not sure how much true diversity do the pool of 500 or even 5000 policies exhibit. It might just be a few cluster of policies and within each cluster the policies are very similar to each other. Appendix M does help address some of the concern here because it also includes neural network policies trained with RL, but the PPO policies in the pool may not have large amount of diversity.
  • Related to the first point, although the paper have tested on a few games, the games mostly fall into the "proof-of-concept" tier in terms of their complexity (maybe except for overcooked).

Nonetheless, I think the strength of the paper generally outweigh these weakness, and none of the weaknesses are critical for a reject. So I would lean towards acceptance.

Questions for authors
  • In the overcooked experiment, what's the size of the policy pool for pretraining PlayRead, is it the entrie pool of 100?
  • Also in the overcooked experiment, how are the 50 PPO checkpoints selected, from one training PPO training run? How is the 50 human proxies trained? Are they trained on the same dataset? How big is that dataset. Is there a score distribution for the policies? Just trying to understand the diversity in the polcies.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Nv8u

Thank you for the positive assessment and the careful questions. We answer each point below.

  1. Diversity of the strategies

In the hill game we built the strategies around three archetypes, a camper, a pusher and an orbiter, that beat one another in a cycle. We did so on purpose, to plant a known structure and check whether PlayRead recovers it (Appendix L). Around each archetype the strategies still differ, since their weights are drawn at random and half also adapt to the opponent or the condition (Appendix D). Because this population has such a structure, we also test on agents trained with PPO outside it, where PlayRead still leads every baseline that reads the same record (Table 11, Appendix M).

We also measured this diversity directly. Given the same 512 states from the recorded matches, two PPO agents trained against different single opponents typically differ in their actions, whose components lie between -1 and 1, by 0.41. Two scripted strategies drawn at random typically differ by about as much, 0.40. In the prisoner's dilemma, we played each of the 221 published strategies of the Axelrod library against the same fixed sequences of opponent moves, so that all face the same opponent behavior, and compared their probability of cooperating in each round. Two strategies drawn at random differ by 0.44 per round on average. Close variants such as Tit For Tat and Tit For 2 Tats differ by only 0.20, and a typical strategy differs from a coin flip by 0.50. The Overcooked partners are described in answer 4.

  1. Complexity of the games

Larger games are indeed the natural next step, and the method is not tied to small ones. It needs only a simulator and recorded play, and a new strategy requires no change to the network, only its record in the memory. Within the present scope, Overcooked, the standard benchmark for human-AI coordination (Carroll et al., 2019), already involves games of 400 steps, and its partners include clones of human play. In the prisoner's dilemma, the targets include six open language models (Section 4.4).

  1. Size of the training population in Overcooked

The pool of 100 partners is used only for evaluation. One PlayRead model is trained across the four layouts on 4296 strategies, 949 to 1203 per layout, made up of 3584 PPO checkpoints and 712 human proxies. The training strategies, the held-out strategies, the pool and the probes are disjoint (Appendix N).

  1. How the pool was built

The 50 PPO checkpoints come not from one run but from 9 to 12 separate self-play training runs per layout, which differ in seed, entropy coefficient and reward-shaping horizon. The 50 human proxies follow the standard practice of Carroll et al. (2019), behavioral cloning of their recorded human play. All are trained on that same dataset, each on the pooled games of a different group of 2, 3, 4, 6 or 8 players, ten per group size. The dataset holds the games of 24 to 38 human players per layout. The pool and the training proxies use the games of 18 to 28 of these players, and the held-out proxies those of the remaining 6 to 10. The partners vary widely in skill. Scoring 20 points per soup, the 5th to 95th percentile of a partner's mean score is 47 to 131 in cramped_room, 18 to 182 in asymmetric_advantages, 31 to 100 in coordination_ring and 10 to 69 in forced_coordination.

Changes to the paper

The revision, marked in blue, adds the measurements of behavioral diversity in Appendix S and the Overcooked training population and the construction of its pool in Appendix N.

Thank you for your response. I already vote for acceptance and will keep my rating.

AuthorsOpenReview ↗

Thank you for your support

Thank you for your support. Your questions have helped us describe our strategy populations more clearly in the revised paper.

Reviewer_261AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PLAYREAD, a retrieval-augmented imitation policy for zero-shot evaluation of agents from limited behavioral records. The method trains on a population of runnable strategies to infer a policy for an unseen agent, which is then evaluated against new opponents and under unseen conditions through simulation. PLAYREAD combines retrieval-based memory, a causal Transformer, and interactive imitation learning. Experiments across four game environments demonstrate improvements in matchup prediction, opponent search, and identifying vulnerable conditions.

Critical strengths

The paper addresses the practical problem of evaluating agents that cannot be directly accessed, using only limited historical interactions. Inferring runnable surrogate policies provides a useful alternative to directly predicting match outcomes.

Critical weaknesses

The proposed architecture mainly combines existing techniques, including retrieval-augmented memory, causal Transformers, and interactive imitation learning. More importantly, when existing readers such as ANP, CNP, and context-window models are trained using the same training recipe, their performance becomes comparable to PLAYREAD. This raises my concerns about whether the improvements stem from the proposed architecture or primarily from the training procedure.

The outcome prediction baselines do not explicitly incorporate environment conditions, while PLAYREAD leverages condition-specific dynamics through simulation. A condition-aware predictor would provide a fairer comparison.

Questions for authors

The unseen conditions mainly involve changes in damping and wind, some of which exhibit symmetries with seen conditions. Can PLAYREAD handle more substantial changes in environment dynamics or interaction rules?

Since PLAYREAD is trained through action imitation rather than payoff optimization, how reliably do the inferred policies preserve interaction outcomes? Could opponent search exploit modeling errors rather than genuine weaknesses?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 261A

Thank you for the careful review.

  1. Contribution, architecture and training procedure

Most of the gain comes from playing a policy inferred by imitation rather than predicting payoffs, which is our central contribution. Predicting outcomes directly from PlayRead's memory gives 0.735 from one episode in the unseen condition, against 0.882 when PlayRead plays the inferred policy in the simulator (Table 1).

The rest comes from the architecture and its training, built together for zero-shot evaluation (Section 3.2). A shared network holds what is common to all strategies. A memory holds what is specific to the target, returning what it did next in similar situations. Training gives each part its role. Since no input says which strategy it imitates, the network learns to take each strategy's behavior from the memory. Interactive imitation trains this reading on situations the policy itself reaches, so errors compound less over a match. In Table 2, with 500 training strategies and one episode in the unseen condition, another strategy's record in the memory drops PlayRead from 0.829 to 0.380. Without interactive imitation it falls to 0.780, and without retrieval as well to 0.751.

Swapping only the reader in our pipeline separates the procedure from the architecture. With ANP as the reader, adding interactive imitation raises it from 0.812 to 0.858 from one episode in the unseen condition (Table 7). Architectural differences become clearer when the target lies outside the population. On 90 PPO agents in the hill game (Appendix M), PlayRead reaches 0.838 and 0.824 from one episode in the seen and unseen conditions, against 0.777 and 0.770 for a Prompt-DT-style context window and 0.757 and 0.791 for ANP with the same training and play (Table 12).

  1. A condition-aware predictor

The outcome predictor of Table 1, an embedding with a pairwise readout, follows the outcome prediction of Grover et al. (2018), which takes no condition. We gave its readout the condition as an input and trained it as in Table 1, with the same number of training matches spread over the six seen conditions on which PlayRead trains. In the unseen condition of Table 1, it improves from 0.729 to 0.747 from one episode and from 0.805 to 0.887 from 18 episodes, against 0.882 and 0.900 for PlayRead (Table 1). PlayRead remains far more accurate from a single episode. The condition input does not consistently help. Under the speed and force changes of answer 3 below, the physical parameters add only 0.002 and 0.005 from 18 episodes. The predictor stays below PlayRead in all three unseen conditions of Table 8 and under all three physical changes.

  1. Larger changes of dynamics and rules

The unseen condition of Table 1 is not the mirror image of any seen condition, and all three unseen conditions lie outside the convex hull of the seen ones. PlayRead is the most accurate reader in each (Appendix K, Table 8).

To change the environment dynamics, we reduced the speed cap to 30 percent, halved the maximum force and increased the integration step by 50 percent. PlayRead and the other methods that learn from the population of strategies were trained once under randomized physics, with the physical parameters as inputs, and each change lies outside the randomized range. From 18 episodes, PlayRead reaches 0.939, 0.888 and 0.942 under the speed, force and step changes, above every other method trained this way. The outcome predictor given the physical parameters reaches 0.897, 0.869 and 0.875, and the pipeline with ANP in place of our reader 0.917, 0.869 and 0.913. To change the interaction rules, we lowered the temptation payoff of the iterated prisoner's dilemma, added 5 percent action noise, shortened matches to 100 or 50 turns, and ended them at random. All training matches and records come from the standard 200-round game. The readers do not take the new rule as an input. Under all six rules, including the standard one, PlayRead beats a count table fitted to each strategy's record and played under the same rule, for example 0.900 against 0.862 with the lower temptation payoff and 0.891 against 0.857 with 50-turn matches.

  1. Do inferred policies preserve outcomes, and does search exploit error

With 18 recorded episodes in the hill game (Appendix R), the inferred policy wins or loses exactly where the original does in 97 percent of matchups in the seen condition and 91 percent in the unseen one, against 91 and 86 percent for behavioral cloning.

Opponents found by searching the inferred policy earn a mean payoff of 0.84 against the originals, close to 0.86 for opponents found by searching the originals directly. The payoff they promise exceeds what they realize by only 0.007 for PlayRead, against 0.115 for behavioral cloning (Section 4.5, Figure 4a).

Changes to the paper

The revision, marked in blue, adds the condition-aware predictor to Table 1, the studies of dynamics and rules to Appendix Q and the win-loss agreement to Appendix R.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts, and the new results address some of my concerns. I understand that the authors emphasize playing inferred policies rather than directly predicting payoffs as their main contribution. However, this approach builds on established ideas in imitation learning and opponent modeling, and its methodological novelty remains unclear, especially since alternative readers achieve comparable performance under the same training procedure. Additionally, the experiments remain largely focused on structured games with known dynamics and accessible simulators, leaving the broader applicability to more complex agent settings uncertain. Therefore, I have decided to maintain my original score.

AuthorsOpenReview ↗

Follow-up Response to Reviewer 261A

Thank you for the follow-up and for acknowledging the new results. We would like to briefly clarify how PlayRead differs from prior work and how far it applies.

Prior work evaluates an agent from its record in two ways. Representation methods predict outcomes directly, without playing, while clones trained on the data of each agent play in its place but require long records. PlayRead infers a policy from a short record, without per-strategy training, and plays it. Even from a single episode, it is more accurate than both.

Evaluation requires a faithful copy of the strategy, one that behaves as the original would against any opponent and under any condition. This is why PlayRead imitates the strategy. Opponent modeling, by contrast, models the opponent only as far as it matters for the response of its own agent, and only in play against that agent. Accordingly, when in-context imitation and opponent-modeling readers such as Prompt-DT and the pooled opponent encoder are used as evaluators, they fall short of PlayRead. Our training procedure and reader are designed specifically for faithful reproduction. The procedure improves other readers as well. The reader keeps each strategy in its own external memory, so a new strategy can be evaluated from its record alone. This is what gives PlayRead its zero-shot generalization to new strategies.

Beyond the controlled hill game, the strategies we evaluate include behavioral clones of human play in Overcooked, a standard benchmark for human-AI coordination, and open language models in the prisoner's dilemma. The revision also adds changes to the physics and to the interaction rules that no reader saw in training (Appendix Q). Like any evaluation based on play, PlayRead runs matches in a game environment, as do multi-agent benchmarks such as Melting Pot, closed-loop driving simulators and game arenas for language-model agents. What PlayRead removes is the need to run the evaluated agent itself.

Thank you again for your time and the careful discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.