- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies zero-shot evaluation of a strategy that cannot be run, using only a short record of its past matches. The goal is to predict its payoffs against unseen opponents and under unseen game conditions, and to find opponents that exploit it. To do this, the authors propose PLAYREAD, an in-context imitation policy with a retrieval memory, trained on a population of runnable strategies. PLAYREAD infers a runnable stand-in for the target strategy and plays it in a simulator, which supplies the rules, the opponent, and the condition, so the only unknown is how the strategy responds. At each step of a new match, it retrieves recorded moments that resemble the current situation and the responses that followed them, while the match in progress is kept in its context. Therefore, a new strategy is evaluated zero-shot by replacing its record in the memory. The authors conduct PLAYREADS on four games and these results show that with short records or unseen conditions, the advantage comes from playing the inferred policy rather than from a readout of the same memory
Critical strengths
-
The problem studied in this paper is important for real-world applications and the proposed method that achieving high zero-shot prediction accuracy against new opponents by inferring a runnable stand-in for the target strategy and playing it in a simulator, is novelty and useful.
-
This paper provides sufficient and carefully controlled empirical study. In the experimental part, the method is evaluated across four domains with different payoff structures. These controlled comparisons make the main claims credible and clarify when the approach works and when it does not.
Critical weaknesses
One concern is the applicability of the method in the real-world setting. Specifically, the method heavily rely on a population of runnable strategies. However, in some motivating scenarios, such as a rival bot, a population of similar runnable strategies may not be available. The claim of zero-shot evaluation of unavailable strategies is well supported mainly when the target resembles the training population, which may not hold in real applications.
Questions for authors
-
Could the authors explain the applicability issue? In other words, for the motivating scenarios, such as a rival bot, how would one obtain a population of runnable strategies similar to the target? If such a population is not available, are there alternative solutions?
-
This work is closely related to in-context opponent modelling. Could the authors discuss more explicitly how your formulation and method differ from these works?
3 comments
Thanks for the authors' responses. The discussion about in-context opponent modeling and the explanation of applicability address most of my concerns, and I would like to keep my score, which is originally positive.
Thank you for the follow-up
Thank you for your follow-up and positive assessment. We are glad our responses addressed your concerns, and the clarifications are now reflected in the revised paper.
Response to Reviewer v5i4
Thank you for the review and for recognizing the practical importance of the problem. We answer each point below.
Every strategy we evaluate is held out from training, and PlayRead stays accurate when the target differs from the training population and its record is short. A reader trained only on scripted strategies evaluates agents trained by reinforcement learning from one recorded episode, reaching 0.824 in an unseen condition against 0.785 for the best baseline (Table 11).
This follows from how PlayRead is built (Section 3.2). The shared network holds what is common to all strategies. The memory holds what is specific to the target, its own record. Since no input says which strategy it imitates, the network learns to take each strategy's behavior from the memory. A held-out strategy therefore needs no change to the network, only its record in the memory.
Accuracy improves further as the population comes closer to the target. Adding 270 other reinforcement-learning agents to the scripted training population raises the one-episode correlation on the 90 held-out agents from 0.838 to 0.890 in a seen condition (Table 12). The population can also be built from players that cannot be run. For a rival bot, replays of other players and bots can be turned into strategies by behavioral cloning, as our Overcooked population already does with 712 clones of recorded human play.
The two address different problems. In-context opponent modeling, as in TAO, OMIS and OEOM, aims to control an agent. It builds one that adapts to the opponent it faces. Our aim is evaluation, from the standpoint of an evaluator rather than a player. We predict how a strategy fares against others before it is met.
The objectives differ accordingly. In these methods the agent's return is maximized, so the opponent model only has to capture how the opponent plays against that agent. PlayRead maximizes the likelihood of the strategy's own actions in matches against opponents absent from its record (Eq. 4). Its stand-in must reproduce the strategy's payoffs against any opponent and in unseen conditions, playing whole matches alone in the simulator. Because its own mistakes carry forward over a match, the method trains it with interactive imitation on matches it plays itself (Section 3.2).
Because each stand-in can play any opponent, matches between strategies that have never met can be simulated, a question that the objective of opponent modeling does not target. In a round robin among 200 held-out strategies in the seen condition, both sides of every match are stand-ins built from 18 recorded episodes. The predicted outcomes correlate with the true ones at 0.935, against 0.952 when one side is the real strategy (Appendix L).
Changes to the paper
The revision, marked in blue, adds the comparison with in-context opponent modeling to the related work in Section 5 and states in Section 3.2 how a population can be assembled from replays.