- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies the problem of Offline EF. Existing equilibrium-finding methods mainly rely on online interaction. The paper argues that ideas from offline reinforcement learning can be adapted, but behavior cloning and model-based learning are effective under different types of datasets: behavior cloning works better with expert data, while model-based methods benefit from broader coverage. The authors therefore propose BOMB, which combines the two methods using a mixing parameter . The paper also constructs four types of datasets, including random, expert, learning, and hybrid datasets, to provide a unified benchmark for evaluation. Experiments on these datasets show that BOMB generally outperforms the offline RL baselines as well as the individual behavior-cloning and model-based methods.
Critical strengths
- The problem is novel and meaningful. The paper introduces offline equilibrium finding, which is relevant to settings where online interaction is costly or impractical, such as sports games.
- Building an Offline EF benchmark is a valuable contribution. The authors construct random, expert, learning, and hybrid datasets across eight commonly used IIEFGs, which can facilitate future comparison of different methods.
Critical weaknesses
- The use of grid search for selecting in the main experiments conflicts with the offline setting. The paper acknowledges that grid search requires interaction with the real game, yet RQ2 states that grid search is used as the default method in the remaining experiments. Therefore, the subsequent comparisons of BOMB with BC and MB rely on information that would not be available in a strictly offline setting.
- The baselines are outdated. The two offline RL baselines, BAIL and MOPO, are both from 2020.
Questions for authors
- Why did the authors not use the learning-based CKA estimator, or another fully offline method, as the default for the main experiments? Could the authors report the main BOMB results using a fully offline -selection method, such as the CKA-based predictor? If BOMB still clearly outperforms BC, MB, and the baselines with a fully offline -selection method, this would improve my assessment.
- Could more recent baselines be included?
- The abstract states that BOMB can compute both NE and CCE, but the CCE experiments only evaluate MB-JPSRO. Could the authors clarify this claim?
2 comments
Dear Reviewer 8uzU,
Thank you again for your valuable review. As the discussion period is coming to an end, we would like to kindly check whether our responses have addressed your concerns. In particular, following your comments, we added results using a fully offline method to select (offline grid search) in the revision, which we hope addresses your main concern. We would be happy to provide any further clarification if needed.
Best regards, The Authors
Response to Reviewer 8uzU
We thank the reviewer for the valuable review and provide the following responses to your concerns.
W1 & Q1: Using a fully offline method to select α.
Response: We thank the reviewer for this comment. We agree that grid search uses information from the true game, which is not available in a strictly offline setting.
We originally used grid search to show the potential of combining BC and MB, separately from the difficulty of selecting α. We did not use the CKA predictor as the default because it requires labels from a source game, i.e., a simulator for a related game, and its accuracy depends on the similarity between the source and target games (e.g., it was weaker on 2-player Leduc poker in Fig. 7).
In the revision, we report results with a fully offline method, offline grid search method, which selects α by minimizing the equilibrium gap in the learned game (Theorem 4.6(ii)). We evaluate it on 2- and 3-player Kuhn poker and Leduc poker with different dataset sizes (Appendix F.2, Figs. 20–22). The results are as follows:
In summary, offline grid search is a fully offline alternative that performs close to grid search whenever the data have reasonable coverage, while near-expert data remain challenging. We acknowledge that it is difficult for a single fully offline selection method to work well in all cases. This is why we proposed several methods for selecting α, which require different information and therefore apply to different scenarios. Grid search applies when a small number of policy evaluations in the target game is possible, as in offline-to-online settings. It is the most reliable method. The learning-based method applies when a simulator exists for a related or simplified game, but not for the deployed one. For example, a smaller variant of a card game, an abstracted version of a security game, or an earlier version of a system for which both logs and a simulator exist. The predictor is trained once there and then applied with no interaction with the target game. Finally, the offline grid search method can be used fully offline, but it is not guaranteed to perform well in all fully offline settings.
W2 & Q2: More recent baselines.
Response: We thank the reviewer for this suggestion. We want to clarify why we chose BAIL and MOPO, and why we expect more recent offline RL methods to face the same limitation.
Choice of baselines. BAIL and MOPO represent the two main families of offline RL: model-free and model-based methods. These two families correspond to the two components of BOMB, behavior cloning and the model-based method. Fig. 6 therefore compares, within each family, methods that optimize a single player’s return with our approach that targets an equilibrium.
The limitation lies in the objective, not in the specific algorithm. Single-agent offline RL methods, including more recent ones, aim to maximize each player’s return against the opponents’ behavior recorded in the data. This tends to move each player’s strategy away from the equilibrium mixing toward a near-deterministic strategy, which can be exploited by an adapting opponent and increases NashConv. This limitation comes from the objective rather than from the particular algorithm, so we expect it to persist for more recent offline RL methods as well.
Q3: The claim that BOMB computes both NE and CCE.
Response: We thank the reviewer for pointing this out. In the CCE experiments, we did not include BC or the full BOMB framework because the offline datasets are generated by independent strategies of each player rather than a joint (correlated) strategy. BC imitates these independent strategies. It could still produce a candidate CCE, since any profile of independent strategies is a valid candidate (Remark D.14), but it cannot learn correlations between players, which is what distinguishes a CCE from an NE. We therefore focused the CCE experiments on MB-JPSRO, which can compute joint strategies. Accordingly, we have revised the abstract to state that BOMB computes an approximate NE, and its model-based component computes approximate CCE in multi-player games.