acceptodds
Under review as a conference paper at ICLR 2027

Offline Equilibrium Finding in Extensive-form Games: Datasets, Methods, and Analysis

Abstract

Offline reinforcement learning (Offline RL) has emerged as a promising paradigm for solving real-world decision-making problems using pre-collected datasets. However, its counterpart for extensive-form games (EFGs) has received little attention. To fill this gap, we introduce the *offline equilibrium finding (Offline EF)* paradigm, which aims to compute equilibrium strategies of an EFG from a fixed offline dataset without interacting with the game. Offline EF is harder than offline RL, and the difference lies in the solution concept: offline RL seeks an optimal policy, whereas offline EF seeks an equilibrium. In single-agent offline RL, deciding whether one action is better than another only requires comparing the two in the data. An equilibrium, on the other hand, is a claim about every player's best response: no one can gain by switching to any other strategy. A dataset that records only how the players actually played says nothing about these deviations, and so cannot certify an equilibrium. Existing equilibrium-finding algorithms sidestep this by querying the game for best responses, which a fixed dataset cannot answer. Additionally, there are no public datasets or evaluation protocols for this problem, which leaves methods incomparable and progress hard to measure. To address these issues, we first construct a suite of offline datasets over a range of EFGs, spanning random, expert, learning, and hybrid data-generating strategies. We then propose BOMB, a framework that combines behavior cloning with a model-based method and allows any online equilibrium-finding algorithm (e.g., CFR, PSRO) to be used in the offline setting. We identify the sufficient conditions under which each component of BOMB is guaranteed to converge to an equilibrium, and show that BOMB, with its optimal mixing weight, is at least as good as the better of its two components. Extensive experiments show that BOMB consistently outperforms offline RL baselines and computes approximate Nash and coarse correlated equilibria in both two-player and multi-player games.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Extensive-form Games, Equilibrium Finding, Offline Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8uzUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the problem of Offline EF. Existing equilibrium-finding methods mainly rely on online interaction. The paper argues that ideas from offline reinforcement learning can be adapted, but behavior cloning and model-based learning are effective under different types of datasets: behavior cloning works better with expert data, while model-based methods benefit from broader coverage. The authors therefore propose BOMB, which combines the two methods using a mixing parameter α\alpha. The paper also constructs four types of datasets, including random, expert, learning, and hybrid datasets, to provide a unified benchmark for evaluation. Experiments on these datasets show that BOMB generally outperforms the offline RL baselines as well as the individual behavior-cloning and model-based methods.

Critical strengths
  1. The problem is novel and meaningful. The paper introduces offline equilibrium finding, which is relevant to settings where online interaction is costly or impractical, such as sports games.
  2. Building an Offline EF benchmark is a valuable contribution. The authors construct random, expert, learning, and hybrid datasets across eight commonly used IIEFGs, which can facilitate future comparison of different methods.
Critical weaknesses
  1. The use of grid search for selecting α\alpha in the main experiments conflicts with the offline setting. The paper acknowledges that grid search requires interaction with the real game, yet RQ2 states that grid search is used as the default method in the remaining experiments. Therefore, the subsequent comparisons of BOMB with BC and MB rely on information that would not be available in a strictly offline setting.
  2. The baselines are outdated. The two offline RL baselines, BAIL and MOPO, are both from 2020.
Questions for authors
  1. Why did the authors not use the learning-based CKA estimator, or another fully offline method, as the default for the main experiments? Could the authors report the main BOMB results using a fully offline α\alpha-selection method, such as the CKA-based predictor? If BOMB still clearly outperforms BC, MB, and the baselines with a fully offline α\alpha-selection method, this would improve my assessment.
  2. Could more recent baselines be included?
  3. The abstract states that BOMB can compute both NE and CCE, but the CCE experiments only evaluate MB-JPSRO. Could the authors clarify this claim?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 8uzU

We thank the reviewer for the valuable review and provide the following responses to your concerns.

W1 & Q1: Using a fully offline method to select α\alpha.

Response: We thank the reviewer for this comment. We agree that grid search uses information from the true game, which is not available in a strictly offline setting.

We originally used grid search to show the potential of combining BC and MB, separately from the difficulty of selecting α\alpha. We did not use the CKA predictor as the default because it requires labels from a source game, i.e., a simulator for a related game, and its accuracy depends on the similarity between the source and target games (e.g., it was weaker on 2-player Leduc poker in Fig. 7).

In the revision, we report results with a fully offline method, offline grid search method, which selects α\alpha by minimizing the equilibrium gap in the learned game (Theorem 4.6(ii)). We evaluate it on 2- and 3-player Kuhn poker and Leduc poker with different dataset sizes (Appendix F.2, Figs. 20–22). The results are as follows:

  • When the data contain at least 20% random data, the offline grid search method closely matches grid search in most settings. In particular, on Leduc poker, BOMB with offline grid search outperforms both BC and MB in most data compositions. On Kuhn poker, the offline grid search method mostly chooses MB, which already performs near-optimally in these small games when the data contain random play.
  • When the proportion of random data is at most 10%, the offline grid search method is less reliable. Near-expert data cover only a small part of the game, so the learned game is inaccurate in the uncovered parts, and evaluating strategies using it becomes unreliable. As the proportion of random data increases, the learned game becomes closer to the true game, and the offline evaluation becomes more reliable.

In summary, offline grid search is a fully offline alternative that performs close to grid search whenever the data have reasonable coverage, while near-expert data remain challenging. We acknowledge that it is difficult for a single fully offline selection method to work well in all cases. This is why we proposed several methods for selecting α\alpha, which require different information and therefore apply to different scenarios. Grid search applies when a small number of policy evaluations in the target game is possible, as in offline-to-online settings. It is the most reliable method. The learning-based method applies when a simulator exists for a related or simplified game, but not for the deployed one. For example, a smaller variant of a card game, an abstracted version of a security game, or an earlier version of a system for which both logs and a simulator exist. The predictor is trained once there and then applied with no interaction with the target game. Finally, the offline grid search method can be used fully offline, but it is not guaranteed to perform well in all fully offline settings.

W2 & Q2: More recent baselines.

Response: We thank the reviewer for this suggestion. We want to clarify why we chose BAIL and MOPO, and why we expect more recent offline RL methods to face the same limitation.

  • Choice of baselines. BAIL and MOPO represent the two main families of offline RL: model-free and model-based methods. These two families correspond to the two components of BOMB, behavior cloning and the model-based method. Fig. 6 therefore compares, within each family, methods that optimize a single player’s return with our approach that targets an equilibrium.

  • The limitation lies in the objective, not in the specific algorithm. Single-agent offline RL methods, including more recent ones, aim to maximize each player’s return against the opponents’ behavior recorded in the data. This tends to move each player’s strategy away from the equilibrium mixing toward a near-deterministic strategy, which can be exploited by an adapting opponent and increases NashConv. This limitation comes from the objective rather than from the particular algorithm, so we expect it to persist for more recent offline RL methods as well.

Q3: The claim that BOMB computes both NE and CCE.

Response: We thank the reviewer for pointing this out. In the CCE experiments, we did not include BC or the full BOMB framework because the offline datasets are generated by independent strategies of each player rather than a joint (correlated) strategy. BC imitates these independent strategies. It could still produce a candidate CCE, since any profile of independent strategies is a valid candidate (Remark D.14), but it cannot learn correlations between players, which is what distinguishes a CCE from an NE. We therefore focused the CCE experiments on MB-JPSRO, which can compute joint strategies. Accordingly, we have revised the abstract to state that BOMB computes an approximate NE, and its model-based component computes approximate CCE in multi-player games.

Dear Reviewer 8uzU,

Thank you again for your valuable review. As the discussion period is coming to an end, we would like to kindly check whether our responses have addressed your concerns. In particular, following your comments, we added results using a fully offline method to select α\alpha (offline grid search) in the revision, which we hope addresses your main concern. We would be happy to provide any further clarification if needed.

Best regards, The Authors

Reviewer_ntrbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies learning equilibrium strategies in extensive-form games from a fixed dataset. It introduces datasets and an evaluation protocol, and proposes BOMB, which combines behavior cloning with equilibrium finding in a learned game model. It also studies conditions on the data under which these components can recover equilibria.

Critical strengths
  • The controlled experiments across different data compositions and sizes are useful for understanding how behavior quality and coverage affect offline equilibrium finding.
Critical weaknesses

Offline EF is harder than offline RL, and the difference lies in the solution concept: offline RL seeks an optimal policy, whereas offline EF seeks an equilibrium.

This sentence in the abstract names the different objectives but does not explain why one problem is harder. The following explanation also oversimplifies offline RL: evaluating actions or policies that are poorly represented in the dataset is a difficulty there as well, and comparing observed actions alone does not establish policy optimality. The authors should explain what additional difficulty arises from strategic interaction, such as the coverage needed to evaluate unilateral deviations against a candidate strategy profile, and specify the setting in which its comparison holds. This argument appears in both the abstract and the introduction.

The rest of my concerns are outlined below:

  1. The data are generated using standard procedures in existing OpenSpiel environments, and the authors acknowledge that the games are relatively small-scale. A fixed dataset can provide a common evaluation protocol, but the paper does not establish a substantial data-access or simulation-cost advantage for these games. The offline restriction can still be useful for studying incomplete coverage; the contribution should therefore be supported by new methodological or theoretical insight rather than the availability or size of the logged data alone.
  2. Section 4.1 selects the mixing weight by minimizing the true-game equilibrium gap, and Section 5 adopts this grid search as the default for the remaining experiments. This gives useful evidence about the combination's potential, but does not establish the same performance when the learner has only the fixed dataset. The learned predictor is evaluated separately and requires true-game labels from a source game. Theorem 4.6(ii) also permits selection in the learned game, but assumes uniform coverage and a uniform expected-utility error bound.
  3. Theorem 4.6(i) provides limited insight into the benefit of combining behavior cloning and the model-based method. The search includes α=1\alpha=1, which uses behavior cloning alone, and α=0\alpha=0, which uses the model-based method alone. Selecting the weight with the smallest true-game equilibrium gap therefore automatically performs at least as well as either component. This guarantee does not explain when combining the policies improves on using the better component alone.
Questions for authors
  1. What makes offline equilibrium finding harder than offline RL beyond the difference in their objectives? In particular, how do the data coverage requirements for evaluating unilateral deviations differ from the coverage problems already present in offline RL?
  2. What new insight does the dataset suite provide beyond standard rollouts from the existing environments? How do the experiments support the motivation based on costly or unavailable simulators?
  3. Could the authors report the main comparisons using mixing weights selected entirely offline, including selection in the learned game, and distinguish these results from selection using true-game gaps?
  4. Under what conditions does combining the two policies improve on using the better component alone? Please clarify what Theorem 4.6(i) establishes beyond selecting the best weight from a grid that includes both individual components.
4 comments
AuthorsOpenReview ↗

Response to Reviewer ntrb (1/2)

We thank the reviewer for the valuable review and provide the following responses to your concerns.

Q1: Why offline EF is harder than offline RL.

Response: We thank the reviewer for this comment. We agree that offline RL also faces the difficulty of evaluating actions and policies that are poorly represented in the data. The additional difficulty in offline EF comes from the data coverage needed to verify an equilibrium, which is stronger than in offline RL.

For offline RL, if the dataset covers an optimal policy, i.e., the states and actions it visits, this is typically sufficient for offline learning. The target policy does not depend on any other agent. However, for offline EF, covering the equilibrium itself is not sufficient. This argument already appears in Appendix D.2. In Theorem D.7, we construct two games with different equilibria and a dataset that covers the equilibria of both games but is consistent with either game, so no algorithm can distinguish them from the data. To verify that a strategy profile σ∗\sigma^* is an equilibrium, one needs to know the payoff of each player’s unilateral deviation, i.e., what happens when one player deviates from σ∗\sigma^* while the others keep playing σ∗\sigma^*. This corresponds to the unilateral coverage condition studied by Cui & Du (2022). Furthermore, in Theorem D.10, we also show that for the model-based approach, even unilateral coverage may be insufficient. We have rewritten the corresponding sentence in the abstract, the introduction, and Section 2.3 in the revision.

W1 & Q2: Contribution of the datasets and the motivation of costly simulators

Response: We thank the reviewer for this comment. We agree that the value of our datasets does not lie in their size or in the difficulty of obtaining them, since they are generated by standard procedures in existing OpenSpiel games.

  • The dataset benchmark serves as a controlled testbed. First, the dataset can systematically vary the data composition along the trade-off between data quality and coverage, from random to expert data and their hybrids. Real-world logs usually come from a single, unknown behavior policy and do not allow such a systematic study. Second, because these games are small, the strategies learned offline can be evaluated in the true game, so we can measure how far an offline method is from an equilibrium. This is why we chose small games: not because they are costly to simulate, but because they make this evaluation possible. We have added this discussion in Section 3.3.
  • The motivation of costly simulators. We agree that the experiments do not show a simulation-cost advantage in these games. The costly-simulator motivation refers to the real-world settings that offline EF ultimately targets. Our experiments instead use the offline restriction, as the reviewer suggests, to study the effect of incomplete coverage in a setting where the results can be verified.
  • Insights from the datasets. By varying the data composition, the datasets reveal findings that a single set of rollouts would not show. BC and MB fail under different data compositions, i.e., BC degrades as the proportion of random data increases, while MB tends to improve (Fig. 8). These findings are consistent with our theoretical analysis, which shows that MB and BC require different coverage conditions (Theorems 4.4 and 4.5). Together, the empirical and theoretical results motivate combining the two methods in BOMB.

W2 & Q3: Selecting the mixing weight fully offline

Response: We thank the reviewer for this comment. We agree that grid search uses the equilibrium gap in the true game, which is not available when the learner has only the fixed dataset. Following the reviewer’s suggestion, we have run the experiments with α\alpha selected entirely offline in the learned game (Theorem 4.6(ii)), which uses only the offline dataset.

  • Comparison with other selection methods (Fig. 7). We have updated Fig. 7 to include the offline grid search method (i.e., selecting α\alpha in the learned game). As the proportion of random data increases, its performance approaches that of grid search, especially on 3p Kuhn poker. When the proportion equals one, i.e., the fully random dataset, these two methods perform very similarly. This is consistent with Theorem 4.6 (ii), which shows that under uniform coverage, the equilibrium gap of the weight selected in the learned game is at most 4nδ4n\delta larger than that of the weight selected by grid search.
AuthorsOpenReview ↗

Response to Reviewer ntrb (2/2)

  • Results on more games and dataset sizes (Appendix F.2, Figs. 20–22). We further report results of the offline grid search method on 2- and 3-player Kuhn poker and Leduc poker with different dataset sizes. From these figures, we can find that when the data contain at least 20% random data, the offline grid search method closely matches grid search in most settings, and on Leduc poker, it outperforms both BC and MB in most data compositions. The weights selected by offline grid search also follow the same trend as those of grid search, as shown in Fig. 22, i.e., the weight of BC decreases as the proportion of random data increases. This indicates that the offline grid search method captures the complementarity between BC and MB.
  • Limitation on near-expert data. When the proportion of random data is at most 10%, the offline grid search method is less reliable. Near-expert data cover only a small part of the game, so the learned game is inaccurate in the uncovered parts, and evaluating strategies using it becomes unreliable. As the proportion of random data increases, the learned game becomes closer to the true game, and the offline evaluation becomes more reliable.

In summary, offline grid search is a fully offline alternative that performs close to grid search whenever the data have reasonable coverage, while near-expert data remain challenging. We acknowledge that it is difficult for a single fully offline selection method to work well in all cases. This is why we proposed several methods for selecting α\alpha, which require different information and therefore apply to different scenarios. Grid search applies when a small number of policy evaluations in the target game is possible, as in offline-to-online settings. It is the most reliable method. The learning-based method applies when a simulator exists for a related or simplified game, but not for the deployed one. For example, a smaller variant of a card game, an abstracted version of a security game, or an earlier version of a system for which both logs and a simulator exist. The predictor is trained once there and then applied with no interaction with the target game. Finally, the offline grid search method can be used fully offline, but it is not guaranteed to perform well in all fully offline settings.

W3 & Q4: What Theorem 4.6(i) establishes, and when combining the two policies helps.

Response: We thank the reviewer for this comment. We agree that Theorem 4.6(i) by itself follows directly from the fact that the grid includes $\alpha = 0 $ and α=1\alpha = 1, and that it does not explain when mixing improves on the better component. As noted in Remark D.22, intermediate values of α\alpha carry no general guarantee, since the equilibrium gap is not necessarily convex in α\alpha. We clarify the role of the theorem and the conditions under which mixing helps below.

  • The main role of part (i) is to connect Theorems 4.4 and 4.5. Theorem 4.4 gives a guarantee for MB under uniform coverage, and Theorem 4.5 gives a guarantee for BC under equilibrium coverage. Part (i) shows that BOMB inherits a guarantee whenever either condition holds. Since the learner usually does not know which condition holds for a given dataset, this property is important in practice. In addition, part (ii) shows that the weight can be selected entirely within the learned game, without interacting with the true game, at the cost of at most 4nδ4n\delta under uniform coverage.
  • Intuitively, mixing is most likely to help when the dataset is neither close to an equilibrium nor fully covering the game, e.g., a hybrid of expert and random data. In this case, neither BC nor MB is reliable everywhere, and a mixture can combine the strengths of the two. Although we do not have a theoretical result about when mixing improves on both components, we can provide empirical evidence. Since the grid always includes α=0\alpha = 0 and α=1\alpha = 1, an interior selected α\alpha indicates that the mixture outperforms both components. As shown in Figs. 8(d) and 10, this occurs in most settings of Leduc poker and Liar’s Dice, where neither coverage condition is likely to hold fully. In contrast, in the smaller Kuhn poker games, one component tends to clearly dominate, and the selected α\alpha is close to an endpoint. Specifically, the 2- to 4-player Kuhn poker games have relatively few information sets, so even a small fraction of random data may cover most of the game tree. The learned model is then likely to be accurate, which favors MB, and α\alpha quickly drops to 0. In 5-player Kuhn poker, the trend appears to be reversed. The environment model is harder to learn with more players. MB may therefore perform poorly, and BC retains a relatively high weight in most settings (Fig. 10(c)). A precise theoretical characterization of when mixing strictly improves on both components is an interesting direction for future work.

Thank you for your responses. I believe the paper is now relatively clear in exposition. I raise my score to a 3.

Thank you for carefully reading our responses and for raising your score. We appreciate your comments, which have helped us improve the clarity of the paper. Thank you again for your constructive feedback throughout the review process.

Reviewer_JHXNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the problem of offline equilibrium finding (offline EF): computing equilibrium strategies in EFGs with only fixed offline datasets. This is a natural problem setting, similar to offline reinforcement learning, but with the goal of finding game-theoretic strategic equilibria strategies instead of just single-player optimal strategies.

To this end, the authors develop datasets of 8 common games, propose a method (BOMB) for offline EF, and show experimental results.

Their method, BOMB, is simply a combination of two simple methods: behavior cloning on the offline dataset (BC), and training a world-model from the offline dataset and then doing equilibrium finding in it (e.g. CFR or PSRO) (MB). BOMB simply uses both methods and mixes between them according to some constant.

Critical strengths

The paper's greatest strength is that this problem is indeed very natural and yet heretofore not investigated from the game theoretic angle.

The methods tested (behavior cloning, model-based equilibrium-finding, and BOMB [their combination]) are simple and the results are elucidating.

Critical weaknesses

Perhaps the biggest contributions of this work are defining the problem and showing that it is hard. I say that they show that it is hard because the experimental results (Section 5), to my eye, don't look great. For example, in Figure 8b, we see that no method has a better NashConv in Leduc poker than ~2.

Further, for most results, the best result is achieved by behavioral cloning on expert data, although BOMB is able to somewhat maintain performance as the expert data is diluted by random data. However, as this is the first work that examines offline EF for these games, there is no obvious better method, and it is a useful scientific contribution to show that these simple methods don't work well (i.e. the problem is hard).

The issue with this paper is that there's not too much there: the results are weak, and the methods aren't much either. The BOMB method is not nothing, but it's barely anything: mixing between the two methods is already questionably useful (the results look like they are often just a min of the other two methods, although not always!), and the method is even more silly because the mixing constant can't be derived a priori. The authors do alleviate this with their "learning method", although it's not clear what realistic setting admits such a model.

Second, the novelty of the problem setting is hard to ascertain. It's true that the existing offline equilibrium finding works (Cui & Du, 2022; Zhong et al., 2022) are theoretical and concern Markov games instead of EFGs. But what about offline MARL? The paper mentions StarCraft II Unplugged, but not any other Offline MARL works (even in the appendix, as far as I can see). While from what I can see from a cursory search, it's true that offline MARL research comes from a different community that uses different terminology and often doesn't explicitly aim for equilibria, the underlying problem and techniques are similar/the same. Some literature review of this field would be welcomed.

Surely the bound on BC (behavior cloning) in Theorem 4.5 is not really good, right? There should probably be some discussion of that in the paper, as otherwise I don't see the purpose of having this in the paper (it feels trivial and pointless).

In summary, the paper is borderline. Despite not-so-great results and somewhat simple methods, the paper does genuinely introduce a new problem (offline equilibrium finding) that is sure to garner further study. Although this problem (or similar ones) has been studied previously, it's either from the MARL research community instead of the game theoretic one (so they have different game benchmarks, may not explicitly measure NashConv), or it's theoretic instead of practical (and on Markov games instead of EFGs). Therefore, I weakly recommend acceptance, although I am on the fence.

Questions for authors

The paper notes that there has been previous research on offline equilibrium finding, but that the works (Cui & Du, 2022; Zhong et al., 2022) are theoretical instead of empirical and on Markov games instead of EFGs. Both are true, but the authors go on to say that "it does not address the imperfect-information structure of EFGs" -- does this paper take advantage of such a structure?

In Figure 6 (comparison with offline RL), I'm surprised that BOMB outperforms Offline RL consistently even in the expert offline data setting (0 proportion random data). In this case, as Figure 8 shows, BOMB reduces to BC, which is very naive offline RL. Why, then, do MOPO and BAIL perform worse in this regime?

4 comments
AuthorsOpenReview ↗

Response to Reviewer JHXN (1/2)

We thank the reviewer for recognizing offline EF as a new problem that is likely to attract further study. We provide the following responses to your concerns.

W1: The issue with this paper is that there's not too much there: the results are weak, BOMB is barely anything, the mixing constant can’t be derived a priori, and it is not clear what realistic setting admits the learned predictor.

Response: We thank the reviewer for the comment.

  • Contributions beyond BOMB. BOMB is only one part of this paper. The first contribution of this paper is the formulation of the offline EF problem and the construction of a suite of offline datasets for it, which we believe will benefit future research in this direction. Second, we analyze the dataset coverage required for the offline EF problem. We show that covering the equilibrium alone is insufficient, and we identify the coverage condition under which each component provably converges. Third, we propose BOMB. Our analysis shows that BC and MB fail under different data conditions, and BOMB combines them so that the learner does not need to know in advance which condition holds. We intended BOMB as a simple, solver-agnostic first baseline for the offline EF problem rather than a final solution. Although simple, it is a practical baseline: any online equilibrium-finding algorithm, such as CFR, PSRO, or JPSRO, can be plugged into it.
  • The mixing parameter α\alpha is not always an endpoint. As Fig.8(d) and Fig.10 show, α\alpha equals 0 or 1 mainly in the Kuhn poker variants. In Leduc poker and Liar’s Dice, the selected α\alpha lies strictly between 0 and 1 in most settings. A possible explanation for the endpoint values in Kuhn poker is that one component tends to clearly dominate in these games. The 2- to 4-player Kuhn poker games have relatively few information sets, so even a small fraction of random data may cover most of the game tree. The learned model is then likely to be accurate, which favors MB, and α\alpha quickly drops to 0. In 5-player Kuhn poker, the trend appears to be reversed. The environment model is harder to learn with more players. MB may therefore perform poorly, and BC retains a relatively high weight in most settings (Fig. 10(c)).
  • Selecting the mixing weight without interacting with the target game. We propose several methods for estimating α\alpha. Grid search requires evaluating candidate policies in the target game, so we use it only as a reference. The learning-based method needs no interaction with the target game. It targets a realistic situation: a simulator exists for a related or simplified game, but not for the deployed one. For example, a smaller variant of a card game, an abstracted version of a security game, or an earlier version of a system for which both logs and a simulator exist. The predictor is trained once there and then applied with no interaction with the target game. Our experiments instantiate this by training on 2-player Kuhn poker and transferring to 3-player Kuhn poker and to 2-player Leduc poker. The weaker transfer to Leduc in Fig. 7 indicates that its performance depends on the similarity between the source and target games. In the revision, we also added a fully offline method, offline grid search method, which selects α\alpha in the learned game (see Section 4.1 for details and Appendix F.2 for the results).

W2: Novelty of the problem setting and the offline MARL literature.

Response: We thank the reviewer for this suggestion and agree that a review of the offline MARL literature was missing. We have added a related work subsection on offline MARL in Appendix B.5, and we summarize the main points below.

  • Most offline MARL methods consider cooperative settings in which all agents share a team reward. They are typically evaluated on cooperative benchmarks such as SMAC and multi-agent MuJoCo. Their goal is to maximize the collective return, rather than to find strategies that are robust against strategic opponents.
  • We agree with the reviewer that the techniques largely overlap. BOMB itself builds on behavior cloning and model-based learning, two standard tools in offline RL and offline MARL. The main difference lies in the solution concept and its evaluation. Offline EF aims for an equilibrium and evaluates it by the equilibrium gap. This requires reasoning about each player’s unilateral deviations. To our knowledge, practical algorithms and benchmarks for this problem in imperfect-information EFGs have received relatively little attention, and our work aims to provide a starting point for this direction.
AuthorsOpenReview ↗

Response to Reviewer JHXN (2/2)

W3: The bound on BC in Theorem 4.5.

Response: We thank the reviewer for this comment. We agree that the bound in Theorem 4.5 is a worst-case bound. Its purpose is not to give a tight guarantee, but to serve two roles in our analysis.

  • It complements Theorem 4.4. Theorem 4.5 shows that BC works well when the data are generated by a near-equilibrium policy. Theorem 4.4 shows the opposite for MB: it works well when the data cover the whole game, regardless of how good the data-generating policy is. Together, the two theorems show that BC and MB succeed under different data conditions, which motivates combining them in BOMB.
  • It is needed for Theorem 4.6. Theorem 4.6 states that BOMB is at least as good as the better of its two components. To turn this into a guarantee for BOMB, we need a guarantee for each component. Theorem 4.4 provides it for MB, and Theorem 4.5 provides it for BC. As a result, BOMB inherits a guarantee whenever either condition holds: uniform coverage (via Theorem 4.4) or equilibrium coverage (via Theorem 4.5).

We have added discussions after Theorems 4.5 and 4.6 that clarify these two roles in the revised version.

Q1: Does this paper take advantage of the imperfect-information structure of EFGs?

Response: We thank the reviewer for this question. BOMB does not use the imperfect-information structure to make learning more efficient, but it is designed to handle this structure correctly. The results of Cui & Du (2022) are developed for Markov games, where states are fully observed, and policies are defined over states. They therefore do not directly apply to imperfect-information EFGs, where a player cannot distinguish histories within an information set. We have clarified this in the introduction. The following explains how we handle the imperfect-information structure.

  • Each data point in our offline dataset records the information set of every player separately, together with the game-level information outside the players’ control, the acting player, and the legal actions (Section 3.1). This keeps track of what each player can and cannot observe. For example, in Fig. 2, the two states of Player 2 belong to the same information set, which the dataset preserves, while the game-level information still allows the underlying histories to be told apart.
  • The strategy is computed over information sets. BC is trained only on the acting player’s information set and action. The learned strategy assigns the same action distribution to all histories within an information set. MB learns a model of the game and computes the equilibrium with solvers designed for imperfect-information EFGs. Furthermore, our analysis is stated for imperfect-information EFGs with perfect recall.

Q2: Why do MOPO and BAIL perform worse than BOMB on expert data?

Response: We thank the reviewer for this question.

  • On expert data, BOMB reduces to BC, but the problem does not reduce to offline RL. When the dataset contains only equilibrium data, the selected α\alpha is close to 1, so the performance of BOMB reduces to that of BC. However, this does not mean that the problem becomes an offline RL problem. The goal is still to compute an equilibrium, i.e., a strategy profile in which no player can gain by deviating, rather than to optimize the strategy of one player independently.
  • BC imitates, while offline RL improves each player independently. BC simply imitates the behavior policy in the data. On expert data, it therefore recovers a strategy close to the equilibrium. In contrast, offline RL methods such as MOPO and BAIL try to improve each player's strategy independently, treating the other players' behavior in the data as fixed.
  • Independent improvement hurts NashConv. Against an opponent playing an equilibrium strategy, many actions may yield the same expected return, so maximizing return independently gives no guidance on how to mix among them. Offline RL therefore tends to move away from the equilibrium mixing toward a near-deterministic strategy. Such a strategy can be exploited by an opponent who adapts to it, which increases NashConv even though the data are generated by an equilibrium.

Thank you for the responses. I still maintain W1. For W2, I believe your changes have improved the paper. I still maintain W3. For Q1 and Q2, your responses helped me understand.

I will maintain my score of weak acceptance (3), although I believe it is still borderline to a weak reject (2). I believe the quality of this work is still not quite at the bar for ICLR, but just introducing the problem and showing that it is hard are important contributions that should be published.

Note: I believe improving the work would require the creation of an actually good method, or significant demonstration that the non-method contributions (suite of offline datasets) are non-trivial and significant contributions.

Thank you for carefully reading our responses and for maintaining your positive score. We appreciate your recognition that introducing the offline EF problem and showing its difficulty are important contributions worth publishing. We agree that developing a stronger method is an important next step, and we regard BOMB as a first baseline that future methods can improve upon using our benchmark and analysis. In the final version, we will further clarify the limitations you raised. Thank you again for your constructive comments throughout the whole review process.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers and AC

We thank all reviewers for their constructive feedback. We have uploaded a revised version of the paper, with all changes highlighted in blue. The main revisions are summarized below.

  • Fully offline selection of the mixing weight. We added the offline grid search method, a parameter estimation method that selects α\alpha in the learned game using only the offline dataset (Section 4.1). We compare it with the other estimation methods in Fig. 7 and RQ2, and further evaluate it on two- and three-player Kuhn poker and Leduc poker with two dataset sizes each (Appendix F.2, Figs. 20–22).
  • Discussion of the theoretical results. We added discussions after Theorems 4.5 and 4.6. They clarify that the bound for BC is mainly qualitative, how Theorems 4.4 and 4.5 complement each other, what Theorem 4.6 establishes, and when mixing BC and MB is likely to help.
  • Related work. We added a review of offline multi-agent RL and offline learning in Markov games in Appendix B.5.
  • Clarification of inaccurate statements. We rewrote several statements that were imprecise in the original version. These include the argument for why offline EF is harder than offline RL, the role of the datasets as a controlled testbed, and the claim about CCE computation.
AuthorsOpenReview ↗

Summary of Discussion and Contributions

We sincerely thank all reviewers for their thoughtful reviews and engagement during the discussion period. To facilitate the following reviewer–AC discussion, we briefly summarize the outcomes of the discussion and the contributions of our work.

  • Significance of the problem. The reviewers recognize the value of studying offline equilibrium finding. Reviewer JHXN noted that the paper genuinely introduces a new problem that is likely to garner further study, and that “just introducing the problem and showing that it is hard are important contributions that should be published.”

  • Concerns addressed during the discussion. Following the reviewers’ suggestions, we added offline grid search, a fully offline method for selecting the mixing weight (Reviewers ntrb and 8uzU). We also clarified why offline EF is harder than offline RL and the role of our theoretical results (Reviewer ntrb), and added a review of offline MARL (Reviewer JHXN). Reviewer JHXN found that the added review of offline MARL “has improved the paper,” and Reviewer ntrb found the revised paper “now relatively clear in exposition.”

  • Summary of contributions. Our work introduces the offline EF problem with a suite of datasets that serve as a controlled testbed, characterizes the data coverage under which behavior cloning and the model-based method provably converge, and provides BOMB as a simple, solver-agnostic first baseline. We hope it will provide a useful foundation for future research on offline equilibrium finding.

Thank you all again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.