acceptodds
Under review as a conference paper at ICLR 2027

Towards Learning Representations of Policies in Two-Player Zero-Sum Imperfect-Information Games

Abstract

Search-based methods for two-player zero-sum imperfect-information games reason over public belief states, whose evolution depends on the players' policies. Scaling these methods therefore calls for compact representations of policies, not just of states. We study self-supervised learning of policy embeddings: low-dimensional vectors that summarize a policy's strategic behavior and, ideally, can be decoded back into a playable policy. We make three contributions. First, we introduce three ways to generate diverse policy populations. Second, we compare six representation methods: weight-space autoencoders, behavioral encoders trained with contrastive or imitation objectives, and the conditioning embeddings of a NeuPL population network. Third, we propose a suite of downstream tasks. They range from linear probes of payoff and exploitability to zero-shot best response and equilibrium search carried out directly in embedding space. On Kuhn and Leduc poker, behavioral encoders cut exploitability-prediction error by up to 87% while weight-based ones fail, and policy optimization in latent space improves exploitability on Leduc. Our results point to behavior-grounded, decodable embeddings as a building block for latent-space planning in imperfect-information games.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, reinforcement learning, game theory, two-player zero-sum games, imperfect information games, multi-agent reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LmUJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims at analyzing policy embedding methods given different neural policy initializations and input formats (NN weights versus behavior versus decoder-only) on several downstream tasks - payoff against random opponent, payoff against another decoded policy, exploitability prediction, zero-shot best response, agent identification, and equilibrium search in latent space.The experiments show that several methods can create reasonable embeddings, much better than trivial baselines, but they do not show the performance being really strong in any of the tasks. In the key task of equilibrium finding in latent space, the embeddings allow reducing the exploitability from 2.6 to 0.7.

Critical strengths
  • The paper studies an important question: how to configure proper embeddings depending on task is a necessary question to have answered in the field
  • Good breadth of experiments – they correctly ask whether the representation is able to encode the strategically relevant behavior for a given task
  • Important finding that raw parameter space similarity is a terrible proxy for strategic/behavioral similarity
  • I like the set of tasks chosen for the embeddings that may become a standard benchmark
  • The presented results show that embeddings that are more informative than trivial baselines can be found.
  • The results on Kuhn poker show that there exists a game in which the learned embedding can be used for latent strategy search.
Critical weaknesses
  • The paper shows very interesting initial exploration of options, but does not show any task where using the embeddings would actually be good for something. The paper needs a clearer positive use case demonstrating that the found strategy embeddings is practically useful. What is the real impact of 80% improvement in MSE in Tasks A-C? What is the impact of improving opponent prediction from 34% to 58%? I would like to see much more grounding of these numbers in comparison to SotA techniques.
  • The method somewhat works on the tiny game of kuhn poker, but the result in Leduc is much weaker and the reasons are not properly analyzed.
  • The whole paper is rather broad and does not analyze the achieved results in enough depth to provide their understanding in the main text. If there are some important additional results in the appendix, they should be moved to the main text or at least better advertised in the main text.
  • Some important details are not clearly explained. For trajectory encoder, trajectories are generated against randomly sampled opponents -> what opponents and do they guarantee that all infosets are visited? If you have opponents which never make joint policies reach decision relevant spots, then BR evaluation or exploitability prediction information might not be in the compression.
  • For the functional autoencoder, sampling distribution I is not defined. The authors need to clarify information-state sampling distribution and evaluate performance under alternative exploration distributions.
  • The paper’s conclusion that “what an embedding is trained on matters more than its size” is not sufficiently supported. There is no sweep on dimensionality of embedding space or explanation as to size selected
  • Is the choice of only using networks as policies supported? Not sure. Main experiments use random policies – not necessarily representative of low-dimensional strategic manifolds encountered with good policies.
  • The novelty claims are a bit overstated, as quick searches show nearly all parts have been previously studied (Policy embeddings, trajectory-based policy encoders, contrastive agent representations, Transformer trajectory representations, and decodable population-conditioned policies)
Questions for authors
  • What is the clearest practical take-away from this paper? What embedding should I use for what practical problem to improve upon the state of the art algorithms in some way?
  • Why Grover et. al is not evaluated on Task F? Do its embeddings also support latent space optimization that can be compared against the other methods?
  • Could the authors do a dimensionality sweep where appropriate to establish some general idea of the compression/performance frontier?
Reviewer_75pfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies policy embeddings for two-player zero-sum imperfect-information games. It compares six embedding methods (weight autoencoders, a functional autoencoder, a trajectory encoder, NeuPL conditioning embeddings, and a reimplementation of Grover et al. 2018) on downstream tasks including payoff prediction, exploitability prediction, agent identification, zero-shot best response, and equilibrium search in latent space. Experiments are on Kuhn Poker (12 information states) and Leduc Poker (936), with preliminary results on two larger games. The main finding is that behavior-based embeddings outperform weight-based ones.

Critical strengths
  1. Well-motivated problem. Policy embeddings are a genuine missing piece for latent-space planning in IIGs. The paper correctly identifies that existing work on belief-state representations (Solinas et al., 2025a,b) needs a complementary policy representation, and frames the problem clearly.
  2. The comparison reveals that different SSL objectives capture different information. The contrastive encoder excels at agent identification while Grover’s imitation-based objective better predicts pairwise payoffs. This decomposition is a useful empirical observation for future work on opponent modeling and game-solving.
Critical weaknesses
  1. No novel method. The trajectory encoder is SimCLR + Transformer applied to trajectories; the functional autoencoder doesn’t work; the rest are existing or trivial baselines. The evaluation tasks are standard probes or taken from prior work. The paper has no methodological contribution that advances the state of the art.
  2. The experiments are on trivially small games and the main finding is unsurprising. Kuhn Poker (12 info states) is a setting where tabular methods dominate without any learning. Showing that behavior-based representations beat weight-based ones on toy domains confirms a well-known intuition but provides no evidence that these approaches will matter at the scale the paper motivates.
Questions for authors
  1. What is the novel method in this paper? The trajectory encoder combines a Transformer with NT-Xent, both well-established. The functional autoencoder doesn’t work. Can you articulate what methodological advance this paper makes beyond applying existing techniques to a new domain?
Reviewer_tVnCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper empirically studies policy representation for populations of policies and how they can be used for downstream tasks. They evaluate multiple tasks on a mixture of Kuhn and Leduc poker, 1 dice 4 faces liar's dice and phantom tic-tac-toe over a dataset of policies generated in different ways.

Critical strengths

The paper feels like the start of an important empirical investigation,

Critical weaknesses

but ultimately falls short. The analysis falls apart in a few places,

First the games are way too small. Kuhn poker, Leduc poker, and the Liar's dice variant (1 die 4 faces) have too few terminal sequences. Probably the models have more parameters than these games have sequences. Compared to the games 12, under 1K, and around 1K terminal sequences in order, the policy networks has around 100K parameters. In the same vein, the encoding size (64) is too big compared to the games themselves, in particular for Kuhn poker. While PTTT is large game, it only "preliminary" (361) and only covers task A, B, and E. In the encoder-decoder pipeline, typical baselines such as variational auto-encoders are missing.

The choices of the paper for learning objective are not grounded in the self-supervised learning literature on POMDPs. For example [1] organizes representation learning objectives by what it should encode. The paper never establishes what they want the policy embedding to contain. In particular, reconstruction seems to play an auxiliary role in the tasks as the main objective seems to be predictive expected return against different policies, closely related to expected reward prediction (RP) in [1].

The paper doesn't ablate the mean pooling vs transformer architectural changes with the NT-Xent, and a triplet identification+BC hybrid loss changes. It doesn't explain why these losses are interesting yet makes claim such as "Taken together, these results reveal that different SSL objectives produce representations with different epistemic content."

The paper is taking methods that simply are not competitive (PSRO, see [2] for instance) or populations that are extremely unsmooth. A more reasonable alternative could have been equilibria (or iterates along the path to them) games under specification mismatch. This could have been perturbations to the game itself or PPO/MMD-style policies with regularization toward randomly drawn magnets rather than the uniform policy. Randomly initialized networks in particular should be extremely easy to identify on longer horizons, the chance of taking high probability (compared to population) actions goes up, providing more identification evidence. This is supported by the experiments.

The paper reports so many numbers yet so few insights, I cannot recommend it.

[1] Ni et al., Bridging State and History Representations: Understanding Self-Predictive RL (ICLR 2024). [2] Rudolph, Max, et al. Reevaluating policy gradient methods for imperfect-information games (ICLR 2026)

Questions for authors

Please clarify what exactly is the baseline column in the Table 2.

702: Can the variance in payoff just be attributed to the randomization of the bias of the head? If so doesn't this make the actions of the population extremely over different state?

Can the author include more exploitability curves for the Neupl/PSRO runs to show how they are converging over time?

Can the author talk about the reconstruction quality of the encoder-decoder methods?

If possible please always put captions above tables (e.g., table 9)

Why is Grover missing from Tasks D and F? Functionally nothing is preventing it to be included.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.