- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper aims at analyzing policy embedding methods given different neural policy initializations and input formats (NN weights versus behavior versus decoder-only) on several downstream tasks - payoff against random opponent, payoff against another decoded policy, exploitability prediction, zero-shot best response, agent identification, and equilibrium search in latent space.The experiments show that several methods can create reasonable embeddings, much better than trivial baselines, but they do not show the performance being really strong in any of the tasks. In the key task of equilibrium finding in latent space, the embeddings allow reducing the exploitability from 2.6 to 0.7.
Critical strengths
- The paper studies an important question: how to configure proper embeddings depending on task is a necessary question to have answered in the field
- Good breadth of experiments – they correctly ask whether the representation is able to encode the strategically relevant behavior for a given task
- Important finding that raw parameter space similarity is a terrible proxy for strategic/behavioral similarity
- I like the set of tasks chosen for the embeddings that may become a standard benchmark
- The presented results show that embeddings that are more informative than trivial baselines can be found.
- The results on Kuhn poker show that there exists a game in which the learned embedding can be used for latent strategy search.
Critical weaknesses
- The paper shows very interesting initial exploration of options, but does not show any task where using the embeddings would actually be good for something. The paper needs a clearer positive use case demonstrating that the found strategy embeddings is practically useful. What is the real impact of 80% improvement in MSE in Tasks A-C? What is the impact of improving opponent prediction from 34% to 58%? I would like to see much more grounding of these numbers in comparison to SotA techniques.
- The method somewhat works on the tiny game of kuhn poker, but the result in Leduc is much weaker and the reasons are not properly analyzed.
- The whole paper is rather broad and does not analyze the achieved results in enough depth to provide their understanding in the main text. If there are some important additional results in the appendix, they should be moved to the main text or at least better advertised in the main text.
- Some important details are not clearly explained. For trajectory encoder, trajectories are generated against randomly sampled opponents -> what opponents and do they guarantee that all infosets are visited? If you have opponents which never make joint policies reach decision relevant spots, then BR evaluation or exploitability prediction information might not be in the compression.
- For the functional autoencoder, sampling distribution I is not defined. The authors need to clarify information-state sampling distribution and evaluate performance under alternative exploration distributions.
- The paper’s conclusion that “what an embedding is trained on matters more than its size” is not sufficiently supported. There is no sweep on dimensionality of embedding space or explanation as to size selected
- Is the choice of only using networks as policies supported? Not sure. Main experiments use random policies – not necessarily representative of low-dimensional strategic manifolds encountered with good policies.
- The novelty claims are a bit overstated, as quick searches show nearly all parts have been previously studied (Policy embeddings, trajectory-based policy encoders, contrastive agent representations, Transformer trajectory representations, and decodable population-conditioned policies)
Questions for authors
- What is the clearest practical take-away from this paper? What embedding should I use for what practical problem to improve upon the state of the art algorithms in some way?
- Why Grover et. al is not evaluated on Task F? Do its embeddings also support latent space optimization that can be compared against the other methods?
- Could the authors do a dimensionality sweep where appropriate to establish some general idea of the compression/performance frontier?