- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether the equilibrium search method can be scaled to competitive Pokémon VGC. To do this, they first propose a Rust battle engine, PokaiEngine, to enumerate a joint action’s full weighted outcome distribution in one pass. Then, they propose PokaiTrainer, which adapts Student of Games to the competitive Pokémon setting and trains it by self-play over hundreds of human teams. Finally, they conduct experiments, and the results show that on the live Showdown ladder, it wins 59% of 150 best-of-three sets against a human field averaging ∼1320 Elo, holds a 1350–1400 Elo band, and at its peak reached 1492 Elo, entering the format’s top 500.
Critical strengths
The main contribution for this paper is the PokaiEngine, which can enumerate the full weighted outcome distribution of a joint action in one pass. The engine makes the used method feasible since it can also enumerate all chance nodes.
Critical weaknesses
The method seems to be incremental, based on Student of Game and special for Pokémon, and the paper provides no evidence that its adaptations generalize, or even that they are necessary in Pokémon. Specifically, there is no comparison with the original Student of Game or ReBeL to demonstrate that these adaptations are necessary.
This paper is hard to understand, and the presentation of this paper could be improved. This limits the paper’s accessibility to the broader audience.
Questions for authors
-
To assess the benefit of the proposed adaptations, could the authors provide the results of the vanilla Student of Games or explain why the baseline is missing?
-
For the experiments on the live Showdown ladder, was any other baseline agent evaluated under the same conditions? With only testing PokaiTrainer, it is difficult to determine the performance attribution.
4 comments
Thanks again for the review. We've uploaded the revised PDF, and much of it targets the presentation issues you raised. §4 now opens with a table defining the formulation and a diagram of one turn decision, Table 3 shows where we depart from SoG and ReBeL, and the terminology is more consistent throughout. We'd really like to know whether it reads more easily now, and which parts are still hard to follow. Happy to answer anything else before the discussion closes.
Thanks for the authors' response.
These responses partially addressed my concerns. The remaining concern is whether the adaptations can generalize to other games. If not, this paper is likely to adapt SoG to the specific Pokemon game. Furthermore, the authors agree that the training-side changes aren't shown to be necessary. Therefore, I would maintain my score for the current version.
For the presentation, the main issue is the Pokemon game terminology. Some VGC terms are used without explanation, which may influence understanding of the game. Additionally, Figure 1 is still too abstract to understand.
Thanks for the reply and for the constructive feedback on the presentation! We've uploaded a revision that adds a glossary of the VGC terms we use, including how the stats work (App. F), and made Figure 1 more concrete, with HP numbers for one example attack and a caption that explains each chance event. Hopefully that helps.
On generalization, we agree the adaptations are shaped by VGC, and we see this mainly as an empirical paper. It shows that SoG-style equilibrium search scales to a game where chance follows nearly every action, with hundreds of outcomes per joint action every turn. In SoG's own domains chance appears only in poker's card deals, which come between betting rounds, so a subgame can stop before a deal instead of branching on its outcomes. A VGC subgame branches after every joint action inside it, so the outcomes compound with depth. The piece we expect to carry over is the enumeration engine, for games where chance follows most actions and its outcomes are public, such as turn-based RPG combat.
Thanks for the review.
Q1. Neither vanilla SoG nor vanilla ReBeL runs on VGC as is, which is why there's no such row. Run on a standard Pokémon simulator, they'd have to sample chance outcomes, at ~500× the cost of our exact enumeration, and solve full joint-action menus of ~10⁴ cells per node. We did swap in some of their components one at a time. SoG's node-count expansion matches our strength at 19% more search time. ReBeL's exhaustive fixed-depth subgame wins only 43% against our expansion, and its fictitious-play solver trails CFR. These comparisons will move to the main text. We agree the training-side changes aren't shown to be necessary, and the revision presents them as design choices rather than contributions. On generalization, we agree the adaptations are specific to VGC, and our reply to F2N8 (Q1) covers what we think could carry over.
Q2. We didn't. No other VGC agent had reported a ladder rating when we ran ours, and we chose not to add more bots, since a 75-set run measures win rate only to about ±11 points, and we wanted to keep our footprint on the public ladder small. Attribution comes from controlled comparisons instead, against earlier training rounds, heuristic search, a PPO policy, the behavior-cloned parent, and a non-equilibrium solver.
On the presentation, other than the internal run names, which parts were hardest to follow? We'd love to fix them if possible.