acceptodds
Withdrawn from ICLR 2027

Simple Neural Techniques for playing games of imperfect information

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Developing intelligent policies for partially observable environments is challenging, as it requires reasoning about uncertainty over hidden states, planning actions under uncertain beliefs, valuing information acquisition and concealment, and finding robust policies that cannot be easily exploited. Imperfect-information games provide a natural testbed for these capabilities. In this paper, we develop simple, general techniques for efficiently training neural policies for such games, focusing on Fog-of-war chess, a game that combines the strategic complexity of chess with the challenges of imperfect information. Inspired by methods for solving imperfect-information games based on Counterfactual Regret, we adapt these ideas to reinforcement learning with large neural networks. Our approach achieves competitive performance against the state-of-the-art search techniques while doing no inference time compute or search and only requiring only a fraction of the compute used by other regret-based neural policy learning methods.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, partial observability, nash equilibria, imperfect information

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_8oCoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using llm training techniques to train a chess bot. They start from next observation prediction (i.e., belief modeling), then do embedding learning from "Searchless Chess", and then use entropy regularized ppo on top of the per-trained model.

Critical strengths

I commend the authors for trying to use model-free RL on FoW chess.

Critical weaknesses

However, the win-rate is way too low for a method that requires among other thing, a pre-trained chess bot. There is lot's of text that should be put in the appendix. The introduction of RM is not particularly useful in this paper. Pre-training on text data (next token prediction) is done out of necessity, there is no reason to do it here when sampling optimal chess games is cheap. The authors could have done any number of things, including minimizing KL/boosting the stockfish action but instead decided to predict sub-optimal board games and not even the full state (i.e. train a belief model). These choices are really opaque to me and consequently make the paper uninformative.

Questions for authors

Can you use the classification of [1] for your next observation prediction/latent prediction task? Can you elaborate more on the tokenization scheme and how you came to chose it?

[1]Ni, Tianwei, et al. "Bridging state and history representations: Understanding self-predictive RL." International Conference on Learning Representations. Vol. 2024. 2024.

Reviewer_VpbVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a large transformer to play the imperfect information chess-variant Fog-of-War chess. Training consists of four different parts: next-observation pre-training on lichess games, latent state alignment with the pre-trained Searchless Chess model, training a an infoset sampler, and PPO-selfplay.

The resulting model is compared against the current state-of-the-art model Obscuro, GPT-6, and Claude Opus 5.5. It scores above 50% winrate against Opus, but only achieves 14-31% against Obscuro and Astra.

Critical strengths

The underlying motivation of the paper is sound: exploring more proficient searchless agents for usually search-intensive games, like chess, is an open question. Additionally, the training pipeline appears novel. It includes several different aspects, which make the paper unique and interesting. I applaud the general idea and the approach.

Critical weaknesses

Although I like the overall idea of the paper, I believe the current version to be half-baked in several aspects.

  • As mentioned previously, the training pipeline consists of four stages. However, the impact of these stages and the choices undertaken are not ablated in any way, making it impossible to identify key aspects. Ablations are desperately required.
  • Evaluation is sparse. The model is compared to one previous model, which it loses against convincingly, and two LLMs, one which it again loses to convincingly and one which it narrowly beats. The authors additionally report that the model has a ">80% win rate over >500 games" against humans of "varied ratings", but this is not further specified or analysed in any capacity.
  • I find that the paper frequently contradicts itself motivationally and that it makes unfounded claims. Some examples:
  1. The authors claim that Obscuro does not beat the proposed model by a large margin. However, even in the weakest setting, Obscuro wins 69% of games, which rises to 85% in the stronger settings.Similar claims are being made with being "competitive with the state-of-the-art".
  2. The authors criticise Obscuro for using perfect-information leaf evaluations, as this does not translate well to the imperfect-information setting. However, their own model uses perfect-information guidance for the latent alignment and for the self-play cutoffs.

Other criticism:

  • The title of the paper is largely uninformative
  • Section 2, which spans two pages, reads like an introduction to imperfect-information AI and too extensive and not required here. It touches on several things which are not related to the paper and never used later, such as regret matching. This space should have been used for clearer evaluations of the experiments.
  • The paper references a non-existent Appendix A.
  • Vchess(st,at)V_{chess}(s_t, a_t) is a action/Q-value, not a state value
Questions for authors

Questions:

  1. Why is Chess represented as language in a FEN-string, rather than using the common grid-based representations?
  2. How many games were used for Table 1?
  3. The order of PPO self-play and training the sampling model is unclear to me. The sampler is trained with the games generated by self-play, but the self-play is described as the last step and also requires the sampler. Are these trained jointly? Doesn't this introduce constant shift in the estimated states?
  4. Why is Figure 2 using only four games? It seems overconfident to draw any conclusion from this.
  5. You briefly mention that adding more test-time compute for sampling increases performance by 150 ELO. Why is this not evaluated further, as it seems like a promising find?

Experiments:

  1. Ablations of each training pipeline part, like pre-training on lichess games and aligning latent states.
  2. What is the impact of the sampler? If the sampler itself is proficient, how good is an agent using a perfect-information evaluator of the sampled states? Similarly, how good is the raw model with the sampler?
Reviewer_xWrfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a learning-based method for Fog-of-war chess training a transformer embedding the player information-set in self-play, capable of predicting the next observation and searching for unobserved opponent moves and then use RL to train in such a model. The trained policy is then evaluated via play against the SOTA search-based method and frontier LLMs. The authors show that the model is capable of outperforming CLAUDE Opus.

Critical strengths
  • To the best of my knowledge the work is novel, as I have not seen any approaches to try to deploy model-based RL on a large and heavily imperfect information game, such as Fog-of-war Chess.
  • The research direction is interesting and relevant. I think there is a lot of potential in latent modelling of the environment dynamics and/or the opponent.
  • The used method itself is clear and well explained
Critical weaknesses
  • The biggest issue I have with the paper is that the usage of the model-based techniques is not properly motivated. It was not clear to me why the technique was used instead of directly learning in the game simulator. The paper should clearly state what advantages this brings , and compare gains with respect to that metric in comparison to model-free baselines, such as MMD using the game simulator.
  • As this is a self-play method, it raises the question of overfitting to particular play and vulnerability to OOD when deployed against unseen opponents, which the paper does not test. I think it is a potential explanation for Obscuro managing to outperform the model so much with added search time. An evaluation against opponents deliberately seeking to exploit this fact, i.e., an approximate exploitability computation, would really help assess how much of a risk this poses.
  • The empirical results are underwhelming for a method tuned for a particular game. The model only marginally beats Opus, and convincingly loses to current SoTA, while being more expensive.
  • The claim that the method is more general than Obscuro appears to be untrue. True, the Stockfish value function is not used but instead the Searchless Chess pre-trained model, that is also tuned for perfect information chess, is used to train the transformer, by tuning both its value predictions and the final layer activations. To me, this seems like the same principle, just applied at train time rather than leaf evaluation. Also, Stockfish in Obscuro is only used as a node selection heuristic and initial policy estimate for PCFR, making the claim that the authors value function is “as the CFR equations require” feel somewhat overstated.
  • The claim that “all the baseline solutions require a lot more time, compute, and multiple CPU or GPU cores to play at their reported strengths” is true for the frontier LLMs , however for Obscuro I would not consider the up to 2s used per move on 3 CPU cores to be greater cost than the estimated 250 H100 GPU hours needed to train the proposed method.
  • The notation regarding actions is confusing. The authors use only the time subscript, but otherwise denote an action only as a, without differentiating which player played the action. And yet, right after, the counterfactual reach is defined as “opponent and chance reach probability of h”.
  • Adding onto the previous point, in section 2.1 it is very unclear which game formalism the authors use. It is referred to as “POMDP” and the aforementioned lack of differentiation of player actions would seem to imply a single agent environment. And yet, the authors also use the counterfactual reaches and say that “we use CFR counterfactual formulation”. Given that the paper is about Fog-of-war chess, I assume that the authors mean a multi-agent environment, probably formalized as EFG. But as is, it's incorrectly conflated with single-agent formalization.
  • The used policy learner is said to be PPO in the paper, whereas it was used both with entropy regularization, magnet policy regularization and magnet policy updates, corresponding much more closely to Magnetic mirror descent (MMD) with the “moving magnet” variant. It should be named MMD to prevent confusion.

Additional notes:

  • I really like the direction of the paper and would be very interested in seeing deeper analysis of what the model actually learns. I encourage the authors to continue to work on the paper and shift the focus from trying to outperform the current SoTA to answering the following questions: 1. What benefits does learning the model bring? 2. When does the model bring the greatest benefits? 3. What are the risks? When does such a model fail? I think that if these questions are answered, this paper could be turned into a strong work that I would be excited to read.
Questions for authors
  • What is the main advantage and motivation for using the model-based approach? Did you make a direct comparison to MMD? If the point is the sample complexity, can you show that using the model, you can train a stronger policy with the same number of environment steps?
  • Did you evaluate the robustness of the trained model against diverse opponents pushing the model out of distribution? Is that the reason why Obscuro is winning so much?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.