- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes using llm training techniques to train a chess bot. They start from next observation prediction (i.e., belief modeling), then do embedding learning from "Searchless Chess", and then use entropy regularized ppo on top of the per-trained model.
Critical strengths
I commend the authors for trying to use model-free RL on FoW chess.
Critical weaknesses
However, the win-rate is way too low for a method that requires among other thing, a pre-trained chess bot. There is lot's of text that should be put in the appendix. The introduction of RM is not particularly useful in this paper. Pre-training on text data (next token prediction) is done out of necessity, there is no reason to do it here when sampling optimal chess games is cheap. The authors could have done any number of things, including minimizing KL/boosting the stockfish action but instead decided to predict sub-optimal board games and not even the full state (i.e. train a belief model). These choices are really opaque to me and consequently make the paper uninformative.
Questions for authors
Can you use the classification of [1] for your next observation prediction/latent prediction task? Can you elaborate more on the tokenization scheme and how you came to chose it?
[1]Ni, Tianwei, et al. "Bridging state and history representations: Understanding self-predictive RL." International Conference on Learning Representations. Vol. 2024. 2024.