- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether existing tests show that Othello-GPT uses a world model. It compares Othello-GPT with heuristic MLP baselines, measures how interventions change predictions, and tests learning on extended boards. The baselines reproduce several results previously taken as evidence of a world model. The authors conclude that the evidence is ambiguous and suggest that a heuristic explanation is more likely.
Critical strengths
- The work provides a concrete reason to question commonly used evidence for world models. Its heuristic baselines achieve high prediction and decoding accuracy despite lacking complete board information. This is useful even without identifying Othello-GPT's actual mechanism.
- The analysis goes beyond aggregate accuracy by examining error patterns and the size of intervention effects. The coherent/incoherent board extensions provide an additional test, with several useful controls in Appendix I. These experiments add evidence specific to an influential case.
Critical weaknesses
-
The evidence for preferring a heuristic explanation is weaker than the evidence for questioning existing tests. Similar errors could also arise from an imperfect board model. Even the best learned MLPs reach 98.9% legal-move accuracy and 93.3% decoding accuracy, compared with 99.9% and 99.6% for Othello-GPT (Section 5.5). The authors acknowledge that a world model remains possible, but the stronger preference for heuristics needs evidence that distinguishes these explanations.
-
The added value of the new diagnostics needs stronger validation. The transfer test assumes that a model using board geometry should learn the coherent extension faster. Appendix I carefully matches rule counts and lengths, relevant training examples, and evaluation conditions. However, it does not show that a model known to use board geometry would pass this test. Equal learning speeds could reflect difficulty learning new input/output positions. Likewise, weak intervention effects could reflect an incomplete edit rather than an absent board model.
[1], [2] already examine whether good predictions hide failures in world-model recovery and whether adaptation reflects the underlying world structure. The heuristic baseline, detailed intervention analysis, and coherent/incoherent board extensions are concrete additions. Their significance depends on explaining what these diagnostics can distinguish beyond earlier tests.
References:
[1] Evaluating the World Model Implicit in a Generative Model. NeurIPS 2024. https://arxiv.org/abs/2406.03689
[2] What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models. ICML 2025. https://proceedings.mlr.press/v267/vafa25a.html
Questions for authors
- Why do you think heuristics explain the results better than a board model that sometimes makes mistakes? Does Othello-GPT's higher accuracy than the learned MLP baselines affect this interpretation?
- Have you tested a model known to use board geometry on the same coherent and incoherent extensions? Could difficulty learning the added board positions explain why Othello-GPT learns both at similar speeds?
- What does the larger-board test reveal beyond [2]? How does it relate to the strong synthetic Othello results in [1]? Are the tests measuring different properties, and if so, which ones?
6 comments
Response to Reviewer hxW3 (2/2)
Why do you think heuristics explain the results better than a board model that sometimes makes mistakes? Does Othello-GPT's higher accuracy than the learned MLP baselines affect this interpretation?
We agree that Othello-GPT might have an inaccurate representation of the board. It might also have an accurate representation and use imperfect heuristics to predict moves from it. Our argument is that the evidence is ambiguous between these possibilities and heuristics over the move history.
Othello-GPT is more accurate at predicting legal moves, but the difference could be due to the factors we discuss in the first comment: the information about move order in the input, the number of layers, and the aggregation rule. None of these factors requires a board representation.
Have you tested a model known to use board geometry on the same coherent and incoherent extensions? Could difficulty learning the added board positions explain why Othello-GPT learns both at similar speeds?
Yes (Figure 6a–d). See the separate comment on the controls, and our answer in the first comment.
What does the larger-board test reveal beyond [2]? How does it relate to the strong synthetic Othello results in [1]? Are the tests measuring different properties, and if so, which ones?
[1] considers whether Othello-GPT's predictions depend only on the current board, and not on the particular move history used to reach it. Othello-GPT passes this test when trained on synthetic data. But, regardless of how a board is reached, the same moves are legal, so any network that predicts legal moves with high accuracy will predict nearly the same moves. Othello-MLP demonstrates that a network can predict legal moves with 98% accuracy without using a world model. Therefore, passing this test is ambiguous evidence of a world model.
[2] is closer to our transfer test. Like our test, their test asks how a network behaves when it learns something new. They differ in what the network learns. In [2] it is a new task on the same game: the network is fine-tuned, many separate times, to predict random functions of the board. For us it is the same task on a new game: the network still predicts legal moves, but on an extended board. Our approach has two advantages. First, the task stays the same: legal-move prediction. In [2], the network has to learn a new kind of output. That may require large changes to the network, and those changes could corrupt its representation of the board. So a network with a world model might still do poorly on their test. Second, because the coherent and incoherent extensions differ only in their geometry, our test is specifically about structural relations. In [2], a network only has to tell whether two move histories result in the same board. A network could pass by tracking the contents of each square, with no representation of how the squares are arranged.
So the three tests target different properties. [1] tests whether a network represents the board independently of the move history used to reach it. [2] tests whether a network represents the contents of every square. Ours tests whether a network represents the arrangement of the squares into a grid. In the terms of our Section 2, [2] tests completeness and ours tests structural relations. We will add this comparison to the paper.
We thank the reviewer again for their helpful comments. Their review prompted three new analyses (positive and negative controls for the transfer test (Figure 6a–d), and a sweep of intervention sizes (Figure 6h)) and a comparison of our approach with [1] and [2] that we will add to the paper. We hope these address their concerns and that the reviewer will reconsider their assessment. We would be glad to answer any further questions.
Thank you for the additional analyses. I have three clarification requests based on the existing experiments:
- Given the corrected coherent advantage, why do the transfer results still favor heuristics over an imperfect board model?
- What board-state and grid information is supplied to the CNNs, and how does this affect the interpretation of their comparison with Othello-GPT?
- Please acknowledge that Vafa et al. (2024) test both compression and distinction using subsequent move sequences. Othello-MLP’s next-move legality alone does not establish that it would pass these tests, so the corresponding claim should be qualified.
Response to Reviewer hxW3 (1/2)
We thank the reviewer for their constructive feedback. We address the weaknesses here and the questions in a second comment. The controls for the transfer test that the reviewer asked for are in a separate comment to Reviewers zwxF and hxW3. We have added new figures (Figure 6, page 10), and we will revise the text in the ways we describe.
We agree that similar errors could arise from an imperfect world model. But we believe they could also arise from heuristics, and that the evidence is ambiguous. The reviewer is right that there are remaining gaps in performance and decoding accuracy. But we do not believe that these gaps favor the world model hypothesis. Othello-MLP and the learned MLPs are a lower bound on what heuristics can achieve, not an upper bound. They are one-layer networks over a fixed set of conjunctions, scoring each pattern independently, and in some cases discarding move order. Othello-GPT's eight layers and 64 attention heads allow heuristics that none of them can express: composing one pattern with the result of another, conditioning on when a move was played rather than only on whether it was, and scoring all sixty moves against one another. None of this requires a representation of the board.
Moreover, we think there are two considerations in favor of the heuristics hypothesis. First, the transfer test is the kind of evidence that can distinguish the two hypotheses, and it favors heuristics (Figure 6a–d; see the comment on the controls). Second, the burden of proof lies with those making the positive claim, especially when that claim attributes a more sophisticated ability.
We ran both controls, and we describe them in the separate comment to Reviewers zwxF and hxW3. In brief, two CNNs that represent the board as a grid learn the coherent extension much faster: the hand-coded CNN needs about 200 times as many new-square moves to reach an AUC of 0.90 on the incoherent extension as on the coherent one (Figure 6b), and the ResNet-18 transfers to the coherent extension before any fine-tuning (Figure 6c). Othello-GPT's lead is at most 0.06 AUC (Figure 6a), the same size as that of Othello-MLP, which has no grid representation (Figure 6d).
To test whether difficulty learning new positions explains this, we gave eight existing squares new token ids, so that Othello-GPT had to learn new input and output representations for squares whose rules it already knows. It learned them after about 40,000 new-square moves. By then the shared cost has been paid, yet the lead does not grow (0.06 at 40,000 new-square moves, 0.04 at 400,000). So the small lead is not explained by the cost of learning new positions. It is what we expect from a network that is not using the board's geometry.
We agree that a weak intervention effect does not by itself show that there is no board model. Our claim is that weak intervention effects are ambiguous. However, the same evidence was used in previous work for the existence of a linear world model. Under a linear world model, we would expect to see a much larger causal effect. We also varied the intervention size from 1 to 8 times the calibrated size (Figure 6h). At twice the calibrated size, the probability on the newly legal move rises only to about 4%, far short of the 12.4% such moves receive in positions reached by ordinary play. Larger interventions bring it closer, but only by changing the probe's reading of other squares. So a more complete edit of one square comes at the cost of corrupting others.