acceptodds
Under review as a conference paper at ICLR 2027

What Counts as Evidence for a World Model? Reassessing Othello-GPT

Abstract

Othello-GPT, a transformer trained to predict legal moves in Othello, has become the standard example of a neural network that uses a world model. The evidence falls into three categories: near-perfect prediction of legal moves, near-perfect linear decoding of the board state, and highly successful causal interventions. We argue that this evidence is ambiguous once we adopt the right baseline and the right metrics. Our baseline, which we call Othello-MLP, uses heuristics rather than a world model, and scores nearly as well on all three criteria. We then introduce an additional piece of evidence—transfer learning speeds on geometrically coherent versus incoherent extensions of Othello—which further suggests that Othello-GPT is using a collection of heuristics. We conclude that the total evidence is at best ambiguous, and if anything favors a heuristic interpretation. We draw four general lessons. Two concern how to test for world models: (1) when attributing a world model, the right baseline is a network that uses heuristics rather than a network with randomized weights; and (2) the right metrics take into account patterns of error and the full output distribution. The other two lessons are about how to weigh the different kinds of evidence: (3) causal interventions can provide illusory evidence; and (4) transfer learning can be more revealing than task performance, decoding accuracy, and causal interventions.

Primary area:
interpretability and explainable AI
Keywords:
world models, interpretability, probing, causal interventions, transfer learning, Othello-GPT, heuristics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hxW3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether existing tests show that Othello-GPT uses a world model. It compares Othello-GPT with heuristic MLP baselines, measures how interventions change predictions, and tests learning on extended boards. The baselines reproduce several results previously taken as evidence of a world model. The authors conclude that the evidence is ambiguous and suggest that a heuristic explanation is more likely.

Critical strengths
  1. The work provides a concrete reason to question commonly used evidence for world models. Its heuristic baselines achieve high prediction and decoding accuracy despite lacking complete board information. This is useful even without identifying Othello-GPT's actual mechanism.
  2. The analysis goes beyond aggregate accuracy by examining error patterns and the size of intervention effects. The coherent/incoherent board extensions provide an additional test, with several useful controls in Appendix I. These experiments add evidence specific to an influential case.
Critical weaknesses
  1. The evidence for preferring a heuristic explanation is weaker than the evidence for questioning existing tests. Similar errors could also arise from an imperfect board model. Even the best learned MLPs reach 98.9% legal-move accuracy and 93.3% decoding accuracy, compared with 99.9% and 99.6% for Othello-GPT (Section 5.5). The authors acknowledge that a world model remains possible, but the stronger preference for heuristics needs evidence that distinguishes these explanations.

  2. The added value of the new diagnostics needs stronger validation. The transfer test assumes that a model using board geometry should learn the coherent extension faster. Appendix I carefully matches rule counts and lengths, relevant training examples, and evaluation conditions. However, it does not show that a model known to use board geometry would pass this test. Equal learning speeds could reflect difficulty learning new input/output positions. Likewise, weak intervention effects could reflect an incomplete edit rather than an absent board model.
    [1], [2] already examine whether good predictions hide failures in world-model recovery and whether adaptation reflects the underlying world structure. The heuristic baseline, detailed intervention analysis, and coherent/incoherent board extensions are concrete additions. Their significance depends on explaining what these diagnostics can distinguish beyond earlier tests.

References:
[1] Evaluating the World Model Implicit in a Generative Model. NeurIPS 2024. https://arxiv.org/abs/2406.03689
[2] What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models. ICML 2025. https://proceedings.mlr.press/v267/vafa25a.html

Questions for authors
  1. Why do you think heuristics explain the results better than a board model that sometimes makes mistakes? Does Othello-GPT's higher accuracy than the learned MLP baselines affect this interpretation?
  2. Have you tested a model known to use board geometry on the same coherent and incoherent extensions? Could difficulty learning the added board positions explain why Othello-GPT learns both at similar speeds?
  3. What does the larger-board test reveal beyond [2]? How does it relate to the strong synthetic Othello results in [1]? Are the tests measuring different properties, and if so, which ones?
6 comments
AuthorsOpenReview ↗

Response to Reviewer hxW3 (1/2)

We thank the reviewer for their constructive feedback. We address the weaknesses here and the questions in a second comment. The controls for the transfer test that the reviewer asked for are in a separate comment to Reviewers zwxF and hxW3. We have added new figures (Figure 6, page 10), and we will revise the text in the ways we describe.

Similar errors could also arise from an imperfect board model. Even the best learned MLPs reach 98.9% legal-move accuracy and 93.3% decoding accuracy, compared with 99.9% and 99.4% for Othello-GPT (Section 5.5).

We agree that similar errors could arise from an imperfect world model. But we believe they could also arise from heuristics, and that the evidence is ambiguous. The reviewer is right that there are remaining gaps in performance and decoding accuracy. But we do not believe that these gaps favor the world model hypothesis. Othello-MLP and the learned MLPs are a lower bound on what heuristics can achieve, not an upper bound. They are one-layer networks over a fixed set of conjunctions, scoring each pattern independently, and in some cases discarding move order. Othello-GPT's eight layers and 64 attention heads allow heuristics that none of them can express: composing one pattern with the result of another, conditioning on when a move was played rather than only on whether it was, and scoring all sixty moves against one another. None of this requires a representation of the board.

Moreover, we think there are two considerations in favor of the heuristics hypothesis. First, the transfer test is the kind of evidence that can distinguish the two hypotheses, and it favors heuristics (Figure 6a–d; see the comment on the controls). Second, the burden of proof lies with those making the positive claim, especially when that claim attributes a more sophisticated ability.

However, it does not show that a model known to use board geometry would pass this test. Equal learning speeds could reflect difficulty learning new input/output positions.

We ran both controls, and we describe them in the separate comment to Reviewers zwxF and hxW3. In brief, two CNNs that represent the board as a grid learn the coherent extension much faster: the hand-coded CNN needs about 200 times as many new-square moves to reach an AUC of 0.90 on the incoherent extension as on the coherent one (Figure 6b), and the ResNet-18 transfers to the coherent extension before any fine-tuning (Figure 6c). Othello-GPT's lead is at most 0.06 AUC (Figure 6a), the same size as that of Othello-MLP, which has no grid representation (Figure 6d).

To test whether difficulty learning new positions explains this, we gave eight existing squares new token ids, so that Othello-GPT had to learn new input and output representations for squares whose rules it already knows. It learned them after about 40,000 new-square moves. By then the shared cost has been paid, yet the lead does not grow (0.06 at 40,000 new-square moves, 0.04 at 400,000). So the small lead is not explained by the cost of learning new positions. It is what we expect from a network that is not using the board's geometry.

Likewise, weak intervention effects could reflect an incomplete edit rather than an absent board model.

We agree that a weak intervention effect does not by itself show that there is no board model. Our claim is that weak intervention effects are ambiguous. However, the same evidence was used in previous work for the existence of a linear world model. Under a linear world model, we would expect to see a much larger causal effect. We also varied the intervention size from 1 to 8 times the calibrated size (Figure 6h). At twice the calibrated size, the probability on the newly legal move rises only to about 4%, far short of the 12.4% such moves receive in positions reached by ordinary play. Larger interventions bring it closer, but only by changing the probe's reading of other squares. So a more complete edit of one square comes at the cost of corrupting others.

AuthorsOpenReview ↗

Response to Reviewer hxW3 (2/2)

Why do you think heuristics explain the results better than a board model that sometimes makes mistakes? Does Othello-GPT's higher accuracy than the learned MLP baselines affect this interpretation?

We agree that Othello-GPT might have an inaccurate representation of the board. It might also have an accurate representation and use imperfect heuristics to predict moves from it. Our argument is that the evidence is ambiguous between these possibilities and heuristics over the move history.

Othello-GPT is more accurate at predicting legal moves, but the difference could be due to the factors we discuss in the first comment: the information about move order in the input, the number of layers, and the aggregation rule. None of these factors requires a board representation.

Have you tested a model known to use board geometry on the same coherent and incoherent extensions? Could difficulty learning the added board positions explain why Othello-GPT learns both at similar speeds?

Yes (Figure 6a–d). See the separate comment on the controls, and our answer in the first comment.

What does the larger-board test reveal beyond [2]? How does it relate to the strong synthetic Othello results in [1]? Are the tests measuring different properties, and if so, which ones?

[1] considers whether Othello-GPT's predictions depend only on the current board, and not on the particular move history used to reach it. Othello-GPT passes this test when trained on synthetic data. But, regardless of how a board is reached, the same moves are legal, so any network that predicts legal moves with high accuracy will predict nearly the same moves. Othello-MLP demonstrates that a network can predict legal moves with 98% accuracy without using a world model. Therefore, passing this test is ambiguous evidence of a world model.

[2] is closer to our transfer test. Like our test, their test asks how a network behaves when it learns something new. They differ in what the network learns. In [2] it is a new task on the same game: the network is fine-tuned, many separate times, to predict random functions of the board. For us it is the same task on a new game: the network still predicts legal moves, but on an extended board. Our approach has two advantages. First, the task stays the same: legal-move prediction. In [2], the network has to learn a new kind of output. That may require large changes to the network, and those changes could corrupt its representation of the board. So a network with a world model might still do poorly on their test. Second, because the coherent and incoherent extensions differ only in their geometry, our test is specifically about structural relations. In [2], a network only has to tell whether two move histories result in the same board. A network could pass by tracking the contents of each square, with no representation of how the squares are arranged.

So the three tests target different properties. [1] tests whether a network represents the board independently of the move history used to reach it. [2] tests whether a network represents the contents of every square. Ours tests whether a network represents the arrangement of the squares into a grid. In the terms of our Section 2, [2] tests completeness and ours tests structural relations. We will add this comparison to the paper.

We thank the reviewer again for their helpful comments. Their review prompted three new analyses (positive and negative controls for the transfer test (Figure 6a–d), and a sweep of intervention sizes (Figure 6h)) and a comparison of our approach with [1] and [2] that we will add to the paper. We hope these address their concerns and that the reviewer will reconsider their assessment. We would be glad to answer any further questions.

Thank you for the additional analyses. I have three clarification requests based on the existing experiments:

  1. Given the corrected coherent advantage, why do the transfer results still favor heuristics over an imperfect board model?
  2. What board-state and grid information is supplied to the CNNs, and how does this affect the interpretation of their comparison with Othello-GPT?
  3. Please acknowledge that Vafa et al. (2024) test both compression and distinction using subsequent move sequences. Othello-MLP’s next-move legality alone does not establish that it would pass these tests, so the corresponding claim should be qualified.
Reviewer_mEKMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper resassesses the evidence about the claim of Othello-world model theory. The authors include a heuristic comparing model, Othello-MLP, with flanking patterns where each hidden unit detects an interpretable conjunction of move features, with a weighted vote as the model’s output. Despite its relatively simple design, Othello-MLP achieves strong legal-move prediction accuracy and exhibits several behavioral and representational signatures similar to those observed in Othello-GPT. The authors further perform investigations regarding Othello transfer to extended versions. This paper challenges the widely discussed assumption that Othello-GPT's strong move-prediction performance necessarily indicates that it has learned an internal world model.

Critical strengths
  • The paper brings out a epistemic critique, I like the idea to discuss that a network with one can perform poorly if its model oversimplifies the task, and a network without one can perform well given the right heuristics. Thuis is an interesting perspective to challenge the common assumption. Since most previous Othello-related works assume strong predictive performance necessarily implies the emergence of a world model. Few focus on whether similar performance and experimental signatures could arise from alternative mechanisms that do not require a unified board-state representation.

  • The idea of training Othello-MLP as a decision tree for flanking pattern prediction is interesting. This could provide a valuable perspective on how strong predictive performance can emerge without an explicit, unified world model.

  • The authors perform comparison between Othello-GPT and Othello-MLP from several different perspectives, including predictive performance, probing results, causal interventions, etc. This could be very helpful in understanding the strengths and limitations of existing evidence for gaming world models, as well as motivating more evaluation methods for future research.

Critical weaknesses
  • My biggest concern is that, whether Othello-MLP and Othello-GPT are directly comparable? Even though the formulation of Othello-MLP is interesting, it did abandon some very important data (e.g. the move order). Therefore, I would find Othello-MLP a convincing counterexample, rather than a direct explanation of Othello-GPT’s internal mechanisms. Specifically, the author claimed in Line 241, that 'It fails structural relations, since nothing represents how the squares or patterns are arranged relative to one another.' I doubt this claim bc the 960 pattern is contructed using human-defined spatial relationships such as direction, length. This seems a contradiction to what the author claimed and appears to conflate "lacking an explicit representation of spatial relations" with "lacking spatial structural information."

  • The conclusion of section 5.4 is based on a 'perfect' casual intervention. However, in practice, the observed intervention effects may depend substantially on the intervention method itself. For example, board information could be distributed across multiple non-linear directions in the probe. It is unclear whether a successful intervention on a genuine world model would necessarily cause newly legal moves to receive probabilities comparable to those of legal moves in naturally occurring board states.

  • Another weakness of this paper is that, it seems the authors assume 'world model' and 'heuristic model' are two competing concept. However, these two concepts are not 100% mutually exclusive. That said, Othello-GPT could encode an internal representation of the board state while simultaneously relying on heuristic features or local pattern-matching mechanisms to predict legal moves. Therefore, completely ruling out the possibility that Othello-GPT learns or uses a world model would be an overstatement.

  • The paper challenges existing evidence for world models but provides limited guidance on how to establish stronger evidence. The authors raises several important concerns about characterizing Othello-gpt a 'world model', but offers limited guidance on what would constitute convincing positive evidence for a world model. Some discussion about an evaluation framework for determining a world model could make the contribution more constructive.

Questions for authors
  • In line 194, the authors claimed that 'there are 960 possible flanking patterns in Othello', can the authors clarify how the 960 flanking patterns are enumerated?

  • In Figure 3b, the authors choose the threshould as 25% to show the difference between played-on and flank. I'm curious whether such difference hold under other thresholds (e.g. 0.1, 0.5)?

  • The problem of Othello-GPT in terms of forfeit might also come from the fact that, forfeit could be rare in Othello games, meaning such traning data does not take a large portion in the othello-gpt training dataset. It's necessary for the author to discuss this and take a look at the training data before making a conclusion.

  • When comparing probing results, the dimensions of Othello-GPT (512) and Othello-MLP (~47,000) is very different. It would be worthwhile to provide additional analysis regarding the dimensionality control of the probes, regularization, etc.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mEKM (1/2)

We thank the reviewer for their constructive feedback. We address the main concerns here and the remaining questions in a second comment. We have added new figures (Figure 6, page 10), and we will revise the text in the ways we describe.

I would find Othello-MLP a convincing counterexample, rather than a direct explanation of Othello-GPT's internal mechanisms.

We agree that Othello-MLP is a convincing counterexample rather than an explanation of Othello-GPT's mechanisms. That is how we intend it. Our point is that a network without a world model can still pass the tests established by Li et al. and Nanda et al. That it discards move order makes the counterexample stronger rather than weaker: it is another reason to deny that Othello-MLP has a world model, since an order-free input cannot determine the board. Our conclusion is about the tests used to identify world models. We will say this explicitly in the revision.

I doubt this claim bc the 960 pattern is constructed using human-defined spatial relationships such as direction, length.

We agree that Othello-MLP represents spatial information. As we say in Section 2, each heuristic "looks for its own pattern, and while that might involve representing some squares and some of their relations, there is no single, unified representation that includes all squares and their arrangement into a grid." What Othello-MLP lacks is any representation of the relations between patterns. Each pattern score is read out by its own weights, and no score is used to compute any other. The only step that combines them aggregates over the 960 scores as an unstructured set. Nothing represents that the square in a row pattern is the same square in a diagonal pattern. An analogy: heuristics like "turn left at the river" and "turn right at the school" carry spatial information, but a collection of them is not a map, because nothing in it represents how the river and the school are arranged relative to one another. We will make this explicit at line 241.

it seems the authors assume 'world model' and 'heuristic model' are two competing concept.

We agree that they are not mutually exclusive. Section 2 says that they lie on a continuum, since completeness and structural relations each admit of degree, and that a network might use heuristics for some moves and a world model for others. As the reviewer notes, a network might also represent the board and still rely on heuristics to predict moves from it. Our argument is that the evidence is ambiguous between all of these possibilities. We will address this possibility explicitly in the revision.

The paper challenges existing evidence for world models but provides limited guidance on how to establish stronger evidence.

We agree that this deserves more discussion, and we will expand it in the revision. Our positive proposal is transfer learning: a network with a world model should learn coherent extensions of Othello more easily than incoherent ones (Section 5). We have added controls (Figure 6a–d), described in our comment to Reviewers zwxF and hxW3. Networks that we argue meet our definition learn the coherent extension much faster (Figure 6b–c). Othello-MLP, which does not, shows only a small lead (Figure 6d), as does Othello-GPT (Figure 6a). We suggest transfer because rapid learning has long been evidence for structured representation in humans and animals (Tolman, 1948; Harlow, 1949), and because rapid, flexible generalization is a central challenge for natural and artificial learning systems (Lake et al., 2017).

forfeit could be rare in Othello games

We looked at the training data. Forfeits are common overall: 34.38% of games contain at least one. But 94.67% of them occur at moves 55–59. Our analysis covers moves 5–53, and only 1.4% of games contain a forfeit in that range. So we agree that the forfeits we analyze are rare, and that this explains why Othello-GPT makes errors after them. What is surprising is that the error rate keeps increasing on subsequent moves (Fig. 3d). A network that accurately reconstructed the board after a forfeit should not make more errors on later moves. We will make this clearer in the revision.

AuthorsOpenReview ↗

Response to Reviewer mEKM (2/2)

the observed intervention effects may depend substantially on the intervention method itself.

We agree that the observed effect could depend on the intervention method. We use a linear intervention because that is the one Nanda et al. offer as evidence for a world model, and because their claim is that the board is linearly represented. Non-linear methods have also been tried: Li et al. used non-linear probes, including for their interventions, and their interventions were less successful than Nanda et al.'s. We also find that a non-linear probe decodes the board no better than a linear one (the difference is under 0.1 points; see the probe controls below), so there is little sign of board information in non-linear directions that a linear intervention would miss. We also varied the intervention size from 1 to 8 times the calibrated size (Figure 6h). At twice the calibrated size, the probability on the newly legal move rises only to about 0.04, far short of the 12.4% such moves receive in positions reached by ordinary play. Larger interventions bring it closer, but only by changing the probe's reading of other squares. Still, other methods might succeed, which is one reason we say our evidence does not rule out a world model. But the influence of this line of work rests on the linear results, and we argue that they do not establish a world model.

It is unclear whether a successful intervention on a genuine world model would necessarily cause newly legal moves to receive probabilities comparable to those of legal moves in naturally occurring board states.

We agree that it is not a strict entailment. But it is a natural test. If Othello-GPT derives its move predictions from a representation of the board, it should make the same predictions whenever it represents the board in the same way. According to the probe, it represents the board the same way after the intervention as in the naturally occurring position. Yet the predictions differ by a factor of fifty (0.25% against 12.4%). We do not take this to refute the world-model hypothesis, but it counts against it.

can the authors clarify how the 960 flanking patterns are enumerated?

A flanking pattern is a square, a direction (one of eight) and a length n: along that direction, the square is followed by n consecutive opponent pieces and then one of the player's own (Fig. 2a). We count every combination that fits on the board: 60 playable squares (the four centre squares start occupied), 8 directions, and every n that stays on the board. This gives exactly 960. We will add the derivation to the paper.

I'm curious whether such difference hold under other thresholds (e.g. 0.1, 0.5)?

Yes. Figure 6g shows thresholds of 5%, 10%, 25% and 50% (solid lines). The absolute values change, but the pattern is the same at every threshold.

It would be worthwhile to provide additional analysis regarding the dimensionality control of the probes, regularization, etc.

We projected Othello-MLP's hidden layer onto its top 512 principal components, matching Othello-GPT's width, and retrained the same probe (Figure 6f). The board is decoded from the projected layer at 87.2%, against a majority-class baseline of 54.8%, and at 90.5% from the full 47,061-dimensional layer. So width beyond 512 dimensions accounts for about three points of accuracy, not the 32 points above baseline. Accuracy is flat for weight decay from 0 to 0.1 and loses only 0.2 points at 1. Neither probe overfits: training and held-out accuracy differ by 0.2 points for Othello-MLP and 0.04 for Othello-GPT. A non-linear probe changes accuracy by less than 0.1 points for either model. We will report these controls in a new appendix.

We thank the reviewer again for their helpful comments. Their review prompted three new analyses (the frequency of forfeits, the probe controls (Figure 6f), and Figure 3b at four thresholds (Figure 6g)) and several clarifications that we will make in the paper. We hope these address their concerns and that the reviewer will reconsider their assessment. We would be glad to answer any further questions.

Thank you for your clarification. After reviewing the response, I believe that substantial revisions are still needed before the paper can be accepted. In particular, the scope and main claims of the paper should be more clearly defined, and some analysis design requires a more detailed explanation. Therefore, I maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer mEKM (follow-up)

We thank the reviewer for their response. We appreciate their feedback and, if the paper is accepted, we will incorporate it into the final version. At the same time, we want to emphasize that our initial submission already discussed the weaknesses the reviewer detailed, and clarifying them further will not require any major modifications. For example, the reviewer notes that “completely ruling out the possibility that Othello-GPT learns or uses a world model would be an overstatement.” This is something we consistently emphasize in the paper, both in the abstract (“We conclude that the total evidence is at best ambiguous, and if anything favors a heuristic interpretation”) and in the limitations (“None of this evidence rules out a world model”). Similarly, the clarifications about the analysis design (e.g., on the 960 flanking patterns) require adding only a few sentences. Finally, the reviewer describes Othello-MLP as “a convincing counterexample, rather than a direct explanation of Othello-GPT’s internal mechanisms.” That is how we describe it, both in the abstract (“Our baseline, which we call Othello-MLP, uses heuristics rather than a world model, and scores nearly as well on all three criteria”) and in the introduction (“We build Othello-MLP, a network that satisfies neither of the two conditions and solves the task through a collection of heuristics”). It is a counterexample to the assumption that any network meeting these criteria has a world model. We will make this point more explicit.

Reviewer_zwxFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides evidence that Othello-GPT, which has been held as a canonical example of a world model in the literature, actually relies on heuristics. It does this through four primary arguments: a heuristic-based baseline can achieve strong performance on board state prediction, where metrics degrade is more consistent with a heuristic-based explanation, the heuristic baseline responds similarly to causal interventions as Othello-GPT, and transfer to a geometrically coherent and incoherent Othello variants.

Critical strengths

Overall the paper provided compelling evidence that strongly suggests that OthelloGPT relies on heuristics. I appreciated the simplicity and relevance of each piece of evidence.

Critical weaknesses

Many gaps are quite large though framed as small, e.g., the top-5 legality error rate is 90x larger for Othello-MLP than Othello-GPT, and these figures do not have confidence intervals. I was also not convinced by the transfer section, as the incoherent extension outperforms early on and there is no positive example of a model with a grid representation learning the coherent extension faster.

Questions for authors

I feel that the effect of the decoding accuracy in Figure 4a would be more compelling if it persisted when conditioning on the corresponding location having a token (i.e., you average across all positions that have a token at each location) since it would show that this effect extends beyond a heuristic that simply detects whether a token has been placed at that position. Could you test this? You say on line 443 that the transfer experiment uses "arbitrary squares". Could you define this more concretely (e.g., are the flipping patterns selected uniformly at random and independently)? Could you add confidence intervals for accuracy figures?

2 comments
AuthorsOpenReview ↗

Response to Reviewer zwxF (1/2)

We thank the reviewer for their constructive feedback. We address the concerns about accuracy here and the transfer experiment and remaining questions in a second comment. The controls for the transfer test that the reviewer asked for are in a separate comment to Reviewers zwxF and hxW3. We have added new figures (Figure 6, page 10), and we will revise the text in the ways we describe.

Many gaps are quite large though framed as small, e.g., the top-5 legality error rate is 90x larger for Othello-MLP than Othello-GPT, and these figures do not have confidence intervals.

We agree that the gaps are large for top-5 move prediction, especially when they are measured as ratios of error (0.06% vs. 9.08%, about 150x) rather than as differences in accuracy (99.94% vs. 90.92%). The same is true for board decoding, where the ratio of errors (0.6% vs. 9.5%, about 15x) is again much larger (99.4% vs. 90.5%). We will report error ratios explicitly and remove phrasings such as "nearly as well."

When evaluating these gaps, we think it's important to keep in mind the larger context of our argument. There are two possible hypotheses about Othello-GPT: either it primarily uses a representation of the board, or it primarily applies heuristics over the move history. The standard baseline, Othello-GPT with random weights, performs so poorly (Table 1) that the first hypothesis can seem forced. Othello-MLP shows that it is not. Even though it exclusively uses heuristics over the move history, it recovers most of the distance between the randomized network and Othello-GPT. Moreover, three additional factors could explain part of the remaining gap without requiring a world model:

  1. Information. Othello-MLP's input discards move order, and there are an average of ≥322 boards consistent with each input. This necessarily limits the accuracy of its legal-move predictions. It also limits its board decoding to 92.1% (Table 2). Othello-GPT's input preserves move order.
  2. Capacity. Othello-MLP has a single fixed hidden layer. Othello-GPT has 8 layers and 64 attention heads. Even among the trained one-layer MLPs, widening from 512 to 4,096 units cuts top-5 error from 9.1% to 6.6% (Table 3, one-hot encoding).
  3. Objective. Othello-MLP assigns a score to each of the 960 flip patterns, and each assignment is independent of the others. We then choose the move corresponding to the flip pattern with the highest score. Othello-GPT scores all 60 moves together, using softmax, and it is trained on random walks over legal moves, which rewards it for putting equal probability on every legal move. That is exactly what matters to top-K accuracy. So part of the gap is the aggregation rule, not the hidden layer. Switching Othello-MLP from max to noisy-OR, with nothing else changed, cuts top-5 error from 9.1% to 7.3% (Table 3).

More generally, Othello-MLP and the learned MLPs are a lower bound on what heuristics can achieve, not an upper bound. Othello-GPT's eight layers and 64 attention heads allow heuristics that none of them can express: composing one pattern with the result of another, conditioning on when a move was played rather than only on whether it was, and scoring all sixty moves against one another. None of this requires a representation of the board.

The reviewer also asks for confidence intervals. We computed 95% confidence intervals for Table 1 (below), quantifying variability over games, and we will add them for Tables 2–3 and Figs. 4a–b and 5c. For top-1, top-3, and top-5 legality, the half-widths are at most 0.08 percentage points for Othello-GPT and Othello-MLP; on the game-level rows they reach 0.8 points. No interval overlaps in a way that changes any comparison.

MetricOthello-GPTOthello-MLPRandomized
Top-1 legal (%)99.95 [99.94, 99.96]98.15 [98.09, 98.21]16.32 [16.19, 16.46]
Top-3 legal (%)99.95 [99.94, 99.96]94.57 [94.49, 94.64]16.44 [16.36, 16.53]
Top-5 legal (%)99.94 [99.94, 99.95]90.92 [90.85, 91.00]16.49 [16.42, 16.57]
Games >5% on one illegal (%)15.8 [15.0, 16.6]100.0 [100.0, 100.0]86.1 [85.3, 86.8]
Games >10% on one illegal (%)8.7 [8.1, 9.3]99.6 [99.4, 99.7]0.1 [0.1, 0.2]
Games >5% on all illegal (%)16.7 [15.9, 17.5]100.0 [100.0, 100.0]100.0 [100.0, 100.0]
Games >10% on all illegal (%)10.5 [9.8, 11.2]100.0 [100.0, 100.0]100.0 [100.0, 100.0]

Updated Table 1. Held-out synthetic games, moves 5–53 (N = 392,000 positions from 8,000 games). Randomized Othello-GPT is the mean of 3 initializations.

AuthorsOpenReview ↗

Response to Reviewer zwxF (2/2)

I was also not convinced by the transfer section, as the incoherent extension outperforms early on and there is no positive example of a model with a grid representation learning the coherent extension faster.

On closer examination, the early incoherent advantage turned out to be due to an artifact in how test positions were chosen. Each test position was the first position in a game in which a new square became legal. Because the incoherent rules involve squares scattered across the board, and the coherent rules involve squares on the edge, the first position in incoherent games was much earlier (median move 13) than in coherent games (median move 24). Because earlier positions are easier to learn, this resulted in an early incoherent advantage. We now randomly select a position in the held-out games in which a new square was legal, and the effect has disappeared.

We also decided that we were using the wrong measure. We were measuring how much probability was assigned to newly legal moves. But that rewards a network for uniformly assigning higher probability to the new moves regardless of whether they are legal or illegal. For each of the 8 new squares, we now use the probability the model assigns to that square to discriminate when it is legal from when it is illegal. We report the area under the ROC curve (AUC), where 0.5 is chance and 1.0 means the probability predicts legality perfectly, averaged over the 8 squares. Figure 6a shows both measures (first and third columns). The coherent extension now leads at every evaluation point, albeit slightly. At matched numbers of new-square moves, the AUC for coherent minus incoherent (mean over 3 seeds; seed-to-seed sd below 0.005) is +0.05 after 2,000 new-square moves, +0.06 after 16,000 to 40,000, +0.04 after 400,000, and +0.007 at the end of fine-tuning (0.994 vs 0.987).

Following the reviewer's suggestion, we ran the test on two CNNs that represent the board as a grid, and on Othello-MLP. We describe these controls in a separate comment to Reviewers zwxF and hxW3. In brief, the CNNs learn the coherent extension much faster: the hand-coded CNN needs about 200 times as many new-square moves to reach an AUC of 0.90 on the incoherent extension as on the coherent one (Figure 6b), and the ResNet-18 transfers to the coherent extension before any fine-tuning (Figure 6c). Othello-GPT's lead is at most 0.06 AUC (Figure 6a), the same size as Othello-MLP's (Figure 6d).

I feel that the effect of the decoding accuracy in Figure 4a would be more compelling if it persisted when conditioning on the corresponding location having a token.

Conditioning on occupancy alone lowers Othello-GPT's overall decoding accuracy from 99.36% to 98.65%, as expected since empty squares are easier to decode. However, it also introduces a confound, namely the move number. Edge and corner squares are typically occupied only late in the game, when decoding is worse overall (Fig. 4b), whereas central squares are occupied throughout. We therefore condition on both occupancy and move number. For each square, we compute decoding accuracy separately at each move number (over positions where the square is occupied), then average these per-move accuracies with equal weight. At layer 6, where we probe in the paper, Othello-GPT remains least accurate on central squares (Figure 6e). The one exception is the corners, which are decoded slightly less accurately than the adjacent edge squares (98.5% vs. 99.1%). This is a limitation of the probe: a linear probe trained only to read the colour of an occupied corner reaches 99.9% from layers 1–4 but only 98.6% from layers 6–8. In the final layers, the corner's colour is sometimes dropped, presumably because it is less relevant to next-move computation. The central pattern is considerably stronger at layer 4 (Figure 6e).

You say on line 443 that the transfer experiment uses "arbitrary squares". Could you define this more concretely (e.g., are the flipping patterns selected uniformly at random and independently)?

Yes. For each coherent flip pattern onto the preexisting board, the incoherent extension has one pattern of the same length. Its squares are drawn uniformly at random, without replacement, from the 60 original non-centre squares, so no square appears twice in one pattern, and each pattern is drawn independently of the others. We will make this explicit in the paper.

We thank the reviewer again for their helpful comments. Their review prompted four new analyses (controls for the transfer test, confidence intervals for Table 1, Figure 4a conditioned on occupancy and move number (Figure 6e), and a corrected analysis of the transfer experiment (Figure 6a–d)) and several changes that we will make in the paper. We hope these address their concerns and strengthen their assessment of the paper. We would be glad to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewers zwxF and hxW3: controls for the transfer test

Reviewers zwxF and hxW3 both asked whether a model known to use board geometry would learn the coherent extension faster than the incoherent one. We ran the transfer test on two such models as well as on Othello-MLP (see the new Figure 6a–d, page 10). We report per-square AUC: for each new square, how well the probability the network assigns to it separates positions where it is legal from positions where it is not (0.5 is chance; mean over the eight squares).

Models known to use board geometry. Both are CNNs. Both receive an explicit and accurate representation of the board as input and are trained to predict the next legal moves. We consider whether this representation makes them faster at learning the coherent extension.

The first is a CNN with hand-coded filters (Figure 6b). The input is represented in the same way as probes of Othello-GPT (64×3). It has 48 hand-coded 17×17 convolutional filters, one for each flank direction (8) and length (6). Each filter searches for a certain flip pattern at every square. By design, it predicts legal moves perfectly on the original board. For fine-tuning we added 80 untrained filters of the same size, because that is the number of flip patterns in the incoherent extension that don't preserve the board's geometry, and we left the 48 hand-coded filters trainable as well. The coherent extension is learned almost immediately: its AUC exceeds 0.90 within about 2,000 new-square moves and 0.98 after 10,000. The incoherent extension needs about 200 times as many, about 410,000 new-square moves to exceed 0.90, and ends near 0.95.

The second is a ResNet-18 pretrained on ImageNet (Figure 6c). The input is an image of the board. Nothing about Othello's rules is hand-coded. We freeze it through the third stage and train the final stage, with a readout that scores each square from its own feature vector. It transfers to the coherent extension before any fine-tuning (AUC 0.99). The incoherent extension starts at 0.83 and never exceeds 0.92.

In virtue of their filters, these networks represent a world model, namely the squares arranged into a grid. A filter for "two squares to the right" presupposes that the network already represents, for every pair of squares, whether they stand in that relation. The 48 filters cover eight directions at six lengths, and those relations, represented for every pair, are what it is for the squares to be arranged in a grid.

Othello-MLP (Figure 6d) does not represent the squares arranged into a grid. Each of its 960 outputs fires for one fixed set of squares, and nothing in the network relates one square to another. After fitting a decision tree per new flip pattern, we trained the new readouts combining them. The coherent extension reaches an AUC of 0.93 after 800 new-square moves and 0.97 after 120,000; the incoherent extension reaches 0.86 and 0.93. So a gap of this size (≤\leq.06) cannot be evidence of board geometry. The gap presumably reflects a difference between the two rule sets: the coherent patterns lie along edges and diagonals, which fill in a different order from the random patterns' squares.

Othello-GPT (Figure 6a) is like Othello-MLP and unlike the CNNs. Its lead on the coherent extension is at most 0.06 AUC at any matched amount of training, about the size of Othello-MLP's and far below the CNNs'. Unlike the CNNs, it has to acquire embeddings and output logits for the new squares, a shared cost of learning either extension. In principle, that cost could hide a lead, as Reviewer hxW3 suggests. We therefore gave eight existing squares new token ids, so that the model had to learn new input and output representations for squares whose rules it already knows. It exceeded 0.90 on those squares after about 40,000 new-square moves, for either of two choices of squares. By then the shared cost has been paid, yet the lead does not grow: it is 0.06 at 40,000 new-square moves and 0.04 at 400,000.

We conclude that Othello-GPT is probably not using the board's geometry. We will add these controls to Section 5 and revise Section 5.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.