acceptodds
Under review as a conference paper at ICLR 2027

Playing Dumb: How a Chess Transformer Simulates Users Across Skill Levels

Abstract

Generative models increasingly simulate users at different skill levels in education and AI evaluation. Yet it remains unclear whether lower-skill behavior reflects less internal knowledge or a different use of retained knowledge. This distinction is difficult to study in natural language, where proficiency and relevant knowledge are hard to measure. In this work, we study Maia-2, a chess transformer that predicts human moves conditioned on player skill, using quantitative skill ratings and deterministically labeled chess concepts. We hold board positions and model weights fixed and analyze positions where skill conditioning changes the prediction from a blunder to the best move. First, we measure encoded knowledge by training probes to predict the presence of 172 chess concepts from Maia-2's internal states. Probe accuracy remains nearly constant across skill conditions, and probes trained at one level transfer accurately to the others. To test whether stronger decisions can be recovered from this retained knowledge, we fine-tune only Maia-2's final output layer, raising best-move accuracy in the lowest skill band from 14.4% to 82.8%. Training this layer only on highest-skill representations retains most of the improvement at lower levels, showing that recovery does not require direct training on lower-skill states. Steering concept-relevant features learned by a sparse autoencoder also improves lower-skill decisions while leaving probe-measured concept encoding practically unchanged. These results support an account in which skill conditioning changes how retained concept knowledge influences move selection, showing that weaker behavior can understate the knowledge available within a model.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, skill conditioning, chess, user simulation, capability-behavior gap

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EjFKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The authors defined 172 human-interpretable binary chess concepts and trained a set of linear probes on the output of Maia2 to predict the presence of these concepts at a given state
  • They construct a dataset of positions where the low-skill Maia2 model makes a blunder and does not address a particular concept, while the highest-skill Maia2 takes the optimal action and addresses this concept.
  • They showed that fine-tuning the Maia2 policy head output of the low-skill embedding on the best moves on this dataset leads to strong improvements in the low-skill model’s performance
  • They demonstrated a method for using an SAE to recover activations corresponding to each of the chess concepts, and show that amplifying these activations allows the low-skill model to improve performance at the relevant states
Critical strengths
  • I believe the feature steering section contains the strongest and most interesting results from this paper. The ability to improve low-skill performance by amplifying salient aspects of the board representation is a very promising way of understanding the weaknesses both of a model, and the humans of which it was trained.
Critical weaknesses
  • The construction of the transitional dataset used as a base for all experiments contains a few flaws
    • “93.6% of positions in the Blundered Transitional Dataset contain at least one oracle concept that lower-skill predictions fail to address correctly and higher-skill predictions resolve”, but this is not a useful statement by itself. Considering there are 172 different concepts, I would expect this to be true in high percentage of states just by chance
    • Along this line, having a control where the low-skill move addresses the concept but the high-skill move does not would be useful comparison for each of these results
  • Head-SFT does not seem especially compelling to me.
    • At a high level, it seems we are just taking a model which was trained on both low-skill and high-skill actions and conditioned on low-skill data. Then, by fine-tuning this model on high-skill actions in a subset of the state space, the model is able to perform better in this region of the state space.
    • This would be partially addressed by conducting the same experiment using Maia-1 (i.e. individually trained models where the low-skill condition hasn’t already been trained on high-skill actions)
  • The paper does not spend enough time exploring the feature steering results.
    • For example, we are missing any analysis on how feature steering efficacy varies across different concepts. I would assume that some concepts would lead to higher performance improvements than others, particularaly in states where multiple concepts are relevant. I would also expect that this would change at different skill levels, connecting to the discussion point about education
    • The comment that mean probe accuracy changes by only 0.86% does not seem like the right comparison to me, since probe presence can change significantly (e.g. from 51% to 99%) without changing the accuracy.
    • The fact that random-feature steering improves performace at all is quite surprising and deserves further analysis. At first glance, I would expect highlighting a section of the board which is unrelated to the best move would cause the model to make the best move less often if anything.
Questions for authors
  • What are the practical implications of the finding that low-skill-conditioned representations contain high-skill capabilities?
  • Could you provide additional results from your existing experiments to contetualize the results better?
    • How does probe value change, not just accuracy?
    • How does Head SFT and steering results vary based on the concept?
  • Why does random steering improve performance?
2 comments
AuthorsOpenReview ↗

Authors' Rebuttal

We thank the reviewer for the careful review.

W1 & W2: The 93.6% statistic and a reverse control

The 93.6% statistic only motivates the concept set in Section 4.1, and none of our conclusions rests on it. Whether a concept matters in a position, rather than being present by chance, is what the interventions test. If the targeted concepts were incidental, random SAE features would steer as well as concept features, and a head trained for one concept would work on the others too. Neither holds: random features add almost nothing, as W7 details, and Section 5.2 shows that a head trained for one concept loses much of its gain on the others.

We considered the reverse control, but it does not fit the question the paper asks. In chess, addressing a concept is not good in itself: when the best move ignores a threat, the threat is not what matters in that position, for example because a stronger counterattack exists. Our question is whether lower-skill play fails to use knowledge that the right decision requires, and a position where the right decision does not require the concept cannot answer it.

W3 & W4: Head SFT as fine-tuning on high-skill moves

The gain of Head SFT requires the concept knowledge retained in Maia-2's lower-skill representations. In Section 5.2, a head trained only at ≥2000 never sees a lower-skill representation yet transfers to <1200. To test whether this fine-tuning succeeds from any input, we applied the exact Head SFT recipe to other inputs, and Head SFT stays at least 30 points above both in every skill band:

Acc. (%)<1200
Head on CNN embedding40.1
Head on random-weight Maia-252.2
Head SFT82.8

As the reviewer suggested, we repeated the experiment on Maia-1, whose models are trained separately per rating band. Head SFT lifts its 1100 model, which has never seen high-skill actions, above the untouched 1900 model, while probe accuracy stays flat:

Maia-111001300150017001900
Probe acc.95.595.995.095.695.2
Frozen34.341.251.558.563.9
Head SFT72.774.678.180.581.6

W5 & Q2: Steering and Head SFT by concept and skill level

By concept type, steering helps most when the side to move has an under-defended piece. At <1200, it improves every such concept, by 34 points on average, against 20 points when the opponent's piece is under-defended. Across these concepts, steering improves 95% of them at <1200, and Head SFT improves every concept.

Table 1 reports both interventions per skill band, with the standard deviation across concepts as subscripts. Steering gains are largest at <1200 and shrink steadily with skill, while Head SFT stays high in every band. For education, this means the gap between a simulated player's moves and its retained knowledge is widest for the weakest players.

W6 & Q2: Probe values, not just accuracy

Probe values barely change, so probe-measured knowledge encoding stays close to its original values. Beyond the 0.86-point accuracy change in the paper, we read the probabilities of the pre-steering probes on the steered states at every skill level. Steering shifts the mean probe probability by only 1.3 points for present concepts and by 2.3 points for absent ones, far from a shift such as 51% to 99%.

W7 & Q3: Random steering does not help

Table 1 shows that random steering adds almost nothing: 1.9 points at <1200, at most 1.3 points in any higher band, and none at 1800–1999. Within <1200, what little it adds sits below 1100, where best-move accuracy starts at zero because every lowest-skill prediction is a blunder, so any perturbation can only help. Ignore Rate and Blunder Rate are likewise measured only where Maia-2 starts from a wrong move, which is why random steering lowers them to 83.8% and 47.3%. Concept features, in sharp contrast, raise accuracy at <1200 from 14.4% to 42.7% and lower the two rates to 48.6% and 32.1%.

Q1: Practical implications

The main implication is for AI safety and oversight: weaker behavior can coexist with retained knowledge, so evaluating a model under one conditioning context can understate what it can do. For education, matching novice behavior alone cannot show that a simulated learner models what a student knows. Section 6.1 discusses both implications in more detail.

We hope these responses clarify the points the paper left open to misreading, namely the role of the 93.6% statistic and what Head SFT is trained on, and that the head controls, the Maia-1 replication, the breakdown by concept and the probe probabilities address the remaining concerns. We have added all of them to the revised manuscript.

We greatly value the reviewer's feedback, which has made the paper stronger, and we would be glad to address any further questions or suggestions.

Thank you for the detailed rebuttal.

W1, W4, W6, and W7 are fully addressed.

W2 is partially addressed, I acknowledge the authors saying looking at the reverse control is out of scope of the paper, but I still feel this somewhat limits the understanding of the results and practical implications. E.g. if reverse control steering helps, then that might mean the model should focus on relevant concepts regardless of their correctness in order to improve play, by deciding when to take it into account or deliberately avoid it. If reverse control steering doesn’t help, this could mean the intervention is mostly about increasing preferences for concept-related actions.

W3 is the biggest open question here, and I am still not convinced that the approach here is especially insightful or compelling. I agree that pretraining a model on relevant domain data (low-skill or high-skill) allows for more effective fine-tuning over a random-weight model or a model not trained to solve this particular task (Maia-1 is not trained to make the best move). But this is not very surprising to me, it seems like the expected result that pre-training on a related task will improve finetuning the model on better actions.

W5 was slightly addressed, though I think a full addressal of this is not in the scope of a standard rebuttal. I would like to see a more through discussion of how the specific concepts improve play more than others at different skill levels, why these differences exist, and how those differences could be directly used in education or other human settings.

Overall, I appreciate the additional experiments and agree they strengthen the evidence for the findings. However my primary concern was the novelty and scientific insight of the findings, not the soundness, and I remain unconvinced that the Head SFT experiments demonstrate anything particularily surprising. The feature-steering results are more interesting but remain insufficiently investigated. Therefore, I will maintain my original rating.

Reviewer_6EwrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether Maia-2, a chess model conditioned on player Elo, makes weaker moves because it represents less knowledge or uses retained knowledge differently. Across skill conditions, chess concepts remain similarly decodable with linear probes. Fine-tuning only the policy head substantially improves low-Elo move accuracy, while steering concept-related SAE features provides smaller gains. The authors interpret these results as evidence that weaker behavior can reflect differences in knowledge use rather than knowledge availability, though the experiments do not fully separate this explanation from shared representations learned through training across Elo levels.

Critical strengths
  1. The author tries to use controlled settings for experiments
  2. More than probing, the author also uses SAE to affect output
Critical weaknesses
  1. The probes reach ~96% accuracy, but is this due to pretraining, and does it really show that the model learned these concepts? Without comparisons to a randomly initialized model or a simple classifier trained directly on the board, this is unclear. Since the concepts are deterministic functions of the board, the probe may simply extract them from preserved board information. High probe accuracy alone does not establish concept acquisition.
  2. The first four experiments show that representations across Elo conditions support similar concept decoding and can be trained to predict the same best move. But this does not distinguish “the model knows what to do but uses that knowledge differently” from “the model preserves board information that a supervised readout can learn to use.” The stronger claim about knowledge versus its use therefore seems under-supported.
  3. Since Maia-2 is jointly trained on games across Elo levels, features learned from high-Elo play may already be baked into its shared backbone. Low-Elo conditioning need not remove them, so successful probing may simply reflect this shared training setup. A model trained only on low-Elo games would help clarify what these results tell us about skill adaptation.
Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Authors' Rebuttal

We thank the reviewer for the review and address each point below.

W1: Is probe accuracy just preserved board information?

The probe accuracy comes from training and goes beyond preserved board information. Since all 172 concepts are functions of the board, we report the 21 tactical and safety concepts, such as forks, pins and king danger, where a board readout is weakest. On these, both requested baselines, trained with the paper's probe protocol, fall well short: random weights stay near chance, and a linear readout of the board planes is 17 points below Maia-2. We also probed the Elo-aware contribution, which is what the two Elo-aware transformer blocks add to the CNN embedding. All accuracies are averaged over the 11 skill levels:

Probe inputAcc.
Random-weight Maia-253.8
Raw board71.5
Elo-aware contribution88.9
Residual stream after Block 288.6

The board readout also stays near chance on some of these concepts. The Elo-aware contribution alone is on par with the full residual stream at every skill level, so the part of the computation that skill conditioning controls keeps these concepts. Over all 172 concepts, Maia-2 also exceeds the board classifier, and the Elo-aware contribution stays on par with the residual stream.

W2: Knowledge use, or a readout of preserved board information?

Lower-skill representations retain concept knowledge that Maia-2 does not externalize, beyond preserved board information. We tested this alternative directly by training the same supervised readout, with the exact Head SFT recipe, on preserved board information. With the same supervision, Head SFT stays at least 30 points above both readouts in every skill band. Best-move accuracy at <1200:

Acc. (%)<1200
Head on CNN embedding40.1
Head on random-weight Maia-252.2
Head SFT82.8

Moreover, a head trained only at ≥2000 never sees a lower-skill representation yet performs almost as well at <1200, and a head trained for one concept transfers poorly to the others. Finally, feature steering uses no trained readout at all, yet it raises best-move accuracy at <1200 from 14.4% to 42.7% while probe accuracy changes by under 1 point on average.

W3: A model trained only on low-Elo games

The separation between knowledge encoding and knowledge externalization does not depend on shared training. We ran this test with Maia-1, whose nine models, from 1100 to 1900, are each trained only on games from their own rating band. Probe accuracy stays flat even in the 1100 model, which never trained on high-rated play, and Head SFT lifts this model above the untouched 1900 model:

Maia-111001300150017001900
Probe acc.95.595.995.095.695.2
Frozen34.341.251.558.563.9
Head SFT72.774.678.180.581.6

Shared conditioning is also the setting that motivates the paper, namely one model conditioned to act at many skill levels, like an LLM prompted to simulate users.

We thank the reviewer again for these focused questions. The random-weight and raw-board probes, the head controls on board information and the Maia-1 replication respond to these three concerns, and we have added them to the revision, which also defines terms earlier and states H1 and H2 more plainly.

We greatly value the reviewer's feedback, which has made the paper stronger, and we would be glad to address any further questions or suggestions.

Reviewer_zcCBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Models that imitate users of a given skill raise a basic question: when the model plays a novice, does it know less, or does it know the same things and act on them differently? The paper studies this question in chess, where skill is measured by Elo ratings and many chess facts can be computed exactly from the board. It uses Maia-2, a published model that predicts the move a human of a given rating would play. The rating enters the model as a learned vector, so the same position can be processed under 11 skill levels with the same weights. The authors collect over one million positions where Maia-2 predicts a blunder at low ratings and the engine's best move at high ratings. To measure what the model knows, they train linear probes, simple classifiers that read the model's internal state and predict whether a chess fact holds on the board. A fact, which the paper calls a concept, is a yes or no property of the position that a program can check exactly. The authors use 172 facts, for example "a piece of the side to move on c2 is attacked more times than it is defended", or the presence of a fork, a pin or an open file. To change how the model uses what it knows, they retrain only the final layer that turns the internal state into a move (the policy head). Alternatively, they amplify internal features, found with a sparse autoencoder (SAE), that track one of these facts. The results are the following. (i) Probe accuracy is about 96% at every skill level, and a probe trained at one level works almost as well at the others. (ii) The authors then retrain only the policy head, keeping the rest of the model frozen. They collect positions where a fact is present, the best move responds to it, and the low-rating move ignores it. For example, a piece on c2 is under attack: the best move defends it or moves it away, while the low-rating move leaves it to be captured. Only 71 of the 172 facts have at least 500 such positions, the minimum the authors require to train and test a head, so the interventions use these 71 facts. For each of them, they train a separate copy of the head on its positions, teaching it to output the best move. When the model is set to the lowest ratings, it picks the best move in about 14% of these positions; with the retrained head, it does so in 83%. A head trained only on the internal states computed at the highest rating, then applied to lower ratings, still reaches 80%. A head trained for one fact, however, does much worse on positions built around other facts. (iii) The second intervention does not retrain anything. While the model processes one of these positions at a lower rating, the authors take the SAE features that best track the fact the position is built around, and make them stronger by multiplying their values. At the lowest ratings, the model then picks the best move in about 43% of positions, instead of 14%. Making randomly chosen SAE features stronger in the same way, as a control, gives only 16%. Probes trained before the intervention read the facts from the modified internal state with almost the same accuracy as before, with less than one point of difference. The authors conclude that the model keeps the same knowledge at every skill level, and that skill changes how this knowledge is turned into moves.

Critical strengths

The paper processes the same positions with the same Maia-2 weights and changes only the rating input, setting the rating of the player and of the opponent to the same value. Differences between skill levels therefore come from the rating alone, not from different boards or different training. The chess facts are computed by rules from the board, so their labels need no human annotation, and a chess engine (Stockfish) scores the quality of each move. Probes trained at one rating are also tested at the other ratings, which shows that stable accuracy does not depend on fitting a new probe at each rating. Heads trained for one fact are tested on positions built around other facts, which shows that the retrained head is not simply a better player on all positions. Heads trained only on states computed at the highest rating still reach 80% at the lowest ratings, so the gain does not require training on low-rating states. Steering with random SAE features gives almost no gain, so the effect of steering does not come from changing the internal state in any arbitrary way. Besides the share of best moves, the paper reports blunder rate and loss of winning chance. It is transparent in saying what its results do not show: that the policy head is where Maia-2 implements skill, or that language models work the same way.

Critical weaknesses
  • According to Section 3.1 and Appendix A, Maia-2 first encodes the board with a convolutional network that does not receive the rating. The rating enters only in the two transformer blocks, where it changes the attention. Each block adds its output to its input. So the state that the probes read is the sum of two parts: the board encoding, which is the same at every rating, and the changes made by the two blocks, which depend on the rating. The 172 facts are simple properties of the board, such as which pieces are attacked, and they may already be readable from the board encoding alone. If so, probe accuracy could stay high at every rating whatever the rating changes, and the result would say little about what the model knows at each rating. The paper does not report probes on the output of the convolutional network, before the rating enters, which would show whether this is the case.
  • The retrained policy head reads the same kind of state as the probes: the board encoding, which is the same at every rating, plus the changes that depend on the rating. Since it is trained to output the best move, it may learn to find that move from the board encoding alone. The paper does not train the same head on the output of the convolutional network, or on a model with random weights. Without these comparisons, the rise from 14% to 83% does not show that the gain comes from knowledge that the rating hides. It could also come from the board encoding, which does not depend on the rating.
  • The interventions show that low-rating states can be mapped to better moves. They do not show what the rating vector changes inside the model to produce the weaker moves, for example in the attention patterns or in specific directions of the internal state. The authors state that their interventions do not locate the original mechanism, yet the title promises to explain how the model simulates users of different skill.
  • All 172 facts describe the current board, such as attacks, forks, pins and open files. None of them measures multi-move planning or calculation, as the authors acknowledge (Appendix B.1 and Section 6.2). The claim that weaker play does not come from less knowledge is therefore tested only on this kind of knowledge, while the abstract states it in general terms.
  • The paper states that, to the authors' knowledge, it is the first to separate the knowledge a transformer encodes from the knowledge it uses. Closely related evidence already exists in chess. Karvonen (2024) trains a GPT model on chess game records and finds a linear direction for player skill in its internal state. When games start from random moves, the model plays like a weak player, and adding this skill direction raises its win rate against Stockfish by up to 2.6 times. This work is different from the Karvonen et al. (2024) study on sparse autoencoders that the paper cites. In language models, inference-time intervention recovers truthful answers that probes can read in the internal state but that the output does not give (Li et al., 2023). Probing work has also long distinguished information that a probe can read from information that the model uses (for example amnesic probing, Elazar et al., 2021). The paper discusses none of these works.
  • All experiments use a single model, Maia-2, in which the rating enters only in the attention of two transformer blocks. Other chess models represent skill differently. ALLIE (Zhang et al., 2024) is a decoder-only transformer trained on move sequences, closer to the language models that motivate the paper. Maia-3 (Monroe et al., 2026) attaches the ratings of both players to every board square at the input. Maia (McIlroy-Young et al., 2020) trains a separate model for each rating band. The Transcendence models (Zhang et al., NeurIPS 2024) are trained only on games of players below a rating threshold, from 1000 to 1500. In these last two cases, weaker models learn from weaker players' games rather than being conditioned on a low rating. Repeating at least the probe analysis on one of these models would show whether the conclusions depend on how Maia-2 represents skill.
Questions for authors
  • How accurate are the probes on the output of the convolutional network, before the rating enters the model?
  • Have you measured how the internal state changes between the lowest and the highest rating? For example, does the change lie in a few directions that the original policy head reads?
  • Have you tried the probe analysis on another model that represents skill differently, such as Maia-3 or ALLIE?
  • How were the training, validation and test splits made? Could closely related positions, such as consecutive positions of the same game, appear in both training and test?
5 comments
AuthorsOpenReview ↗

Authors' Rebuttal (Part 1)

We thank the reviewer for the careful review and address each point below.

W1 & Q1: Can probes read the concepts before the rating enters?

Yes, and they should, since a model that did not encode these concepts could not play well at any skill level. Over all 172 concepts, probes on the CNN output, which never sees the skill condition, reach 95.5%, on par with the residual stream. Yet knowledge encoding stays stable even in the part of the state that skill conditioning controls. Through the residual connection, the two Elo-aware transformer blocks could just as well cancel or overwrite what the CNN encodes. Instead, every skill condition adds to it.

We show this on the 21 tactical and safety concepts, such as forks, pins and king danger. On these, we probed the raw board planes, the CNN output, and the Elo-aware contribution, which is what the Elo-aware blocks add to the CNN embedding. All probes follow the paper's protocol and are averaged over the 11 skill levels:

Probe inputAcc.
Raw board71.5
CNN output85.7
Elo-aware contribution88.9
Residual stream after Block 288.6

A linear readout of the board planes stays near chance on some of these concepts. The CNN output already recovers most of them, so it carries what Maia-2 has learned rather than a copy of the board. The Elo-aware contribution then adds about 3 points at every skill level, and a probe trained on it at one skill level transfers to the others with little loss on average. Over all 172 concepts, the Elo-aware contribution is also on par with the residual stream. In the revision, Section 4 reports these probes and rests its test of H1 on the Elo-aware contribution.

W2: Do the heads just read the board encoding?

No. With the same supervision, Head SFT stays at least 30 points above both requested heads in every skill band, so its gain comes from concept knowledge retained in lower-skill representations, which Head SFT externalizes as Maia-2 does at higher skill, and not from the board encoding alone. We trained both heads with the exact Head SFT recipe, and each reads the mean-pooled tokens of its input, as the policy head does with Maia-2's final state:

Acc. (%)<1200
Head on CNN embedding40.1
Head on random-weight Maia-252.2
Head SFT82.8

W3 & Q2: What does the rating change inside the model?

We measured the change in the policy-head input between the lowest and the highest skill condition on the test split of the Blundered Transitional Dataset. The change does not lie in one shared skill direction, but a few directions that the original policy head reads are enough to shift its decision, and which ones depends on the concept at stake.

The average change from the lowest to the highest skill condition, which standard activation steering would add, carries under a tenth of the change and under 5% of the change in how strongly the original policy head prefers the best move over the blunder. Adding it to lower-skill representations does little better than adding a random direction of the same size, at every scale we tried.

The directions that matter are concept-specific. Feature steering amplifies a small set of SAE features that track the concept a position calls for and leaves the original policy head untouched. This raises best-move accuracy at <1200 from 14.4% to 42.7%, whereas random features reach only 16.3%. Within the scales we tried, we found no single skill vector that works this way for all positions.

This matches the concept specialization of Head SFT in Section 5.2, where a head trained on one concept does not act as a generic high-skill policy for the others. We have added this analysis to that discussion.

As for the title, the paper answers "how" at the level of knowledge: under lower skill conditions, Maia-2 keeps the concept knowledge but changes how it reaches the move. The analysis above adds what this change looks like in the internal state, and the introduction now states this scope. We thank the reviewer for this question, which makes the mechanistic picture of the paper more complete.

AuthorsOpenReview ↗

Authors' Rebuttal (Part 2)

W4 & W5: Concept scope and prior work

We agree and have scoped the abstract. Its sentence "we measure encoded knowledge by training probes to predict the presence of 172 chess concepts" now reads "172 chess concepts defined on the current board". Within that scope, these are exactly the concepts the blunders ignore, since each intervention targets a concept that the lower-skill prediction fails to address, such as an under-defended piece. Interpreting multi-move planning in chess models remains challenging [1, 2], so we leave it to future work, as Section 6.2 and Appendix B.1 already state.

We have dropped "first" and now discuss all three works. Each differs from ours in that we hold board positions and model weights fixed and vary only the skill condition. Karvonen (2024) restores play that random openings degrade, so the weakness there comes from the input. Li et al. (2023) recover truthful answers in a model that is not conditioned to act weaker. Amnesic probing (Elazar et al., 2021) asks whether probe-readable information is used at all, whereas we ask how skill conditioning changes that use.

W6 & Q3: Other models that represent skill differently

We added both kinds of model the reviewer lists, and in both, knowledge encoding stays constant across ratings, so this result does not depend on how Maia-2 represents skill. Maia-3 is a single transformer like Maia-2, but it attaches the ratings of both players to every board square at the input. Maia-1 departs furthest, with nine separate models from 1100 to 1900, each trained only on games from its own rating band, so its 1100 model has no skill condition and never sees high-rated play. Every part of these models therefore depends on skill, unlike the CNN of Maia-2. Both run on a single board position, so the concept labels carry over unchanged. We probed all 172 concepts, reading square concepts from the token of their square in Maia-3:

Probe acc. (%)11001300150017001900
Maia-392.993.193.093.093.1
Maia-195.595.995.095.695.2

Q4: Data splits

The splits are random splits of positions, and closely related positions do not drive the results. The dataset comes from the Lichess evaluation database, which stores positions with Stockfish evaluations but no game identifiers. To check for consecutive positions, we flagged every test position whose piece placement matches a training position or is one move away from one, 8.7% of the test set. Removing them lowers Head SFT accuracy by about 0.1 points in every skill band.

We thank the reviewer again for the precise and constructive review. The revision adds the probes on the CNN output and the Elo-aware contribution to Section 4, the head controls on the CNN embedding and random weights, the analysis of skill directions next to the concept specialization result, the Maia-3 and Maia-1 replications and the check for consecutive positions. It also states the scope of the concepts in the abstract and of the title in the introduction, drops "first" and discusses the three related works.

We greatly value the reviewer's feedback, which has made the paper stronger, and we would be glad to address any further questions or suggestions.

References

[1] L. Schut, N. Tomasev, T. McGrath, D. Hassabis, U. Paquet, B. Kim. Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero. arXiv:2310.16410, 2023.

[2] E. Jenner, S. Kapur, V. Georgiev, C. Allen, S. Emmons, S. Russell. Evidence of Learned Look-Ahead in a Chess-Playing Neural Network. arXiv:2406.00877, 2024.

Reviewer_zcCBOpenReview ↗

Thanks for the response; remaining questions

Thank you for the clear answers and the new experiments. I have a few follow-up questions.

  • Does the Elo-aware result also hold for the 71 concepts used in Section 5? The new test covers only the 21 fork, pin, king and queen safety concepts. This matters because the CNN output, which never sees the rating, already reaches 95.5% over all 172 concepts. The 71 concepts look mostly like square-control concepts, but I could not find their list, which Section 5.1 says is in Appendix B.5. It would also help to see the raw-board and random-weight probes over all 172 concepts, which the rebuttal mentions without numbers.
  • Why does the head on the random-weight model (52.2%) do better than the head on the trained CNN embedding (40.1%)? Could averaging the eight CNN tokens lose information?
  • Are the SAE features you amplify weaker at low skill than at high skill, on the same positions?
  • In the Maia-1 experiment, Head SFT raises the 1100 model from 34.3% to 72.7%, above the 63.9% of the untouched 1900 model. You read this as evidence that the 1100 model knows more than it plays. However, Head SFT trains the head on best moves, so the head itself can learn part of the task. Your Maia-2 control shows how large this part can be: the same head, trained on a Maia-2 with random weights, which knows nothing about chess, already reaches 52.2%. So only the gain above that level shows what the model itself knows. For Maia-1 this control is missing, and we cannot tell how much of the 72.7% comes from the 1100 model and how much from training the head. Could you report the same head trained on a Maia-1 network with random weights?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttals and Paper Revisions

We thank the three reviewers for their careful reviews. We have uploaded a revised PDF with the revisions marked in blue.

New results in the rebuttals

  1. Knowledge encoding stays stable even in the part of the state that skill conditioning controls (zcCB, 6Ewr). On the tactical and safety concepts, probes on what the two Elo-aware transformer blocks add to the CNN embedding are on par with the full residual stream at every skill level, while random weights stay near chance and a board readout is 17 points lower. Skill conditioning therefore keeps learned concept knowledge rather than overwriting it.
  2. Head SFT draws on concept knowledge beyond the board encoding (all reviewers). Trained with the exact Head SFT recipe, heads on the CNN embedding or on a random-weight Maia-2 stay at least 30 points below Head SFT in every skill band. Lower-skill representations therefore retain concept knowledge that Maia-2 does not externalize.
  3. Knowledge encoding stays stable in models that represent skill differently, and the separation holds without shared training (all reviewers). Probe accuracy stays flat across ratings in Maia-3, which takes the ratings at the input, and in Maia-1, whose models are each trained only on their own rating band. Head SFT also lifts Maia-1's 1100 model, which never saw high-rated play, above its untouched 1900 model. Stable knowledge encoding therefore does not depend on how Maia-2 represents skill, and the separation between knowledge encoding and knowledge externalization does not depend on shared training.
  4. A few concept-specific directions, rather than one shared skill direction, are enough to shift the original policy head's decision (zcCB). Adding the average change from the lowest to the highest skill condition to lower-skill representations does little better than a random direction of the same size, whereas amplifying a small set of SAE features of the concept at stake raises best-move accuracy at <1200 from 14.4% to 42.7%.
  5. Steering leaves knowledge encoding in place and helps most when the side to move has an under-defended piece (EjFK). It shifts the mean probability of the pre-steering probes by only 1.3 points for present concepts and 2.3 points for absent ones. At <1200, it improves every concept of this type, by 34 points on average, against 20 when the opponent's piece is under-defended.

Changes to the paper

  • The abstract states that the 172 concepts are defined on the current board.
  • Section 1 defines the key terms earlier, states H1 and H2 more plainly, states the scope of the title and no longer claims to be the first.
  • Section 2 discusses Karvonen (2024), Li et al. (2023) and amnesic probing.
  • Section 4 adds the probes on random weights, the raw board, the CNN output and the Elo-aware contribution, and rests its test of H1 on the Elo-aware contribution.
  • Section 5.2 adds the head controls and the analysis of skill directions.
  • Appendix B.2 describes the data splits and the check for consecutive positions, and the new Appendix D holds the Maia-3 and Maia-1 replications and the breakdown of feature steering by concept type and the probe probabilities after steering.

We thank the reviewers again and would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.