- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- The authors defined 172 human-interpretable binary chess concepts and trained a set of linear probes on the output of Maia2 to predict the presence of these concepts at a given state
- They construct a dataset of positions where the low-skill Maia2 model makes a blunder and does not address a particular concept, while the highest-skill Maia2 takes the optimal action and addresses this concept.
- They showed that fine-tuning the Maia2 policy head output of the low-skill embedding on the best moves on this dataset leads to strong improvements in the low-skill model’s performance
- They demonstrated a method for using an SAE to recover activations corresponding to each of the chess concepts, and show that amplifying these activations allows the low-skill model to improve performance at the relevant states
Critical strengths
- I believe the feature steering section contains the strongest and most interesting results from this paper. The ability to improve low-skill performance by amplifying salient aspects of the board representation is a very promising way of understanding the weaknesses both of a model, and the humans of which it was trained.
Critical weaknesses
- The construction of the transitional dataset used as a base for all experiments contains a few flaws
- “93.6% of positions in the Blundered Transitional Dataset contain at least one oracle concept that lower-skill predictions fail to address correctly and higher-skill predictions resolve”, but this is not a useful statement by itself. Considering there are 172 different concepts, I would expect this to be true in high percentage of states just by chance
- Along this line, having a control where the low-skill move addresses the concept but the high-skill move does not would be useful comparison for each of these results
- Head-SFT does not seem especially compelling to me.
- At a high level, it seems we are just taking a model which was trained on both low-skill and high-skill actions and conditioned on low-skill data. Then, by fine-tuning this model on high-skill actions in a subset of the state space, the model is able to perform better in this region of the state space.
- This would be partially addressed by conducting the same experiment using Maia-1 (i.e. individually trained models where the low-skill condition hasn’t already been trained on high-skill actions)
- The paper does not spend enough time exploring the feature steering results.
- For example, we are missing any analysis on how feature steering efficacy varies across different concepts. I would assume that some concepts would lead to higher performance improvements than others, particularaly in states where multiple concepts are relevant. I would also expect that this would change at different skill levels, connecting to the discussion point about education
- The comment that mean probe accuracy changes by only 0.86% does not seem like the right comparison to me, since probe presence can change significantly (e.g. from 51% to 99%) without changing the accuracy.
- The fact that random-feature steering improves performace at all is quite surprising and deserves further analysis. At first glance, I would expect highlighting a section of the board which is unrelated to the best move would cause the model to make the best move less often if anything.
Questions for authors
- What are the practical implications of the finding that low-skill-conditioned representations contain high-skill capabilities?
- Could you provide additional results from your existing experiments to contetualize the results better?
- How does probe value change, not just accuracy?
- How does Head SFT and steering results vary based on the concept?
- Why does random steering improve performance?
2 comments
Thank you for the detailed rebuttal.
W1, W4, W6, and W7 are fully addressed.
W2 is partially addressed, I acknowledge the authors saying looking at the reverse control is out of scope of the paper, but I still feel this somewhat limits the understanding of the results and practical implications. E.g. if reverse control steering helps, then that might mean the model should focus on relevant concepts regardless of their correctness in order to improve play, by deciding when to take it into account or deliberately avoid it. If reverse control steering doesn’t help, this could mean the intervention is mostly about increasing preferences for concept-related actions.
W3 is the biggest open question here, and I am still not convinced that the approach here is especially insightful or compelling. I agree that pretraining a model on relevant domain data (low-skill or high-skill) allows for more effective fine-tuning over a random-weight model or a model not trained to solve this particular task (Maia-1 is not trained to make the best move). But this is not very surprising to me, it seems like the expected result that pre-training on a related task will improve finetuning the model on better actions.
W5 was slightly addressed, though I think a full addressal of this is not in the scope of a standard rebuttal. I would like to see a more through discussion of how the specific concepts improve play more than others at different skill levels, why these differences exist, and how those differences could be directly used in education or other human settings.
Overall, I appreciate the additional experiments and agree they strengthen the evidence for the findings. However my primary concern was the novelty and scientific insight of the findings, not the soundness, and I remain unconvinced that the Head SFT experiments demonstrate anything particularily surprising. The feature-steering results are more interesting but remain insufficiently investigated. Therefore, I will maintain my original rating.
Authors' Rebuttal
We thank the reviewer for the careful review.
The 93.6% statistic only motivates the concept set in Section 4.1, and none of our conclusions rests on it. Whether a concept matters in a position, rather than being present by chance, is what the interventions test. If the targeted concepts were incidental, random SAE features would steer as well as concept features, and a head trained for one concept would work on the others too. Neither holds: random features add almost nothing, as W7 details, and Section 5.2 shows that a head trained for one concept loses much of its gain on the others.
We considered the reverse control, but it does not fit the question the paper asks. In chess, addressing a concept is not good in itself: when the best move ignores a threat, the threat is not what matters in that position, for example because a stronger counterattack exists. Our question is whether lower-skill play fails to use knowledge that the right decision requires, and a position where the right decision does not require the concept cannot answer it.
The gain of Head SFT requires the concept knowledge retained in Maia-2's lower-skill representations. In Section 5.2, a head trained only at ≥2000 never sees a lower-skill representation yet transfers to <1200. To test whether this fine-tuning succeeds from any input, we applied the exact Head SFT recipe to other inputs, and Head SFT stays at least 30 points above both in every skill band:
As the reviewer suggested, we repeated the experiment on Maia-1, whose models are trained separately per rating band. Head SFT lifts its 1100 model, which has never seen high-skill actions, above the untouched 1900 model, while probe accuracy stays flat:
By concept type, steering helps most when the side to move has an under-defended piece. At <1200, it improves every such concept, by 34 points on average, against 20 points when the opponent's piece is under-defended. Across these concepts, steering improves 95% of them at <1200, and Head SFT improves every concept.
Table 1 reports both interventions per skill band, with the standard deviation across concepts as subscripts. Steering gains are largest at <1200 and shrink steadily with skill, while Head SFT stays high in every band. For education, this means the gap between a simulated player's moves and its retained knowledge is widest for the weakest players.
Probe values barely change, so probe-measured knowledge encoding stays close to its original values. Beyond the 0.86-point accuracy change in the paper, we read the probabilities of the pre-steering probes on the steered states at every skill level. Steering shifts the mean probe probability by only 1.3 points for present concepts and by 2.3 points for absent ones, far from a shift such as 51% to 99%.
Table 1 shows that random steering adds almost nothing: 1.9 points at <1200, at most 1.3 points in any higher band, and none at 1800–1999. Within <1200, what little it adds sits below 1100, where best-move accuracy starts at zero because every lowest-skill prediction is a blunder, so any perturbation can only help. Ignore Rate and Blunder Rate are likewise measured only where Maia-2 starts from a wrong move, which is why random steering lowers them to 83.8% and 47.3%. Concept features, in sharp contrast, raise accuracy at <1200 from 14.4% to 42.7% and lower the two rates to 48.6% and 32.1%.
The main implication is for AI safety and oversight: weaker behavior can coexist with retained knowledge, so evaluating a model under one conditioning context can understate what it can do. For education, matching novice behavior alone cannot show that a simulated learner models what a student knows. Section 6.1 discusses both implications in more detail.
We hope these responses clarify the points the paper left open to misreading, namely the role of the 93.6% statistic and what Head SFT is trained on, and that the head controls, the Maia-1 replication, the breakdown by concept and the probe probabilities address the remaining concerns. We have added all of them to the revised manuscript.
We greatly value the reviewer's feedback, which has made the paper stronger, and we would be glad to address any further questions or suggestions.