acceptodds
Under review as a conference paper at ICLR 2027

Human Difficulty Is Not Task Difficulty: What Policy Representations Know Beyond an Optimal Solver

Abstract

Human difficulty is not task difficulty. A policy network is trained to choose actions, not to judge how hard a problem is for people, yet part of it is linearly readable from its activations. Using chess as a model system, where difficulty is calibrated on large human populations, we probe eight networks. In Maia-2 a probe adds ΔR² = 0.0436 over gradient-boosted engine-search controls plus the model's own policy entropy, and 0.0350 with its own move predictions also controlled. Deeper engine search, at the depths tested, predicts human difficulty worse. The signal is shared across seven models after adjusting for the target, controls and a raw board encoding. We distinguish two measured aspects of difficulty: the population-calibrated rating and, separately, how long the original player deliberated, beyond that rating, the clock and engine features (the two readouts correlate at r = 0.59). Under matched ablations, neither direction has a distinctive effect on the solution move (the pre-specified primary outcome); the thinking-time direction shifts the move distribution (pre-specified secondary outcomes, small in absolute terms) without favouring or disfavouring the solution relative to matched controls. A person's error may also depend on how the situation arose: given the game so far, a human-imitation policy raises the probability of the wrong move a player made (0.111–0.136 nats, natural-log units, on months before Maia-3-79M's training window); an engine-trained network does not.

Primary area:
interpretability and explainable AI
Keywords:
representation learning, interpretability, probing, causal ablation, human-AI alignment, representational convergence, human difficulty, deliberation, value of computation, policy networks, chess as model system

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ApPKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks if the chess-playing policy networks encode information about how hard chess positions are for humans, without having been trained to predict difficulty. Paper also addresses if the networks use difficulty information when choosing moves. The authors first fit linear ridge probes on frozen activations of seven networks to predict puzzle ratings. They report the increment over Stockfish search features, the network policy entropy and the player's rating. There are 4 main findings: 1) difficulty can be decoded beyond engine features 2) the signal aggress across all networks after residualization 3) the thinking time is also decodable but is a separate signal from difficulty 4) difficulty isn't used for move selection.

Critical strengths

I found the controls to be very strong, with a three tier control ladder and a solution aware control, with about 80% of the signal surviving. In addition, the ablation design is very rigorous and detailed, including 50 variance-matched control directions, a second architecture, policy-jacobian positive control, etc.

Critical weaknesses

The biggest weakness is probably the disproportionate generality of the paper claim given that all tests are done in a single domain only. I understand that authors selected chess based on meeting 4 criteria, but there are probably other options that do as well (maybe Go?), or at least partially do. Furthermore, there are some concerns regarding interpretation and robustness of results - agreement across networks residual readouts probably should not be immediately interpreted as a difficulty signal (even with all board structure removal steps), and small effect sizes and lack of replication on held out months make the results less convincing.

Questions for authors
  1. Networks' activations range from 256 to 2048 dimensions, but every probe uses the same penalty. So wider layers give the probe more to work with. Can this explain why some models (Leela-CF) show larger increments? Would the ranking hold if all models had the same number of dimensions, or if the penalty was model-adjusted?

  2. The depth comparison result (which the main claim relies on) uses a subsample of all puzzles, and reports no confidence intervals - could you report the intervals for decline in engine-only R^2 and for the probe increments, and full data set results?

  3. In ablation studies, after you remove the difficulty direction, a new probe still recovers most of the difficulty signal - so the information is still int he network just along other directions. Couldn't this be interpreted as testing whether this one direction is used, not whether difficulty information is used? Could you try iteratively refitting and removing probe directions until difficulty is no longer decodable, and test whether the network behavior changes then?

2 comments
AuthorsOpenReview ↗

(1 of 2): width, penalty and robustness; no ranking is claimed

In short. No fixed ranking holds. Read the same way, which large network leads depends on how many components are read, so we claim no order. Changing the penalty moves no gain in the puzzle rating by more than 0.007. Every conclusion holds over eleven fold partitions. Table 5 states what each claim rests on and when its analysis was fixed.

"Would the ranking hold if all models had the same number of dimensions, or if the penalty was model-adjusted?"

No ranking holds, and the paper no longer claims one. We now read every network the same way: one common control block and the same number of principal components. The common block is the 23 engine and puzzle features plus the entropies of Maia-2, Maia-3-79M and Leela-CF.

Read so, Maia-3-79M and a Leela Chess Zero network of its size, whose training is not published, carry comparable amounts. Maia-3-79M leads at 32 and 64 components, the other network at 256 and above. Leela-CF minus Maia-3-79M is −0.0054 [−0.0102, −0.0010] at 32 components and positive, with its interval above zero, from 128 up. We claim no order.

The comparison network need not be the larger one: the network of Maia-3-79M's size removes 82% of Maia-3-79M's gain beyond the first control set.

On the penalty: at full width, with the penalty fixed, scaled with the layer's width, or tuned inside the training folds, no gain in the puzzle rating moves by more than 0.007 (Appendix A.2).

"Can this explain why some models (Leela-CF) show larger increments?" Not the readout's width: Leela-CF and Maia-3-79M are both read at 1,024 dimensions. How many components are read does matter: with 8 components Leela-CF keeps 25% of its full-width gain, against 64% for Maia-3-79M (Appendix A.2). As for network size, these data cannot separate size from accuracy. Across seven Leela Chess Zero networks, the gain rises with size and with first-move accuracy alike (Appendix D.1). At one fixed size, a self-play network's gain beyond the first control set still rises during training, from 0.015 to 0.035. Beyond the larger control sets, that rise is 0.005 or less.

Soundness. Table 5 (Section 9) gives each claim's effect and when its analysis was fixed. Every row is marked as fixed after related results were seen; only the first-set gain dates from the submission; the further month's rules were fixed in advance, and the table says so. The intervals resample source games and hold the fitted predictions fixed, so they leave out refitting. In three reruns of the engine, the gains moved by at most 0.003. Over ten further fold partitions, each gain stays within 0.005 of its value, and every conclusion holds in all eleven. The self-play comparison holds at every control level on all eleven (Section 5). Maia-2's shares depend on the partition, so the paper gives them as ranges. Choosing the layer inside each training fold picks the same layer in every fold for Maia-2, Maia-3-79M and Leela-CF. A separate re-implementation occasionally chose a neighboring layer. For Maia-3-79M the runner-up layer is nearly tied: it gives 0.1108 against 0.1113 (Appendix D.8).

The controls you found strong. You noted "about 80% of the signal surviving" a solution-aware control. The controls are wider now, and less survives them. In the three networks tested, about 40 to 60% of each gain remains beyond the second control set. Beyond the full set, about a quarter to a half remains, now in all eight networks (24 to 45% on our split, 21 to 45% over three fold partitions; Section 4).

In the paper: Sections 3, 5 and 9; Table 5; Appendices A.2 and D.

Continued in part 2 of 2.

AuthorsOpenReview ↗

(2 of 2): one domain, effect sizes, later months, depth, iterative removal

In short. We agree that every test is in chess and that the effects are modest. Residual agreement is no longer offered as evidence. Three later months carry over, but had been inspected before. A fresh month met rules fixed in advance. The depth result now uses all puzzles, with intervals, and no claim rests on it. Iterative removal settles neither use nor non-use.

"all tests are done in a single domain only"

Agreed: every test is in chess, and the paper's title and abstract now say so. We have not tested Go. It would allow the same comparison, but it lacks an openly available problem rating calibrated on many solvers (Section 9).

"agreement across networks residual readouts"

Agreed: it is no longer offered as evidence of a difficulty signal. The split that replaces it (Section 5) measures overlap in rating prediction, not a partition of features inside the networks.

"small effect sizes"

Agreed: they are modest. A readout lowers the error of an engine-based rating estimate by 13.5 to 42.0 points, from about 340 (Table 1). Beyond the full control set, the readouts of Maia-2, Maia-3-79M and Leela-CF explain 2.8%, 8.7% and 13.2% of the variance the controls leave (Table 5). This is an effect size, not a rating method (Section 7).

"lack of replication on held out months"

We tested September 2021, a month this study had not used. Before reading its ratings, we fixed rules from the January 2019 results, and the month met all of them. Two rules go beyond signs and shares. Leela-CF's gain had to reach 0.80 of Maia-3-79M's; it is 1.41 times as large, against 1.13 on January 2019. Maia-3-79M's history shift had to exceed Leela-CF's by 0.04 nats on the matched score; it does by 0.085. The paper's leading score subtracts a further baseline. Computed afterwards and outside the rules, it gives 0.061 nats [0.050, 0.071]. The other eight rules are easier; a random-weight network meets two. Every gain is smaller than on January 2019, and Maia-2's training window covers this month. We changed the month from March 2021 before reading any September rating: one March rating had been printed before the plan was frozen. The rules stayed the same (Appendix A.16).

Three later months, which had been inspected before, show robustness but not a fresh test. Fitted once and applied unchanged, the self-play readout raises the share of puzzle pairs ordered correctly from 70 to 76% to 78 to 82% (Section 7). Its addition is 8.1 [7.7, 8.4], 5.4 [5.3, 5.5] and 5.5 [5.2, 5.8] percentage points in the three months. These intervals resample the target month's puzzles and hold the fit fixed (Table 4).

"could you report the intervals for decline in engine-only R^2 and for the probe increments, and full data set results?"

Yes, and the main claim no longer relies on it. On all 26,352 puzzles, the seven-feature engine summary predicts the rating worse at depth 20 than at depth 12, by 0.0319 [0.0278, 0.0359] in R². Readout gains over the depth-12 summary are 0.0714 [0.0666, 0.0764] (Maia-2), 0.1591 [0.1526, 0.1662] (Maia-3-79M) and 0.2239 [0.2163, 0.2322] (Leela-CF). Over the depth-20 summary they are larger only because the engine baseline falls; engine plus readout predicts no better. So we withdraw the growth of the gain with depth. The decline is consistent with how Lichess selects puzzles, and the main text no longer claims it (Appendix A.14).

"Could you try iteratively refitting and removing probe directions until difficulty is no longer decodable, and test whether the network behavior changes then?"

Yes: removing one direction tests only that direction. We ran iterative removal in Maia-2. Behavior changes, but not selectively, so it settles neither use nor non-use. Non-use is no longer claimed. Removing raw-rating directions makes the solution move more likely than matched random edits do. For the part beyond the engine, the stopping rule fixed in advance was never met. A rule chosen afterwards is met only at 50 to 64 directions. At 64, the solution becomes less likely than under every matched edit. Neither removal is selective: the network's own entropy and solution probability become less readable too (Section 8).

In the paper: Sections 5, 7, 8 and 9; Appendix A.6; Appendix A.14; Appendix A.16.

Thank you. Your feedback was detailed, thoughtful and thorough. We are sorry that this reply comes late in the discussion period: we wanted to do justice to every point by revising the paper thoroughly rather than send a rushed reply. We have addressed every point of your review in the revised paper, which is substantially rewritten for clarity, rigor and completeness. We would be grateful if you would read the revised paper and reconsider your rating in its light.

Reviewer_TRy3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper asks whether human difficulty (operationalised as the probability that humans will solve a Lichess problem) can be read out from the activations of various models that are trained to play chess (either via RL or various supervised methods), even after accounting for some confounding variables, such as how uncertain the model is. "Task difficulty is not human difficulty". But I'm honestly not sure what question the authors are trying to answer here.

If two agents perform a task under slightly different policies, they will most likely have different probabilities of response for some items. This means that the two neural axes representing uncertainty in these models will potentially be misaligned, so after a decoder is trained on one, there will be residual variance due to the other. This is the basic result here: a human (agent 1) and a model (agent 2) are uncertain about different things. This is not in the least surprising. The authors also partialling out variance from another agent (e.g. Stockfish) but that doesn't change the basic conclusion.

The authors do some more detailed investigations. Some of the results are more curious than others. A model not trained on human data explains the most residual human uncertainty. perhaps surprising, but unexplained. a curiosity. Models have partially shared difficulty axes, residualised against several variables, and this is stronger within model families. So the models learn a bunch of correlated stuff. Not that surprising. There's also a representation of thinking time. Not that surprising, IMO, but nice to see. There's something about history effects, which seems to be that human-trained model contain history information, whereas others don't. OK.

They try to ablate this component, to see how it affects performance, but these experiments have pretty equivocal results.

Like so many ICLR papers, this one is written in an ultra-dense style that assumes that the reader was there during every moment of project execution rather than hearing about the work for the first time.

Critical strengths

I honestly can't think of any. it's a fine paper. but what is this for? what do we learn?

Critical weaknesses

see above

Questions for authors

why would information about human difficulty not be decodable from the models?

4 comments
AuthorsOpenReview ↗

what we learn, what it is for, and what changed in the writing

What we learn. Finding a human quantity in a network does not establish that the network learned it from people. In chess, most of what the human-imitation networks encode about how hard a puzzle is for people is also in the readout of Leela-CF, a self-play network not trained on human games. Once Leela-CF's readout is added to the engine controls, Maia-3-79M keeps 14% of what its own readout had added. The practical lesson: before crediting a human quantity found in a model to its human training data, read a model trained on the same task without that data. Without that check, all of Maia-3-79M's gain could have been credited to learning from people. What does differ between the two kinds of network is their response to the game so far. Shown the moves that led to a position, human-imitation networks of two families shift more probability toward the mistake the player made than their self-play counterparts do. The submission said an engine-trained network shows no such shift; the revision narrows this to a smaller one. Within chess, the self-play readout, fitted on one month, also improves the ordering of later puzzles by difficulty before anyone attempts them, by 5.4 to 8.1 percentage points of pairs ordered correctly (not a rating method; Section 7). For models of human planning, the part of people's difficulty that search measures leave out is a target (Section 9).

"why would information about human difficulty not be decodable from the models?" We agree that it would be. It does not come from the architecture alone: random-weight copies of five networks carry at most a fifth of the gain over engine features and entropy (Section 4). So the question, now the first line of the abstract, is whether the information is learned from people.

"perhaps surprising, but unexplained". Which network leads depends on how the networks are read, so the paper now claims no order. A plausible reason for the result is that playing well requires representing what makes the right move hard to find. Two examples are whether it is a quiet move or a capture, and how late its value appears in search. Controls for these absorb about half of the gain in Maia-2, Maia-3-79M and Leela-CF. Of what is left after every search and solution control we built, the largest single part we can name is the forcing moves a position offers, checks and captures. Leaving the forcing-move counts out of a description of the board costs 10 to 16% of the remaining gain in three networks. Other descriptions of the board do most of the same work, so this names a kind of information, not a feature. Puzzles with more forcing moves are rated harder than the controls predict (Section 9).

"a human (agent 1) and a model (agent 2) are uncertain about different things". We agree, and that alone would not be a finding: the network's own uncertainty is among our controls. What we add is a test of whether a readout only restates that uncertainty. If a readout only restated the network's uncertainty, it would add little where the network is sure. Yet where Leela-CF puts more than 0.95 of its probability on one move, its readout still removes 20.6% of the squared error the controls leave. On all puzzles it removes 24.1%: the gain is smaller where the network is sure, but not absent. For Maia-3-79M your reading is partly right: part of its gain does follow the uncertainty it expresses. We cannot test uncertainty that a network holds and does not express. Further checks are in Section 4 and Appendix D.

"ultra-dense style". You are right that the submission was written for readers who had been there. The main text is rewritten in short, plain sentences. The abstract and the introduction open with the question in plain words, and the introduction says what a readout is before it gives any number. Figure 1 now opens with the design in three steps.

The rest. You read the "partially shared difficulty axes" as "the models learn a bunch of correlated stuff"; we agree, and they are no longer offered as evidence. The ablations gave "pretty equivocal results"; that section is now titled "Use is not shown". The thinking-time readout, which you found "nice to see", remains, with fuller controls; its gains are small and come from one month (Section 9).

In the paper: Sections 1 and 4 to 9; Appendix D.

Thank you. Your feedback was thoughtful and candid. We are sorry that this reply comes late in the discussion period: we wanted to do justice to every point by revising the paper thoroughly rather than send a rushed reply. We have addressed every point of your review in the revised paper, which is substantially rewritten for clarity, rigor and completeness. We would be grateful if you would read the revised paper and reconsider your rating in its light.

Thanks for these clarifications. I am sorry not to be more positive but I just don't get this paper. "The practical lesson: before crediting a human quantity found in a model to its human training data, read a model trained on the same task without that data". Well, sure - but did you need to do this study to know that? I agree with the other reviewers that the analyses are well-conducted and rigorous, I can't work out what problem you are solving our question you are answering. So respectfully, I will keep my score - but thanks for engaging.

AuthorsOpenReview ↗

The question in one line, and why its answer was not known

Thank you for reading our reply and for the candid answer. one point only, on "did you need to do this study to know that?" The principle is not new, and we do not claim it. What was not known is the answer in this case. Networks trained on human games are already used to measure how hard a decision is for people, and our own submission read its result that way. Your review called the self-play result "perhaps surprising". That surprise is the finding: once a self-play network's readout is added, the best human-imitation network keeps 14% of its gain beyond the first control set, and that remainder is not shown to be about people. So the question is: in chess, does a policy network need training on human games to encode what people find hard? The answer is no. A self-play network carries about as much, and what human training adds in these tests is the response to the game so far. Thank you again for engaging.

AuthorsOpenReview ↗

( continued) The question in one line, and why its answer was not known

Thank you for reading our reply and for the candid answer. We will not repeat it. Your two questions deserve a detailed answer, so here it is, once, without the technical vocabulary.

The question, in one line.

In chess, does a network have to learn from people in order to know what people find hard?

Why that is a real question.

Networks trained on millions of human games are the standard tool for modelling human players, and they are already used as measures of how hard a decision is for a person. The natural reading is that they know what people find hard because they learned it from people. Our own submission reported that a self-play network also carries this information, and still credited the human-trained networks' gains to what they had learned. It took Reviewer MDuT's objection to make us test that.

An everyday version.

A tutor who has marked thousands of student scripts can tell you which exam questions students will get wrong. You would say she learned that from the scripts. Now suppose a mathematician who has never seen a student's script predicts the same questions about as well. Then you can no longer credit the tutor's skill to the scripts: much of what makes a question hard for students can be seen in the question, by anyone who understands it well. Here the puzzles are the questions, human games are the scripts, and a self-play network is the mathematician.

What we found.

  • A network trained on self-play carries about as much of this information as the best network trained on human games. Once the self-play network's readout is added first, the human-trained network keeps 14% of what its own readout had added beyond the first control set, and that remainder is not shown to be about people.
  • Training on human games does show, but somewhere else. Given the moves that led to a position, human-trained networks of two families shift more probability toward the mistake the player actually made than self-play networks do.
  • So two things that are easy to run together come apart in these tests: how hard a position is for people, which did not need human games, and what the game so far says about the mistake this player makes, where human games help.

"Did you need to do this study to know that?"

Everyone agrees that a drug should be tested against a placebo. That does not tell you what the trial will show. That one should compare against a network not trained on human games is common sense, and we claim no credit for the idea. What the comparison shows was not known. Your review called the self-play result "perhaps surprising, but unexplained", and our submission drew the wrong conclusion from it.

Why it matters beyond this paper.

  • For anyone who predicts difficulty for people, such as rating new puzzles: this signal does not require human game data. A self-play readout fitted on one month, and applied unchanged to later months, puts 5.4 to 8.1 percentage points more pairs of new puzzles in the right order of difficulty before anyone attempts them.
  • For anyone who finds a "human" quantity inside a model trained on human data: this is a worked case in which the obvious attribution does not hold, with the size of the gap measured.
  • For the study of human play: what a strong player's network cannot supply, and human games can, is narrower than assumed. In these tests it concerns the player's actual mistake given the game so far.

What we do not claim.

This is one game, and the effects are modest. The networks that remove most of the gain are also stronger players, so training data and playing strength are not separated.

We understand if this does not change your assessment, and we are grateful that you engaged.

Reviewer_MDuTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether chess policy networks, which are not trained to judge difficulty, encode how hard a puzzle is for people beyond what a chess engine captures. On 26,352 Lichess puzzles, linear probes on the frozen activations of seven networks predict the puzzle rating beyond 23 engine and puzzle features and the network's own policy entropy. A second probe predicts how long the original player thought, beyond the rating. Ablating the difficulty direction changes the network's move predictions no more than ablating random directions of matched variance. Finally, giving Maia-3 the real game history raises the probability of the wrong move the player actually made, while an engine-trained network shows no such gain.

Critical strengths

S1. The measurements are careful and transparent. The controls are strong (a gradient-boosted model on 23 engine and puzzle features plus the network's own policy entropy), and the paper includes raw-board and untrained-network baselines. The ablation is compared with 50 random directions of matched variance and with a positive control. Key results are repeated on later months, and null results and changes to the pre-specified analyses are reported openly.

S2. Section 8 gives a useful result for modeling human errors. Given the real game history, Maia-3 assigns a higher probability to the wrong move the player actually made, at every model size and in all four months tested. Leela-CF, given the same history, shows no gain (Tables 3 and 5).

Critical weaknesses

W1. The main claim depends on a narrow set of controls. The paper defines "task difficulty" as what the features of a Stockfish search capture (§3). A trained policy network can encode much more about a position and its solution than these summary features. Leela-CF, which never saw a human game, shows the largest gain (0.126, Table 1). So the gain may come from a better understanding of the puzzle, not from knowledge about people. The controls also miss direct measures of how hard the solution is to find, such as the search depth at which the engine first finds the solution or the type of the solution move. (§3, §4, Appendix A.2)

W2. The ablation in Section 7 cannot show whether the network uses the information beyond the engine. It ablates the direction of a probe trained on the raw rating, and the controls largely share what this probe explains. So the ablation does not target the part of difficulty the paper is about. In addition, a probe refitted after ablation still recovers R^2 = 0.2773 of 0.3011, so the information remains in the layer. A null result therefore cannot separate "not used" from "used through other directions," as the paper itself notes. (§7, §9, Appendix A.5)

W3. The paper is hard to read. The main text uses many labels, such as "fixed-OOF", "full-OOF-refit", "maximal tier" and C1/C2, reports several interval types side by side, and qualifies many sentences with several clauses, so the main results are hard to follow. (§3, §4)

Questions for authors

Q1 (clarification). Maia-2 and Maia-3 take both players' ratings as input, but the controls include only the mover's rating. Which opponent rating was used when collecting activations? If it was the real rating, does the gain change when the opponent's rating is added to the controls?

Q2 (additional result). Related to W1, how much of the gain remains after adding the shallowest search depth at which Stockfish finds and keeps the solution move, and features of the solution move such as capture, check, quiet move, or material given up?

2 comments
AuthorsOpenReview ↗

(1 of 2): The self-play control (W1), solution-finding controls (Q2), rating inputs (Q1)

In short.

  • W1: You were right. Once Leela-CF's readout joins the first control set, Maia-3-79M keeps 14% of its gain. The paper no longer credits that gain to learning from people.
  • Q2: In the three networks tested, about 40 to 60% of the gain survives the controls you asked for. In all eight networks, about a quarter to a half survives the full control set.
  • Q1: Networks that take ratings received both players' actual ratings. Adding every input besides the pieces, ratings included, to the controls moves no trained network's gain by more than 0.0014.

W1 (people or the position). A gain is the rise in held-out R² of the puzzle rating when a network's readout is added to the controls. The first control set is the submission's: engine and puzzle features plus the network's own policy entropy.

We tested your reading directly. We added Leela-CF's readout to the controls before measuring each human-imitation network's gain. Maia-3-79M then keeps 14% of its gain beyond the first control set. The overlap runs both ways: beyond the first control set, Maia-3-79M's readout removes more than half of Leela-CF's gain, and somewhat less than half beyond the second.

The comparison network need not be the larger one. A Leela Chess Zero network of Maia-3-79M's size, whose training is not published, removes 82% of Maia-3-79M's gain beyond the first control set. Beside its readout, Maia-3-79M keeps 18%, 26% and 35% of its gain at the three control levels (0.013 to 0.021 in R² at every level). So both comparison networks remove most of Maia-3-79M's gain at every control level, on all eleven fold partitions.

So the paper no longer claims that this information is specific to networks trained on people. Where the two kinds of network differ is what they do with the game so far (S2, in our other reply).

Two limits remain:

  • Size is answered only in part, because the network of Maia-3-79M's size has no published training record.
  • Strength is not separated from training data. Every network that removes most of the gain is more accurate on these puzzles than Maia-3-79M. As the comparison network's accuracy and gain rise, the share Maia-3-79M keeps falls from 75% to 14%.

So the 14% that remains is not an estimate of what training on human games contributes. Separating the two would need a human-trained and a self-play network with everything else fixed. We could not run that.

Q2 (how hard the solution is to find). Four blocks from your Q2 and Q1, added to the first set, make the second control set:

  • search effort, including the shallowest depth at which Stockfish finds and keeps the solution;
  • the solution move's kind (capture, check or quiet);
  • material given up along the line;
  • game metadata.

The full control set adds the engine's best line at every depth, the values of alternative moves across depths, and puzzle-theme tags.

Gain beyondMaia-2Maia-3-79MLeela-CF
First control set0.0444 [0.0403, 0.0488]0.1113 [0.1058, 0.1178]0.1258 [0.1201, 0.1310]
Second control set0.0222 [0.0188, 0.0255] (50%)0.0532 [0.0487, 0.0580] (48%)0.0733 [0.0686, 0.0780] (58%)
Full control set0.0122 [0.0096, 0.0151] (27%)0.0375 [0.0335, 0.0416] (34%)0.0569 [0.0525, 0.0614] (45%)

Brackets: 95% intervals over source games. %: share of the first row. Most added columns use the known solution.

We also ran the full control set in five more networks. In all eight, about a quarter to a half of the gain survives it (24 to 45% on our split, 21 to 45% over three fold partitions).

So what survives the full control set is not only search difficulty as we measured it. The limit is in the measuring: one engine, and the first solver position only, not the whole solution line.

Q1 (the rating inputs). The activations were collected with both players' actual ratings, in every network that takes them. Maia-2's probed layer comes before rating conditioning, so its activations there do not depend on either rating.

We then added to the controls every input a network receives besides the position of the pieces. Depending on the network, these are the side to move, castling rights, the en-passant square, the move count since the last capture or pawn move, and both ratings. No trained network's gain moved by more than 0.0014.

The second and full control sets also hold the opponent's rating. So the lower rows of the table are measured with it as a control. Fixing the inputs does not test whether a network infers a player's rating from the board, and we do not claim that.

In the paper: W1 in Section 5 and Table 3, with Appendices A.12 and D.7. Q2 in Section 4, Table 2 and Appendix A.2. Q1 in Section 3 and Appendices A.2 and D.9.

Continued in part 2 of 2.

AuthorsOpenReview ↗

Continue (2 of 2): erasure and use (W2), readability (W3), the history result (S2)

In short.

  • W2: You are right on both points. We erased the part beyond the engine directly. The tests show neither use nor non-use, and the paper now claims neither.
  • W3: Agreed. The main text is rewritten for a first reading.
  • S2: The history result keeps its sign in two families, but not its size.

W2 (the ablation cannot show use). A raw-rating direction does not target the part beyond the engine, and a null cannot show non-use. So we changed the test in three ways:

  • We erased the part beyond the engine itself, with linear concept erasure (LEACE).
  • We read it again with a fresh readout on held-out puzzles.
  • We compared the changed policy with random edits matched to the erased direction.

Two things keep these tests from deciding.

First, the information is not fully removed. On held-out puzzles, a fresh linear readout still recovers R² 0.012 to 0.036 of 0.16 to 0.18 in Maia-3-79M, less when the eraser is fitted on more puzzles. In Maia-2 and Maia-3-79M, nonlinear readers still recover between a third and nearly all of it.

Second, no policy change is shown to be specific to difficulty:

  • Leela-CF: no outcome departs from its controls.
  • Maia-2, probed layer: the next convolution transmits little of the erased direction, so this null says little.
  • Maia-2, last representation: the solution becomes less likely than under two families of matched edits, but stays inside a third family also matched on variance.
  • Maia-3-79M: the policy changes more than under edits matched on size and angle. Edits also matched on variance and on overlap with policy-sensitive directions reproduce the pattern.

We designed those last edits after seeing the change, so we read it as sensitivity, not use. The paper now claims neither use nor non-use.

W3 (hard to read). What changed:

  • Page 1 states the question, the method and the answer.
  • Figure 1 opens with the design in three steps.
  • Section 3 defines the readout, the gain and the three control sets before the results sections.
  • Each results section opens with its question and its answer.
  • Secondary checks sit in the appendix, behind pointers.
  • The labels you list are gone from the main text, and its sentences are short.
  • The gain's interval is defined once, in Section 3; Table 4 states its own.

S2 (the history result). Your reading matches the submission; the revision narrows "no gain" to "less". The result now rests on two families, scored relative to a matched move the player did not make:

  • Maia-3-79M exceeds Leela-CF by 0.051 nats [0.039, 0.064], over three months outside its training window.
  • Each of nine Maia-1 networks exceeds a self-play network with the same input encoding (median 0.017 [0.008, 0.027]; 3 of 9 survive simultaneous intervals).

The nine share one comparison network, so they are not nine independent tests. The sign holds under every way we tried of supplying the missing history. The size does not.

The score and its pooling over months were fixed after related results were seen. The later-month forecasts of Section 7 also use months inspected before: robustness, not a fresh test. On the submitted measure, the played move alone, the second family does not replicate. So that measure no longer leads.

How the score works. We show a network the same position twice: with the recent moves, and with only the opponent's last move. We take the change in log-probability of the player's mistake. We subtract the same change for the nearest wrong move in probability that the player did not make. History might favor that matched move for other reasons, so we also subtract its own matched score.

On a further month, September 2021, a rule fixed before its ratings were read required Maia-3-79M's matched score to exceed Leela-CF's by 0.04 nats; it did so by 0.085. The full score, computed afterwards and outside the rules, gives 0.061 [0.050, 0.071]. Size and strength are not separated from training data here. History may also tell a network who is playing. With Maia-3-79M's rating inputs fixed at 1500, its contrast remains, at 0.056 [0.042, 0.070]. In 2020-01, fixing them at 1900 lowers it by 0.031.

In the paper: W2 in Section 8 and Appendix A.6. W3 in Sections 1 and 3 and Figure 1. S2 in Section 6, Table 5 and Appendices B and D.5.

Thank you. Your feedback was detailed, thoughtful and thorough. We are sorry that this reply comes late in the discussion period: we wanted to do justice to every point by revising the paper thoroughly rather than send a rushed reply. We have addressed every point of your review in the revised paper, which is substantially rewritten for clarity, rigor and completeness. We would be grateful if you would read the revised paper and reconsider your rating in its light.

Comments on the paper

1 comment
AuthorsOpenReview ↗

the paper's one question, what it finds, and what changed

To all reviewers and the area chair. We thank the three reviewers. The paper asks one question: when a network's activations predict what people find hard, did it learn that from people? We ask it in chess and claim nothing beyond chess. Each point is answered in the individual replies.

In two lines, and a check.

  1. A readout of a frozen chess policy network predicts how hard a puzzle is for people beyond an engine's search. A self-play network's readout does so too, so a self-play comparison changes what can be credited to learning from people.
  2. The two kinds of network differ in what they do with the game so far. Human-imitation networks of two families shift more toward the player's actual mistake than their self-play counterparts.
  3. On a further month, the gains, the removal by the self-play readout and the history contrast on its matched score met rules fixed before that month's ratings were read.

What the paper claims. The submission's first result stands, with a corrected estimator: the submitted one stacked the readout's out-of-fold predictions, and nesting them moves no network's rating gain by more than 0.0016 (Appendix A.1). A network's gain is what its readout adds to a prediction of the puzzle rating from engine controls. In all eight networks, about a quarter to a half of that gain survives the full set of search and solution controls. The submission already reported that a network trained only on engine targets carries this signal, and that claim is kept. Reviewer MDuT pointed out what it implies for Table 1, and the revision tests it. With the readout of the self-play network Leela-CF added, Maia-3-79M keeps 14% of its gain beyond the first control set, and that 14% is not shown to be about people. A Leela Chess Zero network of Maia-3-79M's size, whose training is not published, removes 82% of Maia-3-79M's gain beyond the first control set. Read the same way, which large network leads depends on how many components are read, so we claim no order among them. Erasure tests do not show that any network uses the difficulty information.

What is unchanged, and what changed. The data, the readout and the networks of Table 1 are the submission's; further networks were added. The title and the abstract changed: the title is now Is Human Difficulty Learned from Humans? What Chess Policy Networks Encode Beyond an Engine's Search. The supplement was replaced; it holds the code and results of the added analyses. The TL;DR field of the form cannot be edited and still shows the submitted one; its statement that ablation has no distinctive effect is withdrawn (Section 8). .Of the seven submitted claims, two are kept, three are narrowed and two are withdrawn; the change map on the first page of the appendix lists each. Table 5 shows when each analysis was fixed: every row is marked as fixed after related results were seen; only the first-set gain dates from the submission, and the further month's rules were fixed in advance. The main text was rewritten for a first-time reader, and Figure 1 now opens with a three-step picture of the design.

What was added, by reviewer.

  • MDuT: wider search and solution controls, now run in eight networks, and, as a second comparison at every control level, a Leela Chess Zero network of Maia-3-79M's size. The direction of the history result also appears in a second family of networks.
  • TRy3: what we learn, stated plainly: before crediting a human quantity to human training data, read a network trained on the same task without it.
  • ApPK: every network read the same way, with no ranking claimed. Eleven fold partitions, effects as shares of what the controls leave, and intervals for the later-month forecasts. The depth result on all puzzles, with intervals, and iterative removal. A further month under rules fixed in advance.

Appendix D gathers the added analyses, each headed by its question, and lists the hypotheses that failed. We are sorry that these replies come late in the discussion period: we wanted to revise the paper thoroughly rather than send rushed replies. We would be grateful if you would read the revised paper and weigh it in this light.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.