- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces new metrics for evaluating how human-like a model is. Previous approaches have largely relied on human move-matching accuracy to assess human likeness. Using additional metrics, this paper shows that models with similar move-matching accuracy may nevertheless differ from humans in their overall move-quality distributions, including error frequency and error severity.
The paper therefore proposes a post-hoc correction method for these distributions. It uses Stockfish to evaluate the quality of each candidate move in the model’s predicted probability distribution, then adjusts that distribution using human data from different skill levels. The results show improvements on several of the metrics presented in the paper. However, in the actual gameplay experiments, matches between agents conditioned and calibrated for different ratings still produce smaller performance gaps than those observed between corresponding human rating groups.
Critical strengths
- The paper introduces complementary metrics beyond move-matching accuracy, revealing differences in error frequency and severity that similar prediction accuracies can conceal.
- The proposed correction effectively aligns the model with the authors’ chosen targets: aggregate class frequencies and class-conditional mean error severity estimated from human players at each rating level.
Critical weaknesses
-
The relevance of the proposed metrics to human likeness requires further validation. Players at the same rating may have different error rates and compensate through different strengths, so deviation from a group average does not necessarily imply non-human-like behavior. Reporting player-level distributions within each rating group, including medians and interquartile ranges, would help assess how representative the aggregate targets are. Human evaluation is also needed to determine whether these statistical differences actually make an agent appear less human-like.
-
Improvements on the metrics targeted by the correction are expected, but better scores do not necessarily mean the model is more human-like. In actual gameplay, the corrected agents still do not reproduce the performance gaps observed between human rating groups. A human study comparing the original and corrected agents is therefore needed to evaluate whether the correction actually makes their behavior more human-like.
Questions for authors
-
Please refer to the concerns raised in the Weaknesses section.
-
Matching aggregate error rates does not necessarily mean that the model makes errors in the same positions as humans. Why does the correction target aggregate error rates rather than position-specific error probabilities for players at the corresponding rating? Is there evidence that the correction also improves this position-level agreement?
-
The proposed correction focuses on the move policy, but value estimates may also be important for modeling players at different skill levels. If applied to a system such as ALLIE, could correcting the policy without corresponding adjustments to value estimates disrupt other human-like behaviors, such as deciding when to resign? How would the authors assess whether these behaviors are preserved?
11 comments
Official comment by authors - part 2
Is there evidence that the correction also improves this position-level agreement?
Yes, there is. The paper already reports improved human-move cross-entropy (CE) at higher ratings. We now examine positions that are similar in expected difficulty, using 321,836 broader-Lichess positions at rating 2500. MCTS baseline and projection both use their ALLIE-calibrated settings, without recalibration on these positions.
For each position, we sum the raw policy's probabilities of mistakes and blunders, then group positions by that predicted error risk. Higher risk means greater difficulty to find good moves according to the model. Every policy is evaluated on exactly the same positions within each group; the table compares predicted mistake-or-blunder rates with observed human rates.
| Position difficulty group | Positions | Observed human error rate | Raw policy error rate | MCTS baseline error rate | ALLIE projection error rate |
|---|---|---|---|---|---|
| 0-5% | 170,680 | 0.61% | 1.13% | 0.93% | 0.73% |
| 5-10% | 33,975 | 4.27% | 7.26% | 5.06% | 5.53% |
| 10-20% | 36,807 | 8.89% | 14.45% | 9.92% | 11.10% |
| 20-30% | 21,800 | 14.91% | 24.70% | 17.06% | 15.96% |
| 30-40% | 15,190 | 21.28% | 34.73% | 24.38% | 22.60% |
| 40-50% | 11,321 | 28.84% | 44.81% | 31.98% | 29.56% |
| 50-60% | 8,897 | 36.27% | 54.82% | 39.99% | 37.47% |
| 60-70% | 7,341 | 42.98% | 64.78% | 48.51% | 46.43% |
| 70-80% | 5,958 | 53.09% | 74.86% | 57.91% | 57.12% |
| 80-90% | 5,141 | 60.69% | 84.86% | 68.05% | 72.45% |
| 90-100% | 4,726 | 75.35% | 95.07% | 86.20% | 91.13% |
Projection is closer to human error rates than MCTS in the medium-to-high risk range (20-80%), while MCTS baseline has an edge in the rare highest-risk positions (80-100%). This shows different strengths of the two approaches. However, both corrections still hugely overpredict errors in the highest-risk group, which highlights positions that should be focused for improvements.
In actual gameplay, the corrected agents still do not reproduce the performance gaps observed between human rating groups.
The remaining gap does not negate the improvements in human-move prediction and error-distribution agreement, but identifies a limitation that needs to be addressed. The paper’s game-phase analysis suggests one possible cause. The 30-second clock cutoff removes 45–53% of endgame positions, leaving practical endgames underrepresented in both training and calibration. Generated games show a large increase in endgame CPL, even though the projection closely matches human CPL within each phase of the filtered evaluation data. We suspect that this missing endgame behavior contributes to the remaining rating performance gap. Testing this explanation through better endgame coverage in training and phase-specific calibration is a natural direction for future work.
A human study comparing the original and corrected agents is therefore needed to evaluate whether the correction actually makes their behavior more human-like.
Our objective is to match the move distribution expected at a given rating, rather than maximize perceived human likeness. Human judgments are subjective: an accurate population distribution may appear less human-like than a model with distinctive quirks (e.g. aggressive style of play). Assessing rating-specific behavior also requires familiarity with the relevant rating population. A human study would provide useful evidence about perception, but would not directly establish whether the predicted move distributions match observed human behavior.
If applied to a system such as ALLIE, could correcting the policy without corresponding adjustments to value estimates disrupt other human-like behaviors, such as deciding when to resign? How would the authors assess whether these behaviors are preserved?
Correcting the policy would likely affect behavioral decisions that depend on its probabilities. ALLIE compares the resignation token’s probability with legal-move probabilities and also checks the value estimate. Our correction would leave the value estimate unchanged and retain the resignation token’s probability, but changing move probabilities could still change the resignation decision. We would assess this by comparing decisions on identical positions before and after correction.
Thank you for the clarification. I now have a clearer understanding of both the statistical analysis and the paper’s objective. I found the additional results for the 2500 rating group particularly informative and encourage you to include them in the paper.
I agree that matching the move distribution of players at a given rating is distinct from making an agent feel more human-like to its opponents. My remaining concern is how to interpret the trade-off even under the distributional objective. As Reviewer kkvQ noted, the correction improves agreement with human error patterns but worsens mHPD-Err relative to the uncorrected policy. My question about resignation in ALLIE raises a related but untested possibility: correcting move probabilities could also affect a behavior that was not targeted by the correction.
I do not expect every metric to improve. However, when measures of human behavior move in different directions, counting how many improve does not establish that the corrected policy is closer to human behavior overall. What makes the gain in error-distribution agreement more consequential than the loss in HPD calibration for the paper’s intended application? A human study could provide complementary evidence about how players experience this trade-off. Evidence that the correction better serves the stated application would also be persuasive.
Without stronger evidence that this trade-off improves the model for its intended use, I find the contribution too limited to support a claim of improved behavioral fidelity overall. The results more clearly establish that the correction brings the model closer to the specific human error statistics it was designed to match.
Official comment by authors - part 1
Thank you for the thoughtful and constructive review. Below, we provide new player-level and position-level results to address your concerns.
We agree. Our goal is to estimate the average move distribution expected from players at a given rating. Modeling individual players is a separate problem. A population average can miss unusual player profiles, and individual deviation alone does not establish non-human behavior.
The population model is useful precisely because players differ: a reliable rating-conditioned policy lets us study how a player differs from average expectations, identify strengths and weaknesses, and investigate what they could improve.
We conducted player-level analysis at rating 2500, where the raw-policy deficit is largest. We report players from the broader Lichess data with at least 20 games and 500 moves (61,578 positions), and a stricter group with at least 1,000 moves (32,348 positions). The table gives each class's player median and 25th-75th percentiles.
For each player, the reference is the class frequencies and mean CPL observed in that player's human moves. On exactly the same positions, we calculate the class frequencies and mean CPL expected under the raw policy, MCTS baseline and ALLIE projection. Both corrections use their ALLIE-calibrated settings. We then measure each policy's error against the player's observed values. Results show that ALLIE projection produces distributions that are closer to player class frequencies and mean CPLs.
Each position usually has only one observed human action, so its full human move distribution cannot be estimated directly. If the question instead refers to pooling similar types of positions (or positions that are similar in perceived difficulty), we do not currently have a validated way to do so and this is still an open problem. What we could try is to fit separate error targets for opening, middlegame and endgame positions; this remains future work.
This question also ties nicely into one of the main ideas for practical use of accurate behaviour models -> we can use them to try and find positions that are of similar difficulty to players at some given rating. While current raw policy models may still not be accurate enough, they can be useful for approximately grouping the positions by difficulty. Analysing these positions gives further evidence for position-level agreement (shown in the next part).