- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether human perception of creativity is tied to the shape of the underlying game search tree in chess and Wordle. In chess, it applies the search-tree feature pipeline and AggReduce classifier of Zaidi & Guerzhoy (2024) to expert annotations from Chess Informant, reaching 85.1% balanced accuracy. In Wordle, it filters r/Wordle comments to find 173 games praised as creative and runs a logistic regression on 1-step and 2-step lookahead entropy drops (p = 0.047).
Critical strengths
-
The expert-labeled chess data is a real improvement. Moving from crowd-annotated Lichess studies to Chess Informant gives 5,278 brilliant, 8,591 good and 6,192 unlabeled moves (L095-106), and I like that good and unlabeled moves are reported separately, which shows that good moves are the harder negatives (Table 1).
-
The transfer and single-feature results make the hypothesis concrete. The Chess Informant model reaches 75.4% on Lichess while the reverse direction gets 68.7% (L149), and the single-feature analysis points to a wide parent tree followed by a narrow post-move tree (Table 2), which is an intuitive and testable pattern.
Critical weaknesses
-
The chess analysis does not show that tree shape matters beyond move quality. In Section 4.1.2 (Table 2, L184-191), a wide tree before the move and a narrow tree after it is just what happens when a check or capture forces a single reply, and the paper only runs single-feature regressions (L181) without controlling for move strength (L015). Comparing candidate moves within the same board position, or controlling for engine evaluation, is needed to separate creative moves from ordinary forcing tactics.
-
The Wordle study is too weak to support the claim in a second domain. Positive games are picked from comments praising general strategy (L265-269) while negatives are sampled at random (L386-388), and without controlling for the number of guesses, the 2-step entropy drop barely reaches p = 0.047 on 173 positive games while the 1-step drop has no effect (p = 0.841, Table 3, L393). With 173 positives, a coefficient this weak implies a small effect, so the paper should explain how it reaches 83.7% balanced accuracy (L401-404).
Questions for authors
- Does tree shape still predict "!!" after controlling for move quality (L015), for example when comparing candidate moves within the same board position?
- How is the 83.7% accuracy (L401-404) computed from Table 3, how are the 1-step and 2-step entropy drops defined given the sign mismatch between L372 and L383, and does the 2-step drop stay significant when controlling for the number of guesses?
- Which chess analyses are new beyond Zaidi & Guerzhoy (2024)? The paper notes that it follows their pipeline on a cleaner dataset (L035-036, L110-136), and Sections 4.1.1 and 4.1.3 also appear to repeat analyses from that work.
2 comments
Part 2
(2/2)
W2 / Q2 — Wordle. We agree the Wordle evidence is weaker than the chess evidence. We will present it explicitly as preliminary and soften the abstract accordingly.
- The 83.7% figure is wrong, and we retract it. Thank you for catching this. A coefficient with z = 1.99 on 173 vs 1,000 games corresponds to a standardized group difference of about 0.16, which implies an AUC of roughly 0.55, not 83.7% balanced accuracy, so couldn't be right. The revision will report cross-validated balanced accuracy and AUC rather than in-sample accuracy. This does not change the finding itself. The pre-specified test shows a positive association between the 2-step entropy drop and perceived creativity, holding the 1-step drop constant (β = 2.91 per bit, p = 0.047). The effect is small in predictive terms, which we would expect for two reasons: label noise attenuates effects, and perceived creativity has many determinants. Our claim is that the association exists, not that lookahead alone predicts which games are perceived as creative.
- Definitions and sign. The "−log" at L383 is a typo. Entropy is H_t = log₂ N_t, as defined at L372. The k-step drop is ΔH_k = H_t − E[H_{t+k}] under k-step lookahead, maximised over the guesses in the game. Drops are therefore nonnegative, larger means more information gained, and β₂ > 0 means that larger 2-step drops go with perceived creativity. We have rewritten §5.3 with explicit formulas.
- Number of guesses. We agree this is an important control, and we will add it (as a covariate and as fixed effects) in the revision.
- Negatives. As stated at L386–388, the negatives are not random games. They are randomly sampled from games that also received comments, but whose comments did not express perceived creativity. This controls for a game being commented on at all. We will make it explicit in §5.1, where L359 currently says only "We sample 1,000 games not perceived as creative."
Q3 — what's new beyond Zaidi & Guerzhoy (2024). The feature pipeline and the AggReduce architecture are reused unchanged from the prior work, which we will state explicitly. The following are new:
- Data. 5,278 expert-annotated "!!" moves from Chess Informant, vs 820 crowd-annotated Lichess moves in the prior work. We also add cross-dataset transfer in both directions (Table 1), which shows that the expert-trained model generalises better.
- Hard negatives. We separate good ("!") from unlabeled negatives; the prior work reported only brilliant vs. not. This shows that "!" moves are the hard negatives (72.0% vs 94.1% accuracy).
- §4.1.2. Which individual tree-shape features carry the signal, and in which direction. The prior work showed only that all features beat win-chance-only features in a logistic regression (74% vs 61%).
- §4.1.1. Per-class rates of the move being the engine's best across search depths and networks. The prior work used is-best only as a classifier input (64%). With the new CIs and tests, it shows a significant crossover: "!!" moves are less often best than "!" moves under shallow or human-like search, and more often under deep search.
- New controlled analyses. Move quality and forcing controls, nested models, and the within-position comparison (above).
- Wordle, a second domain.
§4.1.3 is a replication of the prior work's Maia-perturbation experiment, on expert data. What's new there is the per-class breakdown: the largest effect is for "!" moves (+41.0%), vs +33.3% for "!!" and +13.2% for unlabeled. We will label it as a replication.
Thank you
Thank you for the questions!
(1/2)
W1 / Q1 — tree shape beyond move quality, and forcing moves. Thank you for this comment -- we agree it was a gap. We now control for move quality and for forcing moves explicitly (details in the general response).
Controls (odds ratio per SD, brilliant vs good, game-clustered SEs):
We get all p < 0.001. Brilliant moves are indeed more often checks or captures (30% vs 16% of good moves). However, restricting to quiet moves (no check or capture) that the strong engine also rates as best (2,449 vs 4,206) leaves the effects essentially unchanged (0.59–0.65, all p < 0.001). The pattern is therefore not explained by forcing tactics or by move quality.
Predictive version. With a gradient-boosted classifier, adding tree-shape features to all evaluation and forcing features raises AUC from 0.794 to 0.855 (+0.060 [0.056, 0.065]).
Within-position comparison (Q1), as you suggested. For 413 "!!" and 413 "!" positions, the annotated move and the strong engine's top four alternatives were each searched from their resulting positions with a fixed node budget (10² and 10³ nodes, both networks). We fitted a conditional logit with position fixed effects, controlling for the candidate's evaluation, check, capture and number of legal replies.
Within a position, the "!!" move leads to a narrower and deeper tree than its alternatives:
All p < 0.001. They hold among quiet moves (288 positions), e.g. mean width 0.64, height 1.46, p < 0.001.
The "!!" move is also much less likely under Maia's human-like policy than its alternatives (OR 0.30 [0.22, 0.40]).
Equally good alternatives only. We kept only alternatives the strong engine rates as equally good (|ΔQ| ≤ 0.05; 124 "!!" positions). The "!!" move is still narrower and deeper: mean width 0.58 [0.42, 0.81], height 1.83 [1.24, 2.70]. It is also still far less likely under Maia's policy: 0.35 [0.23, 0.53].
For "!" moves, the shape effects are weaker and the policy effect reverses: "!" moves are the more expected ones. The "!!"-vs-"!" interaction is significant for width (p = 0.002), height (p = 0.009) and Maia's policy (p < 10⁻⁸). So the within-position pattern is specific to brilliance, not to good moves in general.
One Table 2 feature, branching factor, does not replicate within position, and we will say so.
Search effort vs. shape. Your question also prompted a refinement we will report. At low node budgets, part of the "narrow tree after the move" signal reflects how few visits the human-like engine spends on the move: controlling for those visits removes the branching-factor and max-width effects, though width at depth 2 survives. Two other things survive all controls, including visits:
We will revise §4.1.2 accordingly: brilliance is associated with moves that a human-like search under-explores, and with narrow, deep lines in a strong search, beyond move quality.