acceptodds
Under review as a conference paper at ICLR 2027

Search Trees and Human Perception of Creativity

Abstract

We present empirical evidence that human perception of creativity is correlated with how difficult the search problem for the creative solution is. We use a large dataset of high-quality annotations of chess moves as "brilliant", as well as present a novel dataset of human reactions to Wordle games. In both the analysis of expert-annotated brilliant moves in chess and the small-scale analysis of Wordle, we show that when a play requires a particular shape of search tree, it is more likely to be perceived as brilliant or creative, over and above the quality of the move in the case of chess. Although human creativity is complex and multifaceted, our work indicates that the mathematical structure of the search problem being solved is likely part of what we think of as creativity, across at least two domains.

Primary area:
applications to neuroscience & cognitive science
Keywords:
search trees, creativity, chess, board games, wordle, computational creativity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rRkfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether human perception of creativity is tied to the shape of the underlying game search tree in chess and Wordle. In chess, it applies the search-tree feature pipeline and AggReduce classifier of Zaidi & Guerzhoy (2024) to expert annotations from Chess Informant, reaching 85.1% balanced accuracy. In Wordle, it filters r/Wordle comments to find 173 games praised as creative and runs a logistic regression on 1-step and 2-step lookahead entropy drops (p = 0.047).

Critical strengths
  1. The expert-labeled chess data is a real improvement. Moving from crowd-annotated Lichess studies to Chess Informant gives 5,278 brilliant, 8,591 good and 6,192 unlabeled moves (L095-106), and I like that good and unlabeled moves are reported separately, which shows that good moves are the harder negatives (Table 1).

  2. The transfer and single-feature results make the hypothesis concrete. The Chess Informant model reaches 75.4% on Lichess while the reverse direction gets 68.7% (L149), and the single-feature analysis points to a wide parent tree followed by a narrow post-move tree (Table 2), which is an intuitive and testable pattern.

Critical weaknesses
  1. The chess analysis does not show that tree shape matters beyond move quality. In Section 4.1.2 (Table 2, L184-191), a wide tree before the move and a narrow tree after it is just what happens when a check or capture forces a single reply, and the paper only runs single-feature regressions (L181) without controlling for move strength (L015). Comparing candidate moves within the same board position, or controlling for engine evaluation, is needed to separate creative moves from ordinary forcing tactics.

  2. The Wordle study is too weak to support the claim in a second domain. Positive games are picked from comments praising general strategy (L265-269) while negatives are sampled at random (L386-388), and without controlling for the number of guesses, the 2-step entropy drop barely reaches p = 0.047 on 173 positive games while the 1-step drop has no effect (p = 0.841, Table 3, L393). With 173 positives, a coefficient this weak implies a small effect, so the paper should explain how it reaches 83.7% balanced accuracy (L401-404).

Questions for authors
  1. Does tree shape still predict "!!" after controlling for move quality (L015), for example when comparing candidate moves within the same board position?
  2. How is the 83.7% accuracy (L401-404) computed from Table 3, how are the 1-step and 2-step entropy drops defined given the sign mismatch between L372 and L383, and does the 2-step drop stay significant when controlling for the number of guesses?
  3. Which chess analyses are new beyond Zaidi & Guerzhoy (2024)? The paper notes that it follows their pipeline on a cleaner dataset (L035-036, L110-136), and Sections 4.1.1 and 4.1.3 also appear to repeat analyses from that work.
2 comments

Thank you for the questions!

(1/2)

W1 / Q1 — tree shape beyond move quality, and forcing moves. Thank you for this comment -- we agree it was a gap. We now control for move quality and for forcing moves explicitly (details in the general response).

  • Controls (odds ratio per SD, brilliant vs good, game-clustered SEs):

    FeatureAloneWith all quality and forcing controls
    Branching factor of the post-move tree (Maia, 10²)0.580.67 [0.64, 0.70]
    Width at depth 20.560.63 [0.60, 0.66]
    Max width0.630.67 [0.64, 0.71]
    Mean width0.640.68 [0.65, 0.72]

    We get all p < 0.001. Brilliant moves are indeed more often checks or captures (30% vs 16% of good moves). However, restricting to quiet moves (no check or capture) that the strong engine also rates as best (2,449 vs 4,206) leaves the effects essentially unchanged (0.59–0.65, all p < 0.001). The pattern is therefore not explained by forcing tactics or by move quality.

  • Predictive version. With a gradient-boosted classifier, adding tree-shape features to all evaluation and forcing features raises AUC from 0.794 to 0.855 (+0.060 [0.056, 0.065]).

  • Within-position comparison (Q1), as you suggested. For 413 "!!" and 413 "!" positions, the annotated move and the strong engine's top four alternatives were each searched from their resulting positions with a fixed node budget (10² and 10³ nodes, both networks). We fitted a conditional logit with position fixed effects, controlling for the candidate's evaluation, check, capture and number of legal replies.

    • Within a position, the "!!" move leads to a narrower and deeper tree than its alternatives:

      • mean width OR 0.54 per SD [0.44, 0.66] (LC0, 10³ nodes);
      • max width 0.60 [0.49, 0.73] (Maia, 10³ nodes);
      • width at depth 2 0.65 [0.54, 0.78] (Maia, 10² nodes);
      • height 1.72 [1.43, 2.05] (LC0, 10³ nodes).

      All p < 0.001. They hold among quiet moves (288 positions), e.g. mean width 0.64, height 1.46, p < 0.001.

    • The "!!" move is also much less likely under Maia's human-like policy than its alternatives (OR 0.30 [0.22, 0.40]).

    • Equally good alternatives only. We kept only alternatives the strong engine rates as equally good (|ΔQ| ≤ 0.05; 124 "!!" positions). The "!!" move is still narrower and deeper: mean width 0.58 [0.42, 0.81], height 1.83 [1.24, 2.70]. It is also still far less likely under Maia's policy: 0.35 [0.23, 0.53].

    • For "!" moves, the shape effects are weaker and the policy effect reverses: "!" moves are the more expected ones. The "!!"-vs-"!" interaction is significant for width (p = 0.002), height (p = 0.009) and Maia's policy (p < 10⁻⁸). So the within-position pattern is specific to brilliance, not to good moves in general.

    • One Table 2 feature, branching factor, does not replicate within position, and we will say so.

  • Search effort vs. shape. Your question also prompted a refinement we will report. At low node budgets, part of the "narrow tree after the move" signal reflects how few visits the human-like engine spends on the move: controlling for those visits removes the branching-factor and max-width effects, though width at depth 2 survives. Two other things survive all controls, including visits:

    • a deeper post-move tree;
    • narrower post-move and parent trees in the strong engine's deep (10⁵) search (OR 0.80 and 0.66).

    We will revise §4.1.2 accordingly: brilliance is associated with moves that a human-like search under-explores, and with narrow, deep lines in a strong search, beyond move quality.

(2/2)

W2 / Q2 — Wordle. We agree the Wordle evidence is weaker than the chess evidence. We will present it explicitly as preliminary and soften the abstract accordingly.

  1. The 83.7% figure is wrong, and we retract it. Thank you for catching this. A coefficient with z = 1.99 on 173 vs 1,000 games corresponds to a standardized group difference of about 0.16, which implies an AUC of roughly 0.55, not 83.7% balanced accuracy, so couldn't be right. The revision will report cross-validated balanced accuracy and AUC rather than in-sample accuracy. This does not change the finding itself. The pre-specified test shows a positive association between the 2-step entropy drop and perceived creativity, holding the 1-step drop constant (β = 2.91 per bit, p = 0.047). The effect is small in predictive terms, which we would expect for two reasons: label noise attenuates effects, and perceived creativity has many determinants. Our claim is that the association exists, not that lookahead alone predicts which games are perceived as creative.
  2. Definitions and sign. The "−log" at L383 is a typo. Entropy is H_t = log₂ N_t, as defined at L372. The k-step drop is ΔH_k = H_t − E[H_{t+k}] under k-step lookahead, maximised over the guesses in the game. Drops are therefore nonnegative, larger means more information gained, and β₂ > 0 means that larger 2-step drops go with perceived creativity. We have rewritten §5.3 with explicit formulas.
  3. Number of guesses. We agree this is an important control, and we will add it (as a covariate and as fixed effects) in the revision.
  4. Negatives. As stated at L386–388, the negatives are not random games. They are randomly sampled from games that also received comments, but whose comments did not express perceived creativity. This controls for a game being commented on at all. We will make it explicit in §5.1, where L359 currently says only "We sample 1,000 games not perceived as creative."

Q3 — what's new beyond Zaidi & Guerzhoy (2024). The feature pipeline and the AggReduce architecture are reused unchanged from the prior work, which we will state explicitly. The following are new:

  • Data. 5,278 expert-annotated "!!" moves from Chess Informant, vs 820 crowd-annotated Lichess moves in the prior work. We also add cross-dataset transfer in both directions (Table 1), which shows that the expert-trained model generalises better.
  • Hard negatives. We separate good ("!") from unlabeled negatives; the prior work reported only brilliant vs. not. This shows that "!" moves are the hard negatives (72.0% vs 94.1% accuracy).
  • §4.1.2. Which individual tree-shape features carry the signal, and in which direction. The prior work showed only that all features beat win-chance-only features in a logistic regression (74% vs 61%).
  • §4.1.1. Per-class rates of the move being the engine's best across search depths and networks. The prior work used is-best only as a classifier input (64%). With the new CIs and tests, it shows a significant crossover: "!!" moves are less often best than "!" moves under shallow or human-like search, and more often under deep search.
  • New controlled analyses. Move quality and forcing controls, nested models, and the within-position comparison (above).
  • Wordle, a second domain.

§4.1.3 is a replication of the prior work's Maia-perturbation experiment, on expert data. What's new there is the per-class breakdown: the largest effect is for "!" moves (+41.0%), vs +33.3% for "!!" and +13.2% for unlabeled. We will label it as a replication.

Reviewer_fh83OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the relationship between search-tree structure and human perception of creativity in chess and Wordle.

Critical strengths
  • This paper explores an interesting reseach question.
Critical weaknesses
  • The Wordle experiment contains only 173 positive examples, and the key result is marginally significant (p = 0.047). Moreover, the one-step and two-step entropy features are highly correlated, making it difficult to conclude that deeper lookahead is a better predictor of perceived creativity.
  • The search trees are generated by chess engines, while they are not observed from human reasoning. Therefore, I think the results establish correlations between engine-derived features and human judgments.
Questions for authors

Please see the weakness above.

1 comment

Thank you for the review!

Wordle sample size and p = 0.047. We agree. The Wordle result is preliminary, and the paper's main evidence is the chess analysis, which we have substantially strengthened (see the general response). We will present Wordle as a preliminary second domain, soften the abstract, and remove the incorrect 83.7% accuracy figure (see our response to rRkf).

Correlation between the 1-step and 2-step drops. At r = 0.8, the variance inflation factor is 1/(1 − 0.8²) ≈ 2.8. This widens the standard errors but does not bias the estimates. The Table 3 test of the 2-step coefficient, with the 1-step drop in the model (z = 1.99, p = 0.047), already tests whether the 2-step drop adds information beyond the 1-step drop. We will report it as weak evidence that it does. We will not claim that 2-step lookahead is a "better predictor" in general; the revision will add each predictor alone and a likelihood-ratio test.

Engine search trees vs human reasoning. Agreed. Our claims concern features of the search problem, measured by engines, not human cognitive processes, and we will revise the wording throughout (e.g. "engine-derived search-tree features"). We partly address this by using Maia, a network trained to predict human (1900-rated) moves. The strongest signals involve how Maia explores, and how likely Maia's policy is to play the move: brilliant moves are ones a human-like search under-explores. We will discuss this limitation explicitly.

Presentation. We have added a diagram of the feature construction (§3.2), CIs for Fig. 1, and a clearer description of the Wordle quantities.

Reviewer_LwNNOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper finds that the human perception of creativity (or brilliance) of the solution is correlated with how difficult the search problem/path is during problem solving.

Critical strengths

Great paper. Very interesting.

Critical weaknesses

I have two theoretical issues with the paper.

  1. Why is a brilliant chess move per se classified as creative? Are you not conflating problem solving ability with creativity? In problem solving and for reasoning more generally, complexity (i.e. item difficulty + response strategy) are already known predictors of ability. Same goes for Wordle. I didn’t see comments that made me think this was really about creativity, but more about problem solving strategies. → you address this in the discussion, so I am ok with this.
  2. The argument about features of the search space being predictive of perceived creativity seems a bit circular. Please correct me if I’m wrong, but I think this is because Boden’s work also points to the size of the search space helping define whether a task is creative or not. A huge search space means that it is intractable to find an optimal search strategy and we therefore need to use creative strategies and insights in order to solve it. So indicators of search space seem to be both predictor of degree of creativity required for the problem and how creative the problem is perceived to be. → perhaps this conundrum can be mentioned in the discussion
Questions for authors

Clarification requests: 3.2 Feature Engineering, especially first paragraph would benefit from a figure demonstrating how the different trees were created.

4.1.1 Brilliance Vs Best Move, can you report standard errors and/or run statistical analyses to substantiate your claims? Similarly: “ class balancing with 50% weight towards brilliant moves, and 25% weight towards each of good and unlabeled, summing to 50% weight on non-brilliant moves” → why not 33.33%, 33.33%, 33.33%? sensitivity test: if you change the proportions does this change the results? OR you could just correct for this mismatch with proper statistical analyses of effects

Perhaps interesting to look into: I see parallels with other creativity research, e.g. on the building of tension and resolving this in literary texts with this part of your paper: “ a brilliant move may be one which has a wide search tree before the move, but a thin one following the move, suggesting brilliance in chess may partially be finding the narrow path to victory among many possible paths”

3 comments

Thank you very much for the encouraging review!

Figure for §3.2. Will be added (diagram of the parent tree, the post-move subtree, the four sibling groups with their aggregations, and how the 3,980 features are assembled), thank you.

Standard errors for §4.1.1. Will be added: Wilson 95% CIs, two-proportion z-tests, and game-clustered logits.

  • At 10⁵ LC0 nodes: brilliant 67.8% [66.6, 69.1] vs good 59.6% [58.5, 60.6] (z = 9.8, p < 10⁻²¹) vs unlabeled 55.1% [53.9, 56.4].
  • There is a significant crossover. Under shallow or human-like search (Maia at 10–10³ nodes, LC0 at 10 nodes), brilliant moves are less often the best move than good moves (e.g. 9.3% vs 20.1% for Maia at 10 nodes, z = −16.9), and under deep search they are more often best.
  • Fig. 1 now shows CIs.

Why 50/25/25 rather than 1/3 each; sensitivity.

  • Why 50/25/25. It makes the brilliant/non-brilliant problem balanced (chance = 50%) while weighting the two negative types equally. Under 1/3 each, the trivial "never brilliant" classifier already scores 66.7%.
  • Network. Retraining the network under 50/25/25, 1/3 each, 50/40/10, 50/10/40 and unweighted gives essentially identical AUCs: 0.866–0.874 for brilliant vs good and 0.966–0.972 for brilliant vs unlabeled. The weighting only shifts the operating point between brilliant recall and good-move accuracy.
  • Single features. The top feature (Maia 10² post-move branching factor) is ranked first or second under every weighting, and 7–10 of the top 10 features are tree-shape features under every weighting. Feature rankings correlate at ρ = 0.94 and 0.98 (50/40/10, 50/10/40), and at ρ = 0.89 with a weighting-free AUC ranking.

Creativity vs problem solving; circularity with Boden. We will expand the discussion on both. On circularity: our claim is not that large search spaces require creativity. It is that, holding the domain fixed and controlling for move quality, the specific shape of the search problem for a given solution predicts whether humans call that solution creative. The new controlled analyses make that distinction sharper. We will note the conceptual overlap with Boden's view of search spaces explicitly.

Tension and resolution. Thank you; we will add the parallel (wide before, narrow after ≈ building and resolving tension) to the discussion, with references to the literary-tension literature.

Many thanks again.

Great work! Get well soon and good luck! I'll check my scores and re-rate where needed.

Thank you!

Comments on the paper

2 comments

We apologize for this late comment. The lead author has been ill and (separately) one of our laptops died. Please find our responses here. We are sorry we missed the discussion. We appreciate all the reviews.

We thank the reviewers. In response, we ran the following new analyses, which we will add to the paper:

We will be adding the following analyses to the paper

  1. Tree shape beyond move quality (rRkf). On the expert data (5,278 "!!" vs 8,591 "!"), we controlled for:

    • the engine evaluations of the move and of the best move, is-best and in-tree, with both networks at 10², 10³ and 10⁵ nodes, plus the position evaluation;
    • forcing features: check, capture, captured value, promotion, number of legal replies, number of legal moves, moved piece left en prise, and material.

    With all of these controls, the top shape features from Table 2 remain strongly predictive, with odds ratios per SD of 0.63–0.68 (all p < 0.001). They also remain predictive when restricted to moves the strong engine rates as best and that are neither checks nor captures (0.59–0.65, p < 0.001). Predictively, adding tree-shape features to a gradient-boosted model with all 1,100 evaluation features and the forcing features raises brilliant-vs-good AUC from 0.794 to 0.855 (+0.060, 95% CI [0.056, 0.065]). Shape features alone (0.845) outperform evaluation features alone (0.787).

  2. Within-position comparison (rRkf Q1). For 413 "!!" and 413 "!" positions (a fuller analyis will be run for the camera ready if accepted, but the numbers here seem definitive), we compared the annotated move with the strong engine's top four alternatives in the same position. Each candidate was searched from its resulting position with a fixed node budget, and we fitted a conditional logit with position fixed effects, controlling for each candidate's evaluation and for check, capture and number of legal replies. Within the same position, the "!!" move leads to a narrower and deeper search tree than its alternatives. For example, mean width (LC0, 10³ nodes) has OR 0.54 per SD [0.44, 0.66], and height has OR 1.72 [1.43, 2.05]; both p < 0.001, also among quiet moves. The "!!" move is also far less likely under Maia's human-like policy (OR 0.30 [0.22, 0.40]). These effects are weaker for "!" moves, and the "!!"-vs-"!" difference is significant (p = 0.002 for width; p < 10⁻⁸ for Maia's prior), so the pattern is specific to brilliance.

  3. Statistics for §4.1.1 (LwNN). We added CIs and tests. The brilliant/good difference is significant (e.g. 67.8% vs 59.6%, p < 10⁻²¹), with a significant crossover: brilliant moves are less often best under shallow or human-like search, and more often under deep search.

  4. Class-weight sensitivity (LwNN). Changing the 50/25/25 weighting moves the operating point but not the ranking ability: AUC 0.866–0.874 (brilliant vs good) across five weightings.

  5. Wordle (rRkf, fh83). We retract the incorrect 83.7% accuracy figure, fix the entropy definitions, describe how the negatives were chosen, and present Wordle explicitly as a preliminary second domain.

AuthorsOpenReview ↗

Revised pdf uploaded

New text is in red.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.