acceptodds
Under review as a conference paper at ICLR 2027

CASE: Computerized Adaptive Strength Estimator for Chess

Abstract

Accurately estimating a player's strength remains a challenging problem for adapting game AI to players of different strengths. Most existing approaches require players to complete or provide approximately 20–30 games before obtaining a reliable estimate, making the assessment time-consuming. Inspired by computerized adaptive testing (CAT), which sequentially selects informative questions according to the current ability estimate, we propose Computerized Adaptive Strength Estimator (CASE), a novel framework that estimates playing strength through adaptively selected board positions. Specifically, CASE uses an estimator to update the player's strength from the question-response history and a selector to choose the next question based on the current history and strength estimate. Experiments on chess show that CASE achieves over 80% rank accuracy with only 34 questions, whereas none of the compared baselines reaches this accuracy even after 200 questions. Compared with the prior game-based strength estimation method, CASE requires approximately 27 times fewer player responses to reach the same accuracy. Further analyses show that the selector identifies informative positions and adapts its question preferences according to the player's current strength estimate. These results demonstrate that accurate strength estimation can be achieved through a short adaptive assessment without requiring multiple complete games.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
strength estimation, chess, computerized adaptive testing, adaptive assessment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HMAKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CASE brings computerized adaptive testing to board-game strength estimation: instead of asking a player to complete or supply 20–30 games, it shows one position at a time, records the move, updates a rank estimate, and picks the next position accordingly. An estimator tokenizes each pair as (position, played move, value before, value after), encodes the sequence with a Transformer, and outputs a rank distribution. A selector encodes each candidate with the same board encoder, conditions on the history embedding plus the current rank embedding, and asks the candidate farthest from a fixed zero anchor. Because no candidate has a calibrated response model, informativeness labels are manufactured: hypothesize the simulated player's reply and supervise the selector on the resulting rank-posterior confidence gain. Training is staged: AlphaZero-style board/value pretraining, estimator pretraining, then joint training through simulated assessments. Chess instantiation uses eight Elo ranks and 48 Maia-style simulated players, four per rank for training and baseline calibration, two held out plus ALLIE for evaluation. CASE passes 80% rank accuracy at 34 questions while SE, CAT_bin and CAT_nom never reach it in 200, scoring 89.3% against 63.5% at 200; a reversed selector collapses to 14.4% against 12.5% chance.

Critical strengths
  • The selector supervision is the transferable contribution. Adaptive testing's practical bottleneck is the calibrated item bank, and Section 2.2 identifies it correctly as the reason CAT cannot absorb new questions. Deriving a per-question informativeness target from the rank-posterior confidence gain that a hypothesized response would produce gives a calibrated-bank-free substitute, and it costs only rollouts from an existing simulated player. I can see this reused well outside chess — any adaptive elicitation setting where a response model is expensive to estimate but cheap to simulate.

  • The mechanism is localized by ablation rather than only by end-to-end numbers. Selecting the lowest-scoring question instead of the highest drops accuracy to 14.4%, close to the 12.5% chance floor, which means the selector score genuinely separates informative from uninformative positions rather than merely being a fancy sampler. Section 4.4 then shows adaptivity rather than asserting it: two players of different estimated ranks are given the identical candidate set and their full score-rankings are compared by Spearman correlation, and the resulting matrix is strongest along the diagonal and decays with rank distance (Figure 4a), with a companion analysis over 1,000 high-scoring questions identifying which ranks prefer which positions. The candidate-pool scaling from 75.3% at 32 candidates to 89.3% at 1,024, while training used only 64, shows the ranking is meaningful rather than saturated. Appendix C.1 is also the honest version of the headline: per-evaluator spread for CASE is modest (97.7%, 85.8%, 84.4%) and CASE holds up on ALLIE, while CAT_bin falls to 16.4% there, barely above chance.

Critical weaknesses
  • The construct validity is untested — every ground truth is a simulated player. The 34-question/80% claim is rank accuracy over two held-out Maia-style networks plus ALLIE. The evaluation's "strength" is therefore which rank-stratified training set a network was fit on, and all three evaluators are neural policies. The paper's motivation in Section 1 is entirely about the practical burden on human players, and the abstract's "estimates playing strength" reads as human strength; nothing in the paper measures whether adaptive position-based testing recovers a human's Elo. This matters because the two confounds below become much harder to exclude without humans: a policy trained on a rank stratum has a move distribution that is tied to its data source in a way a person's is not, and CASE can plausibly reach high rank accuracy by identifying provenance rather than by measuring ability. A small human study, or a retrospective evaluation on publicly logged human games with known ratings, would change my score materially, and I do not think a new large-scale experiment is required — existing logs would serve.

  • Position provenance is a live confound that the design does not rule out. The question pool is built rank-stratified — 1.2 million positions sampled per rank from separate 200k-game sets, 9.6 million in total — and each step draws 1,024 candidates at random from it, so the candidate pool is balanced over source ranks in expectation and I cannot fault it there. The problem is upstream: because positions come from rank-partitioned game records, a position's features and the answerer's rank are not independent, and the estimator consumes the position itself, not only the response. Both components can therefore lean on a provenance shortcut — the selector preferring positions whose source stratum is currently in contention, the estimator reading "this looks like a position from a 2000-rated game" as evidence about the responder. The control that would settle it is a question pool whose source rank is uninformative: a single stratum, or positions generated independently of any rank partition. Until that is reported, I cannot attribute how much of the 89.3% is adaptive psychometrics and how much is source classification.

  • The 27× efficiency comparison converts two different effort units into one currency. "One question = one player response" is defensible as a count, but a CASE question is an isolated position in which the player selects a single move, without planning against an opponent's plan and without time pressure across a game, and the framework itself supplies v_before and v_after from the stage-one value head — so the position is effectively pre-assessed for the respondent, which a real game is not. SE's inputs are moves the player produced inside real games against a resisting opponent. The conversion also relies on an average of ~70 plies per game, i.e. ~35 evaluated responses, stated in prose rather than measured. Note also that the three baselines have different per-question costs — CAT_bin runs Stockfish on every candidate question while SE runs one forward pass per position — and none of this is accounted for. Reporting accuracy against elapsed assessment time, or against moves the respondent must actively compute, would support the practical claim far better than the response count does.

  • No stopping rule, and the operating point behind the headline is chosen by threshold intersection. Accuracy keeps rising to 200 questions and no baseline crosses 80%, so "34 questions" is where CASE's curve happens to cross an arbitrary line rather than a point at which the method is ready to stop. Section 4.2 reports that within-one-rank accuracy reaches 95% after five questions and 100% after 22, which suggests the genuinely useful operating point is far earlier and much cheaper to claim — the abstract would be better served by that figure than by the 80%-at-34 comparison. What the paper does not provide is any termination criterion, posterior-based or otherwise, so a real deployment cannot decide how many questions a given confidence level requires, and the reported numbers cannot be read as an accuracy-versus-budget frontier.

Questions for authors
  • Is any evaluation against human players possible, even small, or retrospective on publicly logged human games with known ratings? If CASE's rank accuracy on humans is materially below the 89.3% it reaches on simulated players, that reframes the whole result, and I would treat this as the central question. (Would change my score.)

  • Can you run the provenance control directly: restrict D_eval to positions from a single source stratum, or to a rank-independent pool, and report accuracy? Since candidates are already drawn uniformly over a rank-balanced pool, an imbalance cannot be blamed on sampling, so this is the clean way to test whether the estimator's access to the position itself is load-bearing. Please also report the source-rank distribution of the questions actually asked for each evaluation rank.

  • What does one completed assessment cost end to end? The estimator needs a board-encoder forward pass per answered question plus two value-head evaluations for v_before and v_after, and the selector scores 1,024 candidates at every step. How does that compare with CAT_bin, which runs Stockfish on every candidate question, and with SE, which needs a single pass per position? The paper reports accuracy curves against question count for all methods but compute per question for none of them, and I think the table would strengthen rather than weaken the case if it were added.

  • Why is informativeness read as distance to a fixed zero anchor rather than a learned scalar scorer? Section 3.1 motivates the anchor as a reference point, but I cannot tell from the paper whether that specific choice matters. An ablation against a plain MLP scorer would answer this cheaply.

  • How does the method behave under finer rank discretization, e.g. 16 or 32 bins? With eight ranks, ±1 accuracy hits 100% at 22 questions, which is reassuring; but adjacent Elo intervals of 200 points are wide, and the practical value of an adaptive test depends on the granularity at which it stays accurate.

  • Clarification: Section 4.1 gives 150 independent assessments per evaluation player over 24 evaluation players, but the unit of the shaded 95% band in Figure 2 is not stated. Since the same player contributes 150 sessions, the band cannot be a simple across-session interval without underestimating the variance attributable to the eight rank strata. How is it computed — across sessions, across ranks, or bootstrapped?

Reviewer_REtoOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

Estimating a chess player's rating usually requires reviewing 20 to 30 full games, or roughly 900 moves, which is too slow for quick matchmaking or onboarding. This paper proposes CASE, an adaptive testing system for chess that asks players to make single moves on selected board positions instead of playing complete games.

The method pairs two networks: an Estimator (an AlphaZero-style CNN with a Transformer that maps question history to an Elo bucket among 8 intervals of 200 points) and a Selector (which scores candidate positions by distance from an anchor embedding). Training follows three steps: pre-training on game records, supervised training of the rank estimator, and joint active rollouts trained with ListMLE on confidence gain.

Tested on simulated chess bots (Maia-style CNNs and ALLIE), CASE reaches 80% rank accuracy in 34 questions, compared to 910 moves for the passive SE baseline.

Critical strengths
  1. Adaptive testing without pre-calibrated item banks: Standard computerized adaptive testing requires calibrating item parameters on large participant pools beforehand, which cannot scale to the combinatorial space of chess positions. CASE formulates selection in an embedding space where informativeness is predicted on the fly, allowing adaptive probing across open, uncalibrated board positions.

  2. Substantial interaction reduction in simulation: In experiments with simulated players, CASE achieves 80% rank accuracy in 34 questions, reducing interaction length by roughly 27x compared to the previous move-level baseline.

Critical weaknesses
  1. Limited methodological novelty beyond conventional CAT: The overall framework follows the standard adaptive testing paradigm of estimating ability from response history and selecting subsequent questions based on the current estimate. The main methodological contribution appears to be the learned selector, which uses an anchor-based embedding distance trained through confidence gain. However, the paper does not sufficiently establish why this design is preferable to established information-based selection strategies or whether the proposed approach generalizes beyond chess. Stronger comparisons with standard selection methods and evidence of broader applicability would help clarify the methodological novelty.

  2. Train-test mismatch in candidate pool size (64 vs 1024): CASE is trained with a candidate pool of only 64 positions (Table 1), but evaluation inflates this pool to 1024 candidates (Line 314). Figure 3b shows that evaluating under the actual training candidate size (64) causes accuracy to drop by roughly 8 percentage points. In practical interactive use, evaluating 1024 candidate boards at every step requires 34,816 forward passes over a 34-question test, creating unmeasured computational latency.

Questions for authors
  1. How does CASE perform on real human players with known FIDE or Lichess ratings?

  2. How robust is the estimator to the behavioral diversity of human players, given that training relies on simulated policy networks?

  3. The overall framework follows the standard CAT paradigm of estimating ability from response history and selecting subsequent questions based on the current estimate. What is the specific methodological contribution beyond this conventional framework? In particular, can the authors demonstrate the advantages of the proposed anchor-based selector and confidence-gain training over established information-based selection strategies under a controlled comparison?

Reviewer_sxvVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CASE, which estimates a chess player's strength from a short sequence of selected positions instead of many full games. A Transformer estimator reads the question-response history and predicts one of eight 200-point Lichess blitz rating bands. A selector picks the next position and is trained to choose positions that most improve the estimate. All test-takers are models: two held-out Maia-style policy models per band and ALLIE. CASE reaches 80% exact-band accuracy after 34 questions and 89.3% after 200, against 63.5% for the strongest baseline (SE).

Critical strengths

S1. The problem is useful and the formulation is clean. Estimating strength from a short adaptive test, instead of 20 to 30 full games, would help game AI adapt to new players. Learning both the estimator and the selector removes the need for a calibrated item bank.

S2. The selector learns strength-dependent choices. Figure 4a shows that it ranks the same candidate positions differently for different current estimates. The t-SNE plots show a gradual shift in preferred positions across bands.

Critical weaknesses

W1. The evaluation does not show that CASE works for people. All test-takers are models. Players 5 and 6 are trained on disjoint games from the same Lichess pool, with the same bands and training procedure as the four training players (§4.1). Real players in one band vary much more than a model trained on that band. So the results show that CASE can identify which band a model imitates, not that it can estimate a person's strength. §5 notes this limitation, but the abstract and introduction do not mention that all players are simulated. The claims should be limited to simulated players, or supported with human data. (§4.1, §5)

W2. The paper does not show how much adaptive selection adds over random questions. The selection analyses only vary the selector's own choices: picking the lowest-scoring question gives 14.4%, and a smaller candidate pool gives lower accuracy (75.3% with 32 candidates) (§4.3). No analysis runs the CASE estimator on randomly sampled questions. SE does use randomly sampled questions (§4.2), so the gain over SE mixes a stronger estimator with adaptive selection, and it may come mainly from the estimator. (§4.2, §4.3)

W3. The paper misses a closely related line of work on inferring player properties from limited behavior. Behavioral stylometry identifies a player among thousands of candidates given 100 labeled games [1]. Individual behavior models such as Maia4All learn a model of one player from 20 games [2]. Knowing who a player is, or having a model of that player, also gives the player's strength, so strength estimation is a coarser version of these tasks. §2.1 covers only prior strength estimation methods and §2.2 covers CAT from education. The paper should position CASE against this line. (§2.1)

[1] Reid McIlroy-Young et al. Detecting Individual Decision-Making Style: Exploring Behavioral Stylometry in Chess. NeurIPS 2021.

[2] Zhenwei Tang et al. Learning to Imitate with Less: Efficient Individual Behavior Modeling in Chess. TMLR 2026.

Questions for authors

Q1 (clarification). Do the simulated players answer by sampling from their policy or by taking the top move? If they take the top move, each band answers deterministically, which makes the task much easier than with human players.

Q2 (additional result). What is the accuracy of the CASE estimator when questions are sampled at random, under the same protocol as Figure 2?

Q3 (clarification). A player's move also depends on the opponent, and human-like models such as Maia-2 and ALLIE take both the player's and the opponent's rating as input. Which opponent rating was given to ALLIE in each band? When training the simulated players, was a game assigned to a band by the mover's rating only? A CASE test has no opponent, so it matters how this was set.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.