- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CASE brings computerized adaptive testing to board-game strength estimation: instead of asking a player to complete or supply 20–30 games, it shows one position at a time, records the move, updates a rank estimate, and picks the next position accordingly. An estimator tokenizes each pair as (position, played move, value before, value after), encodes the sequence with a Transformer, and outputs a rank distribution. A selector encodes each candidate with the same board encoder, conditions on the history embedding plus the current rank embedding, and asks the candidate farthest from a fixed zero anchor. Because no candidate has a calibrated response model, informativeness labels are manufactured: hypothesize the simulated player's reply and supervise the selector on the resulting rank-posterior confidence gain. Training is staged: AlphaZero-style board/value pretraining, estimator pretraining, then joint training through simulated assessments. Chess instantiation uses eight Elo ranks and 48 Maia-style simulated players, four per rank for training and baseline calibration, two held out plus ALLIE for evaluation. CASE passes 80% rank accuracy at 34 questions while SE, CAT_bin and CAT_nom never reach it in 200, scoring 89.3% against 63.5% at 200; a reversed selector collapses to 14.4% against 12.5% chance.
Critical strengths
-
The selector supervision is the transferable contribution. Adaptive testing's practical bottleneck is the calibrated item bank, and Section 2.2 identifies it correctly as the reason CAT cannot absorb new questions. Deriving a per-question informativeness target from the rank-posterior confidence gain that a hypothesized response would produce gives a calibrated-bank-free substitute, and it costs only rollouts from an existing simulated player. I can see this reused well outside chess — any adaptive elicitation setting where a response model is expensive to estimate but cheap to simulate.
-
The mechanism is localized by ablation rather than only by end-to-end numbers. Selecting the lowest-scoring question instead of the highest drops accuracy to 14.4%, close to the 12.5% chance floor, which means the selector score genuinely separates informative from uninformative positions rather than merely being a fancy sampler. Section 4.4 then shows adaptivity rather than asserting it: two players of different estimated ranks are given the identical candidate set and their full score-rankings are compared by Spearman correlation, and the resulting matrix is strongest along the diagonal and decays with rank distance (Figure 4a), with a companion analysis over 1,000 high-scoring questions identifying which ranks prefer which positions. The candidate-pool scaling from 75.3% at 32 candidates to 89.3% at 1,024, while training used only 64, shows the ranking is meaningful rather than saturated. Appendix C.1 is also the honest version of the headline: per-evaluator spread for CASE is modest (97.7%, 85.8%, 84.4%) and CASE holds up on ALLIE, while CAT_bin falls to 16.4% there, barely above chance.
Critical weaknesses
-
The construct validity is untested — every ground truth is a simulated player. The 34-question/80% claim is rank accuracy over two held-out Maia-style networks plus ALLIE. The evaluation's "strength" is therefore which rank-stratified training set a network was fit on, and all three evaluators are neural policies. The paper's motivation in Section 1 is entirely about the practical burden on human players, and the abstract's "estimates playing strength" reads as human strength; nothing in the paper measures whether adaptive position-based testing recovers a human's Elo. This matters because the two confounds below become much harder to exclude without humans: a policy trained on a rank stratum has a move distribution that is tied to its data source in a way a person's is not, and CASE can plausibly reach high rank accuracy by identifying provenance rather than by measuring ability. A small human study, or a retrospective evaluation on publicly logged human games with known ratings, would change my score materially, and I do not think a new large-scale experiment is required — existing logs would serve.
-
Position provenance is a live confound that the design does not rule out. The question pool is built rank-stratified — 1.2 million positions sampled per rank from separate 200k-game sets, 9.6 million in total — and each step draws 1,024 candidates at random from it, so the candidate pool is balanced over source ranks in expectation and I cannot fault it there. The problem is upstream: because positions come from rank-partitioned game records, a position's features and the answerer's rank are not independent, and the estimator consumes the position itself, not only the response. Both components can therefore lean on a provenance shortcut — the selector preferring positions whose source stratum is currently in contention, the estimator reading "this looks like a position from a 2000-rated game" as evidence about the responder. The control that would settle it is a question pool whose source rank is uninformative: a single stratum, or positions generated independently of any rank partition. Until that is reported, I cannot attribute how much of the 89.3% is adaptive psychometrics and how much is source classification.
-
The 27× efficiency comparison converts two different effort units into one currency. "One question = one player response" is defensible as a count, but a CASE question is an isolated position in which the player selects a single move, without planning against an opponent's plan and without time pressure across a game, and the framework itself supplies v_before and v_after from the stage-one value head — so the position is effectively pre-assessed for the respondent, which a real game is not. SE's inputs are moves the player produced inside real games against a resisting opponent. The conversion also relies on an average of ~70 plies per game, i.e. ~35 evaluated responses, stated in prose rather than measured. Note also that the three baselines have different per-question costs — CAT_bin runs Stockfish on every candidate question while SE runs one forward pass per position — and none of this is accounted for. Reporting accuracy against elapsed assessment time, or against moves the respondent must actively compute, would support the practical claim far better than the response count does.
-
No stopping rule, and the operating point behind the headline is chosen by threshold intersection. Accuracy keeps rising to 200 questions and no baseline crosses 80%, so "34 questions" is where CASE's curve happens to cross an arbitrary line rather than a point at which the method is ready to stop. Section 4.2 reports that within-one-rank accuracy reaches 95% after five questions and 100% after 22, which suggests the genuinely useful operating point is far earlier and much cheaper to claim — the abstract would be better served by that figure than by the 80%-at-34 comparison. What the paper does not provide is any termination criterion, posterior-based or otherwise, so a real deployment cannot decide how many questions a given confidence level requires, and the reported numbers cannot be read as an accuracy-versus-budget frontier.
Questions for authors
-
Is any evaluation against human players possible, even small, or retrospective on publicly logged human games with known ratings? If CASE's rank accuracy on humans is materially below the 89.3% it reaches on simulated players, that reframes the whole result, and I would treat this as the central question. (Would change my score.)
-
Can you run the provenance control directly: restrict D_eval to positions from a single source stratum, or to a rank-independent pool, and report accuracy? Since candidates are already drawn uniformly over a rank-balanced pool, an imbalance cannot be blamed on sampling, so this is the clean way to test whether the estimator's access to the position itself is load-bearing. Please also report the source-rank distribution of the questions actually asked for each evaluation rank.
-
What does one completed assessment cost end to end? The estimator needs a board-encoder forward pass per answered question plus two value-head evaluations for v_before and v_after, and the selector scores 1,024 candidates at every step. How does that compare with CAT_bin, which runs Stockfish on every candidate question, and with SE, which needs a single pass per position? The paper reports accuracy curves against question count for all methods but compute per question for none of them, and I think the table would strengthen rather than weaken the case if it were added.
-
Why is informativeness read as distance to a fixed zero anchor rather than a learned scalar scorer? Section 3.1 motivates the anchor as a reference point, but I cannot tell from the paper whether that specific choice matters. An ablation against a plain MLP scorer would answer this cheaply.
-
How does the method behave under finer rank discretization, e.g. 16 or 32 bins? With eight ranks, ±1 accuracy hits 100% at 22 questions, which is reassuring; but adjacent Elo intervals of 200 points are wide, and the practical value of an adaptive test depends on the granularity at which it stays accurate.
-
Clarification: Section 4.1 gives 150 independent assessments per evaluation player over 24 evaluation players, but the unit of the shaded 95% band in Figure 2 is not stated. Since the same player contributes 150 sessions, the band cannot be a simple across-session interval without underestimating the variance attributable to the eight rank strata. How is it computed — across sessions, across ranks, or bootstrapped?