acceptodds
Under review as a conference paper at ICLR 2027

CheckMate CP: Bidirectional Conformal Screening of Behavioral Skill Discrepancies in Online Chess

Abstract

Online platforms increasingly rely on behavioral models to estimate user ability and identify unusual patterns of decision-making. A central challenge is determining whether an individual’s observed behavior is consistent with their expected skill. We study this problem in online chess, where recorded ratings guide matchmaking and human-policy models provide a way to infer skill from observed decisions. We introduce CheckMate CP, a calibrated framework for measuring behavioral skill discrepancies. It estimates Behavioral Elo from move likelihoods under a frozen human-policy model and compares it with the player’s recorded rating. The resulting gap indicates whether an observation window exhibits stronger- or weaker-than-rated play. Conformal calibration converts these gaps into directional screening alerts with finite-sample marginal false-alert control under exchangeability with a reference population. We evaluate CheckMate CP across natural games, controlled behavioral interventions, human-player substitutions, and games with logged assistance. The results distinguish rating-estimation accuracy, behavioral response, and alert detection. In our window-length analysis, longer observation windows improve rating accuracy but can reduce alert sensitivity. Logged assistance can also raise Behavioral Elo without necessarily triggering an alert. On ChessFraud, the neural estimator flags 54.5% of 44 assisted windows, compared with 27.3% for Stockfish top-move matching, at a 5% upward alert level. Both methods are calibrated using public games with the same time control and produce comparable empirical alert rates on held-out public windows. CheckMate CP provides an interpretable framework for prioritizing discrepancies between observed play and expected skill for human review, without inferring intent or establishing rule violations.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, anomaly detection, human behavior modeling, chess, cheating detection, sandbagging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gNyWOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes a framework for identifying whether a player’s observed play is substantially stronger or weaker than their recorded Elo rating and warrants further human review. It uses Maia-3 to estimate which playing strength best matches the player’s observed moves through three different methods. It then uses players in a validation dataset to determine whether the discrepancy warrants further human inspection. Its main contribution is the introduction of three methods for estimating playing strength using Maia-3. However, the paper’s experimental design has significant shortcomings, and most of the proposed methods are not compared with existing approaches, making it difficult to determine the paper’s substantive contribution.

Critical strengths

Its main contribution is the introduction of three methods for estimating playing strength using Maia-3.

Critical weaknesses
  1. Missing comparisons with directly relevant prior methods. The paper does not compare its proposed estimators with Kuboki et al. (2025), or its assistance-screening performance with the detectors of Linich et al. (2026), despite citing both and using the ChessFraud dataset. The included baselines do not establish an advantage over these directly relevant approaches, leaving the contribution of the new estimators and screening pipeline insufficiently demonstrated.

  2. Insufficient validation of short-window playing-strength estimates. Section 4.1 evaluates ten-game estimates against recorded ratings without establishing that these ratings reliably represent the strength expressed in the selected windows. Consequently, the reported MAE conflates estimation error with the behavioral discrepancies the framework aims to detect. Figure 2(b–c) provides limited independent validation because Maia-3 both generates and scores the games. Although the human-data experiments provide complementary detection evidence, they do not resolve whether the estimated Behavioral Elo accurately measures short-window playing strength.

Questions for authors
  1. Please refer to the concerns raised in the Weaknesses section.
  2. Why not use existing methods to estimate playing strength, such as [1] and [2], rather than introduce new Maia-3-based estimators? What advantages do the proposed methods offer over these existing approaches?
  3. Why use Maia-3 to generate actions and then use Maia-3 again to evaluate playing strength? Several approaches already support strength adjustment, including [2], [3], and [4]. If games were generated using these independent approaches, would the proposed Maia-3-based estimators still accurately estimate playing strength and detect changes?
  4. Did Section 4.1 select players with stable ratings? If not, what justifies using recorded Elo ratings as labels for the strength expressed in those ten games? Have the authors considered evaluating on strength-adjustable agents under controlled settings to reduce uncertainty about the underlying playing strength? Otherwise, the reported MAE may reflect both estimation error and fluctuations in the player’s performance over those ten games, making it difficult to interpret as evidence of estimation accuracy. References [1] Kuboki et al. Policies of Multiple Skill Levels for Better Strength Estimation in Games. AIIDE, 2025. [2] Chen et al. Strength Estimation and Human-Like Strength Adjustment in Games. ICLR, 2025. [3] Zhang et al. Human-aligned Chess with a Bit of Search. ICLR, 2025. [4] Wu et al. On Strength Adjustment for MCTS-Based Programs. AAAI, 2019.
5 comments
AuthorsOpenReview ↗

Response to gNyW (1/2) — Prior methods and estimator design (N-W1, N-Q2)

Thank you for raising the two main concerns about prior-method comparisons and independent strength validation. We added common-window comparisons in Tables 1 and 2 and an independent Stockfish evaluation in Section 4.2. We address the independent validation and recorded-rating questions in Part 2.

N-W1. How do our estimates compare with Kuboki et al., and our alerts with the ChessFraud detectors?

Table 1 compares the methods on the same 197 ten-game windows, using median recorded platform rating as the target.

EstimatorMean absolute error (Elo)
Neural residual92.8
Kuboki-based window adaptation135.1
Chen SE window adaptation140.2
Chen SE-infty window adaptation140.3

We used Chen's released checkpoints and fitted mappings to Elo. For the 12-feature Kuboki-based adaptation, we substituted Chen SE for SE-infty and refit LightGBM on our training data.

Screening methodFlagged assisted windows /44Flagged public windows /748
Neural residual2442
ChessFraud Allie-only243
ChessFraud combined1232

The screening comparisons use the same 741 calibration windows and nominal 5% upward level. The classifier heads were trained on ChessFraud-Synth. Our Allie inputs differ from the released cache, and we computed the Stockfish 17 features for the combined classifier because the original annotations were unavailable. Public alerts measure observed review workload, which differs across methods. Table 2, Figure 2D, and Appendix G.6 give the full comparison.

N-Q2. What do the Maia-3 estimators offer over the existing Kuboki and Chen methods?

Kuboki- and Chen-based estimates can also feed our screening rule. The three estimators use the same Maia-3 move evidence in different ways. Direct needs no project-specific fitting. Rating-calibrated maps joint move evidence to ratings, and Neural residual learns a window-level correction to Direct. Neural residual has the lowest recorded-rating error (Table 1). In the independent Stockfish test, Rating-calibrated, Neural residual, and Direct correctly ordered all three strength settings in 180, 99, and 67 of 249 matched comparisons, respectively. The Kuboki, Chen SE, and Chen SE-infty adaptations did so in 86, 63, and 29 of the same comparisons (Table E.5).

AuthorsOpenReview ↗

Response to gNyW (2/2) — Recorded ratings and independent strength validation (N-W2, N-Q1, N-Q3, N-Q4)

N-W2. Does agreement with recorded Elo validate the strength expressed over ten games?

Section 4.1 measures agreement with each player's median recorded platform rating over ten games. Its error can reflect both estimation error and a discrepancy between recent play and recorded rating. The independent Stockfish test below supplies a separate check of relative playing strength.

N-Q3. Can the estimators rank strength and flag changes when an independent engine generates the games?

Thank you for suggesting these papers. We reviewed all four and cite them in the revised manuscript. We adapted the strength estimators from Kuboki et al. (2025) and Chen et al. (2025) for our comparisons in Tables 1 and E.5. We used Allie embeddings from Zhang et al. (2025) in the ChessFraud comparison (Appendix G.6), and added Wu et al. (2019) to Appendix E.4 as prior work on adjusting an engine’s strength in Go.

To answer your question about using Maia-3 for both generation and scoring, we kept the Maia-3 estimators fixed and used Stockfish 18 to generate the moves after the opening in 12,450 games across calibration and test. Match results against a fixed Skill 5 opponent confirmed the ordering of the three settings. The test contains 249 matched sets. Within each set, we used the same ten starting positions to generate one ten-game window per setting. The estimates moved in the expected direction on average for both the weaker and stronger settings.

EstimatorCorrect strict ordering /249Strong upward alerts vs. control /249Weak downward alerts vs. control /249
Direct likelihood671 vs. 99 vs. 11
Rating-calibrated18020 vs. 8101 vs. 10
Neural residual9929 vs. 84 vs. 3

Ordering evaluates the continuous estimates. The alert columns use a nominal 2.5% cutoff per tail. Neural residual flagged 29 stronger windows versus 8 controls in the upper tail and 4 weaker windows versus 3 controls in the lower tail. On unchanged controls, Direct and Rating-calibrated flagged 20 and 18 of 249 windows in either direction, above the nominal 5% level for the two directions combined. Opening reuse limits what those counts establish about calibration. This design tests relative strength against a fixed opponent. Section 4.2 and Tables E.5–E.6 give the full estimates and alert counts. Whether these results carry over to the other suggested generators remains open.

N-Q4. Were players selected for stable ratings, and what can recorded Elo tell us about ten-game strength?

We did not select the original 197 test windows for prior rating stability. Following Herbrich et al. (2006), we distinguish underlying skill from game-to-game performance. Among 1,683 natural-game histories with at least 20 games, the median span from the first to the tenth game start was 46.55 hours. In a 209-window diagnostic, 24 windows overlapped exactly with the original test set. Among 131 accounts whose earlier ten pregame ratings spanned at most 50 Elo, the later window's median recorded rating differed from its first pregame rating by 9.66 Elo on average in absolute terms. This measures recorded-rating variation in that subgroup. The Stockfish outcomes provide separate evidence of relative strength in simulation.

N-Q1. Where do we address the missing comparisons and the validation of ten-game strength?

Part 1 presents the directly relevant comparisons. N-W2 and N-Q3 above explain the rating target and report the independent validation.

Thank you to the authors for the response and for adding several experiments. These additions largely address my questions about why Maia-3 is used for detection, although this rationale could be explained more clearly in the paper. The additional comparisons are also helpful and make the rationale for the overall method more complete.

However, the experiment using a different game generator also reveals potential limitations in the method’s generalization. The original paper used Maia-3 to generate games and then evaluated them using Maia-3, achieving good performance. However, detection performance remains limited in the experiment using Stockfish 18 to generate games. Since the rebuttal period is nearly over, I understand that the authors may not have time to add further experiments. For future work, using more human-like strength-adjustment methods, such as [2][3], could be a better direction. Another possibility would be to use a human-like model to play and have another AI engine modify some of its moves, without requiring those modifications to be human-like. This may also be closer to practical scenarios. Overall, generating games with AI in this way would allow the underlying policy and strength settings to remain fixed across the ten games, enabling further validation of the detector’s generalization ability.

Regarding the score, I will maintain my original overall rating because I still have substantial concerns about the method’s generalization. However, the additional baseline comparisons have provided more information, so I will increase Informativeness to 2.

[2] Chen et al. Strength Estimation and Human-Like Strength Adjustment in Games. ICLR, 2025.

[3] Zhang et al. Human-aligned Chess with a Bit of Search. ICLR, 2025.

Reviewer_GJ7EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on the question of whether observed decision behavior can reveal when an individual’s expressed level of competence differs from their expected profile. The authors use online chess as a testbed and propose CheckMate CP, a framework for checking whether play matches an account’s rating, which combines human-policy models with distribution-free calibration. It uses a fixed Maia-3 model to assign likelihoods to observed moves at different skill levels. It constructs three Behavioral Elo estimators including Direct likelihood, Rating-calibrated likelihood, and Neural residual. Comparing Behavioral Elo with the account’s recorded rating yields a signed discrepancy. It further applies conformal calibration to convert the gaps into directional screening alerts. Experimental results show that Neural residual estimator of CheckMate CP achieves an MAE of 92.8 Elo in rating estimation and can detect windows containing games with logged engine assistance.

Critical strengths
  1. The problem addressed in the paper is of practical significance.
  2. The methodological pipeline is clear and easy to follow.
Critical weaknesses
  1. The paper does not sufficiently substantiate the limitations of existing behavioral anomaly detection methods, leaving it unclear how the proposed method advances beyond prior work and why this study is necessary.
  2. The experiments lack direct comparisons against exist methods for skill estimation and engine-assistance detection, and therefore do not convincingly demonstrate the advantages of the proposed approach over prior work.
  3. Three Maia-3 estimators use opponent moves, while the human-donor experiment changes both the focal player’s behavior and the opponent context. Without input ablations using the same backbone, the results cannot disentangle the contributions of focal-player skill changes from those of opponent-context information.
Questions for authors
  1. Could the authors compare the proposed method with existing methods for chess skill estimation and engine-assistance detection?
  2. Could the authors compare focal-player-only, opponent-only, and full-model variants under the same training and calibration conditions, and report rating MAE and detection rates at the same review burden?
2 comments
AuthorsOpenReview ↗

Response to GJ7E (1/2) — Contribution and direct comparisons (G-W1, G-W2, G-Q1)

Thank you for asking us to clarify the screening task and compare directly with existing approaches. We added same-window comparisons for both skill estimation and assistance screening.

G-W1. What does rating-relative screening add to existing behavioral anomaly detection methods?

Engine-based performance estimates and assistance detectors provide useful measures of move quality and evidence of engine use. CheckMate CP illustrates a general framework for screening behavioral deviations from a recorded expectation. In chess, we use a human-policy model to estimate skill from ten games and ask whether recent play is unusually strong or weak for the account's recorded rating. This gives human reviewers a complementary signal alongside engine matching and assistance classifiers.

We can recalibrate the alert thresholds on a different chess platform using suitable local reference windows. With a fixed scoring rule, if the calibration scores and the score of a new reference window are jointly exchangeable, conformal calibration bounds the marginal probability of an alert for that window at the chosen level. Detection sensitivity still needs to be evaluated on the target platform. Existing estimators and assistance scores can also be calibrated. Tables 1 and 2 test how these choices perform for rating agreement and assistance screening. Section 5 discusses how this screening approach could be tested beyond chess.

G-W2 and G-Q1. How does CheckMate CP compare with existing skill estimators and assistance detectors?

Table 1 uses the same 197 ten-game windows and compares predictions with median recorded platform rating.

EstimatorMean absolute error (Elo)
Neural residual92.8
Kuboki-based window adaptation135.1
Chen SE window adaptation140.2
Chen SE-infty window adaptation140.3

The Chen adaptations use released checkpoints and fitted mappings to Elo. The Kuboki-based adaptation uses 12 features, substitutes Chen SE for SE-infty, and refits LightGBM on our training data.

Screening methodFlagged assisted windows /44Flagged public windows /748
Neural residual2442
Stockfish top-match1241
Stockfish 18 average centipawn loss832
ChessFraud Allie-only243
ChessFraud combined1232

These rows use the same windows and 741 calibration windows at a nominal 5% upward level. Neural residual flagged 24 assisted windows versus 12 for top-match at similar observed public workload. In a separate raw-gap sensitivity analysis on 19 windows without repeated games, the counts were 10 versus 8 (Appendix G.2). Public alerts measure workload because assistance status is unknown. Table 2 and Figure 2D report the screening comparison. Our Allie inputs differ from the released cache, and we computed the Stockfish 17 features for the combined classifier because the original annotations were unavailable (Appendix G.6).

AuthorsOpenReview ↗

Response to GJ7E (2/2) — Focal and opponent inputs (G-W3, G-Q2)

G-W3. Can the donor experiment separate the focal player's behavior from the opponent's influence?

Donor substitution changes the evaluated player's moves, the opponents, and the board histories together. We therefore refitted three versions of Neural residual using the same frozen Maia-3 model. We kept the Stockfish opponent at Skill 5. In four games per window, we assigned assistance with a 20% probability on each eligible turn by the player being evaluated. The table gives mean changes in predicted rating, in model-rating units, across four matched seeds and 249 pairs of ten-game windows.

Move evidence usedMean prediction shift
Both players (full)+93.2
Focal player only+50.7
Opponent only+9.9

Changing the opponent's Stockfish settings while keeping the evaluated player's settings fixed also changed which windows triggered alerts. All three versions use board positions created by both players, so the separate contributions to the original donor alerts remain unresolved. Section 4.3 and Appendix E.5 report these checks.

G-Q2. How do focal-only, opponent-only, and full models compare in rating error and detection at the same review workload?

We used the same 880 fitting accounts, fitting procedure, and four matched seeds. We calibrated each version separately using the same protocol. The table shows ranges across seeds. We measured rating error on 197 ten-game windows from real games. For this screening comparison, we chose each version's cutoff after scoring the same public windows, so each flagged 45 of 897. We then applied the cutoffs to the same 44 assisted windows. This public sample differs from the one in Table 2.

Move evidence usedRating mean absolute error (Elo)Flagged assisted windows /44 at matched workload
Both players (full)91.73–97.9217–19
Focal player only135.99–138.948–12
Opponent only95.88–99.7219–21

Opponent-only flagged more assisted windows than full in every seed on the 44-window panel. On the 19-window subset without repeated physical games, the full counts were 10, 8, 7, and 9 across the four seeds, and the opponent-only counts were 8, 7, 7, and 7. In the 13-player paired analysis, the opponent-only model flagged three windows with logged assistance in four games. In every seed, it also flagged the corresponding windows with no logged assistance for the player being evaluated. These checks leave the source of the opponent-only alerts unresolved. Appendix G.7 and Table G.6 also report the public alert counts obtained with cutoffs chosen on a separate reference sample.

Reviewer_oy5yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Online chess platforms pair players by rating. An account that plays much better or much worse than its rating therefore distorts its matches, for example when a player uses an engine, or loses on purpose to face weaker opponents later. The paper proposes CheckMate CP, a procedure that flags windows of ten games in which the play does not match the account's rating. It uses Maia-3, a published model that predicts the move a human of a given rating would play, kept frozen. From the probabilities that Maia-3 assigns to the moves actually played, at ratings from 800 to 2600, the method estimates the rating that best explains the play, called Behavioral Elo. Three estimators are proposed. Direct likelihood fits no parameters. Rating-calibrated combines the evidence differently and maps it to recorded ratings with a fitted linear function. Neural residual starts from the Direct estimate and adds a correction in Elo predicted by a small neural network. All three also use the moves of the opponents. The difference between Behavioral Elo and the recorded rating is then compared with the same difference in a reference set of ordinary windows, using conformal p-values. A window raises an upward or downward alert when its difference is among the most extreme. On windows that behave like the reference ones, this keeps the false-alert rate below a chosen level, 5% in the main experiments. The evaluation uses Lichess games from 2026, grouped into windows of ten consecutive games of the same account. (i) The first test checks how close Behavioral Elo is to the recorded rating. On 197 test windows, Neural residual is off by 93 Elo on average. This is better than Maia-2 and the other learned models, but the average rating of the opponents is much closer, 39 Elo, because the platform pairs players of similar rating. The authors note that this simple baseline cannot react to the player's own moves. (ii) In the second test, the authors let Maia-3 continue real games after the opening, making the player stronger, weaker, or helped by engine moves. When nothing is changed, about 5% of windows are flagged, as intended, and larger changes are flagged more often. Since the same Maia-3 creates and scores these games, the authors treat this as a sanity check, not as a comparison between estimators. (iii) In the third test, they take the ten games of a real player and assign them to an account rated about 400 Elo higher or lower. Neural residual flags about 80% of these windows, and none of the control windows, where the two ratings are close. (iv) The last test uses ChessFraud, a set of tournament games in which engine use was recorded. Out of 44 windows that contain some assisted games, Neural residual flags 24. A simple engine baseline, which counts how often the player chooses Stockfish's top move, flags 12. Both methods flag a similar share of ordinary public windows, about 6%. Finally, longer windows give more accurate rating estimates, but on generated games they lead to fewer alerts.

Critical strengths

The paper addresses a concrete need of online platforms with a well-defined target: whether the skill shown in recent games matches the rating used for matchmaking. It checks both directions, so the same procedure covers engine assistance and deliberate underperformance. Alerts are set with conformal p-values rather than ad hoc cutoffs. A platform can therefore choose in advance how many false alerts it accepts on windows that resemble the reference set. The experiments confirm this. With the method set to allow 5% false alerts, the three estimators flag between 4.8% and 6.0% of generated windows in which the play was left unchanged. When looking for play stronger than the rating, they flag between 4.8% and 5.8% of ordinary Lichess windows from August.

The evaluation does not rely on generated data alone. Besides games generated by Maia-3 and windows built from real games of other players, it uses ChessFraud, a set of real tournament games in which engine use was logged during play. This gives a direct test on real assisted play, which public games cannot provide because they carry no verified record of engine use. The paper also measures separately how well Behavioral Elo matches the rating, how much it changes when play changes, and how often this change triggers an alert.

Critical weaknesses
  • Rating-calibrated and Neural residual also use the strength of the opponents, estimated from their moves, as evidence of the account's own strength. Consider an account rated 1500 that uses an engine and plays like a 2000 player. On a platform it still meets opponents rated around 1500. These estimators see opponents who play like 1500 players and take this as evidence that the account is also near 1500. This pulls the estimate down and makes the gap harder to detect. The test that assigns another player's games to an account works the other way. If a 1900 player's games are assigned to a 1500 account, those games were played against opponents near 1900. The opponents then also push the estimate up, which makes the gap easier to detect than in a real case. The detection of about 80% in this test may therefore overstate what happens in practice.
  • Four assisted games out of ten raise Behavioral Elo by 80 to 106 points, but at most 2 of these 13 windows are flagged. On generated games, Neural residual flags a 200 Elo jump in strength only 13% of the time. The method therefore catches only very large gaps and misses a player who uses an engine in a few games.
  • The paper compares its alerts only with the Stockfish top-move baseline. For example, it mentions the move-level classifiers of the ChessFraud authors but leaves them out, because they score single moves rather than ten-game windows. Simple engine measures, such as the average loss in engine evaluation per move, are not tested either. Without these comparisons, it is unclear whether the method adds anything over existing engine-based checks.
  • Play weaker than the rating is never tested on real cases of sandbagging.
  • The skill estimate comes from a frozen Maia-3, and Kuboki et al. (2025) already estimate playing strength from Maia models, but the paper does not compare with their method. The alerts use standard conformal p-values, and the paper's only proposition follows from known results. The methodological novelty for ICLR is therefore limited.
  • With 20 games instead of 10, Neural residual flags a generated 600 Elo jump in strength in 10% of windows instead of 30% (Appendix I). More games should give more evidence, not less. The paper reports this drop but leaves it unresolved, so it is unclear why ten games is the right choice and whether the score makes good use of longer histories.
  • ChessFraud provides only 44 windows with assisted games. Some participants played each other in the tournaments, so 100 games appear in two windows, once from each player's side, and the windows are not independent. With so few windows, the detection rate of Neural residual is very uncertain: its 95% interval goes from 39% to 68%. The test that assigns one player's ten games to an account rated about 400 Elo higher or lower is even smaller: 26 windows with a stronger player and 31 with a weaker one. Here too the same players and games appear in several windows, as the authors note. These tests are too small to draw firm conclusions about real play.
  • Figure 1 and the architecture diagrams in Figures C.1 to C.3 are embedded as bitmap images rather than vector graphics, so their text cannot be selected or searched and loses sharpness at high zoom. Figure 1 also appears to have been produced, at least in part, with a recognizable AI style.
Questions for authors
  • How does the method compare, on the same windows, with the move-level classifiers of the ChessFraud authors and with the average loss in engine evaluation per move?
  • How does the method compare with Kuboki et al. (2025) on the same test windows?
  • Why does the detection of generated changes drop when windows grow from 10 to 20 games?
  • On which data were the three main estimators selected among the exploratory ones?
5 comments
AuthorsOpenReview ↗

Response to oy5y (1/4) — Donor results and human evidence (O-W1, O-W2, O-W4, O-W7)

Thank you for highlighting where our human evidence needs tighter interpretation. We qualified the donor result, added controlled opponent-context comparisons, and checked how shared games affect the ChessFraud comparison.

O-W1. Could changing the opponents along with the focal player inflate the donor detection rate?

The 80.8% result (21/26 upward alerts) applies to substituting complete human ten-game windows. The substitution changes the focal player's moves, the opponents, and the board histories together. In a separate Stockfish simulation, we kept the Stockfish settings of the player being evaluated fixed. In four games of each assisted window, we assigned assistance with a 20% probability on each eligible turn by that player. We used one nominal 2.5% upward rule for Neural residual, calibrated on 498 unassisted Skill 5 versus Skill 5 windows.

Stockfish opponent settingFlagged windows without assistanceFlagged windows with assistance enabled in four games
Skill 012/2494/249
Skill 53/24913/249
Skill 108/24918/249

Each row compares 249 pairs of ten-game windows. We used the same starting positions across all three opponent settings. The assisted condition had fewer upward alerts at Skill 0 and more at Skills 5 and 10. These comparisons leave the contribution of opponent context to the 80.8% donor result unmeasured. Sections 3.4 and 4.3 now state what the substitution changes. Appendix E.3 and Table E.4 describe the simulation.

O-W2. How often does the method flag intermittent engine assistance or a modest increase in playing strength?

At the evaluated cutoff, the method misses most windows in the paired four-game assistance check. In ChessFraud, four assisted games increased mean Behavioral Elo by 79.6–106.4 points across the three estimators, while 0–2 of 13 assisted windows triggered upward alerts. In the generated +200 policy-conditioning condition, Neural residual increased by 68.28 Elo on average and flagged 33/249 windows versus 9/249 controls. The +200 refers to the generator setting. Sections 4.2 and 4.3 now report the score changes alongside the alerts.

O-W4. Did we test downward alerts on verified cases of intentional sandbagging?

We found no public set of verified intentional sandbagging cases suitable for our window-level evaluation. The Discussion explains that our downward evidence comes from simulated weaker play and complete human game windows paired with higher rating references. Validation on verified cases of intentional sandbagging remains open.

O-W7. How do small samples and shared games affect confidence in the human-data results?

The small samples and shared games limit the precision of the human comparisons. The original participant-bootstrap interval keeps calibration fixed and does not adjust for shared games. In a raw-gap sensitivity analysis, we selected 19 assisted windows with no repeated physical games before checking model scores and kept the full-panel cutoffs. Neural residual and Stockfish top-match flagged 10/19 and 8/19, compared with 24/44 and 12/44 under the same rule on the full panel. In a separate set of 20 pairs sharing a game with one-sided assistance, the other player's window had zero logged focal assistance across all ten games. Using standardized Neural residual gaps, 741 calibration windows, and the nominal 5% upward rule, only the assisted player's window alerted in 11 pairs and neither window alerted in nine. Shared opponents and tournament conditions can still introduce dependence. Appendix G.2 reports these checks, and Section 4.3 treats the donor rates as descriptive.

AuthorsOpenReview ↗

Response to oy5y (2/4) — Baselines and contribution (O-W3, O-W5, O-Q1, O-Q2)

O-W3 and O-Q1. How does assistance detection compare with ChessFraud classifiers and average centipawn loss on the same windows?

Table 2 and Figure 2D compare the methods on the same 741 calibration, 748 public test, and 44 assisted windows. The counts below use the nominal 5% upward level.

Screening methodFlagged assisted windows /44Flagged public windows /748
Neural residual2442
Stockfish top-match1241
Stockfish 18 average centipawn loss832
ChessFraud Allie-only243
ChessFraud combined1232

Neural residual flagged more assisted windows than top-match at similar observed public workload. The other methods have different workloads. The public windows have unknown assistance status, so their alerts measure review burden. Figure 2D shows the trade-off across cutoffs. We computed the engine measures from Stockfish output and trained the classifier heads on ChessFraud-Synth. Our Allie inputs differ from the released cache, and we computed the Stockfish 17 features for the combined classifier because the original annotations were unavailable. Appendix G.6 describes aggregation and these reproduction limits.

O-Q2. How does rating estimation compare with Kuboki et al. on the same windows?

On the same 197 ten-game windows, mean absolute error against median recorded platform rating was 135.1 Elo for our 12-feature Kuboki-based adaptation and 92.8 Elo for Neural residual (Table 1). The adaptation refits LightGBM on our fitting data and uses Chen SE in place of SE-infty. Appendix D describes the adaptation. We report the version that predicts a rating bin separately on the 171 windows within its supported rating range.

O-W5. What does this paper contribute given prior Maia-based estimators and standard conformal calibration?

Maia-3 is pretrained and the conformal rank result is standard. Our contribution is a window-level, rating-relative screening formulation. Ten games of human-policy move evidence produce a Behavioral Elo, the signed gap to the recorded rating gives the direction, and reference calibration converts the gap into an upward or downward alert. An alert identifies a window for further review. Tables 1 and 2 assess the resulting system against existing rating estimators and assistance scores. The independent Stockfish test separates estimate response from alert response.

AuthorsOpenReview ↗

Response to oy5y (3/4) — Window length and alert thresholds (O-W6, O-Q3)

O-W6 and O-Q3. Why does detection drop from ten to twenty games, and why retain ten games?

In the sustained +600 policy-conditioning experiment, upward alerts fell from 63/209 at ten games to 20/209 at twenty. We used the first 10 or 20 games from the same 500 simulated calibration histories and kept each player's pregame rating fixed across lengths. Both lengths use a nominal 2.5% level for upward alerts. The empirical cutoff rose from 247.34 to 318.70 Elo.

Window scoreTen-game cutoff (247.34 Elo)Twenty-game cutoff (318.70 Elo)
Ten-game score63/20923/209
Twenty-game score44/20920/209

At fixed ten-game scores, substituting the twenty-game cutoff lowers alerts from 63 to 23. At the fixed ten-game cutoff, substituting twenty-game scores lowers them from 63 to 44. These are descriptive comparisons on the original Elo-gap scale. With the ten-game model held fixed, extending its input to twenty games raised the mean paired response from 198.45 to 207.33 Elo. The separately calibrated cutoff rose from 247.34 to 281.86 Elo and alerts fell from 63 to 57. Using the within-window median rating in a retrospective recalibration reversed the cutoff movement. Appendix I reports the fixed-model and rating-reference checks. The table above shows the crossed cutoffs and scores.

We retain ten games as a fixed observation point. At cutoffs that flagged 24 of 500 calibration windows, Neural residual flagged 107 of 209 intervention windows at ten games and 101 of 209 at fifteen games. It flagged 10 of 209 separate test controls at each length. The median elapsed time from the first game start was 46.55 hours to the tenth game start and 80.24 hours to the fifteenth. Section 4.4 and Appendix I describe this trade-off for the evaluated cohorts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.