CheckMate CP: Bidirectional Conformal Screening of Behavioral Skill Discrepancies in Online Chess

Online platforms increasingly rely on behavioral models to estimate user ability and identify unusual patterns of decision-making. A central challenge is determining whether an individual’s observed behavior is consistent with their expected skill. We study this problem in online chess, where recorded ratings guide matchmaking and human-policy models provide a way to infer skill from observed decisions. We introduce CheckMate CP, a calibrated framework for measuring behavioral skill discrepancies. It estimates Behavioral Elo from move likelihoods under a frozen human-policy model and compares it with the player’s recorded rating. The resulting gap indicates whether an observation window exhibits stronger- or weaker-than-rated play. Conformal calibration converts these gaps into directional screening alerts with finite-sample marginal false-alert control under exchangeability with a reference population. We evaluate CheckMate CP across natural games, controlled behavioral interventions, human-player substitutions, and games with logged assistance. The results distinguish rating-estimation accuracy, behavioral response, and alert detection. In our window-length analysis, longer observation windows improve rating accuracy but can reduce alert sensitivity. Logged assistance can also raise Behavioral Elo without necessarily triggering an alert. On ChessFraud, the neural estimator flags 54.5% of 44 assisted windows, compared with 27.3% for Stockfish top-move matching, at a 5% upward alert level. Both methods are calibrated using public games with the same time control and produce comparable empirical alert rates on held-out public windows. CheckMate CP provides an interpretable framework for prioritizing discrepancies between observed play and expected skill for human review, without inferring intent or establishing rule violations.