acceptodds
Under review as a conference paper at ICLR 2027

Self-Play Evaluation Under Population Exposure Mismatch: Unidentified Verdicts on Defensive Skill in Competitive Mahjong

Abstract

Game-playing agents are usually selected in-house, against copies of themselves or ancestors, before they meet their deployment population. We study one way this selection can mislead: a skill may pay off only in states that opponents create, and the in-house population may create those states at a different rate. We measure this population-exposure mismatch for a Chinese Standard Mahjong agent, using deal-in incidence as a policy-dependent proxy for threat-state exposure, and ask what an in-house verdict on a defensive intervention identifies. With our simulator's scoring defect corrected, our strongest in-house population exposes the deployed policy at times the live-field rate ( CI –) and at a competition final's rate. Under stated assumptions for transporting a verdict, any exposure gap leaves the deployed sign of a fold policy with a precise negative in-house verdict unidentified: against the live field the sign flips if an exposed game gains over placement points; against the final the verdict carries over. Gate verdicts predate the correction. A live A/B resolves neither the deal-in effect nor placement, which would take thousands of games per arm. In that final, the top two agents' statistical tie splits into opposing score channels; finalists face threat states equally often but differ in how often a threatened discard deals in. We distil an evaluation recipe that separates exposure-validity checks from general hygiene. All evidence comes from one game and one agent lineage.

Primary area:
reinforcement learning
Keywords:
agent evaluation, self-play, distribution shift, partial identification, multi-agent reinforcement learning, Mahjong

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nUAHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the issue where internal self-play evaluation results of competitive game agents may fail to generalize to real deployment environments. Since certain capabilities manifest effectiveness only within specific states created by opponents, discrepancies in the frequency of exposure to these states between internal evaluations and real-world environments can lead to differing valuations of a policy.

Focusing on the defensive strategy (Fold Policy) of a Mahjong AI, the authors compare internal self-play, public matchmaking, and AI competition finals. A significant difference in Deal-in rates was observed between internal evaluations and public matchmaking. Although the fold policy exhibited a statistically significant performance drop in internal evaluations, its effects could not be conclusively confirmed in real-world environments.

Based on these findings, the authors present a mathematical analysis to identify whether policy performance differences observed in internal evaluations persist in real environments.

Critical strengths
  1. Experimentally analyzes the differences between internal self-play and real public matchmaking environments, raising a meaningful issue that agent evaluation must jointly consider opponent populations and the frequency of exposure to risk states.

  2. Moves beyond simple win-rate or average score comparisons, evaluating performance by isolating exposure, conditional risk, and individual score components.

  3. Analyzes the conditions under which performance results from internal evaluation can generalize to real-world environments from a partial identification perspective.

Critical weaknesses
  1. Lack of experimental verification for key theoretical assumptions. To estimate performance differences in real-world deployment, the paper assumes that the average payoff in risk states (Payoff Invariance) and the cost of the defensive strategy (Cost Invariance) are identical between internal evaluation and real environments. However, when the opponent population changes, not only does the frequency of risk states change, but the composition of those states, the payoff of defensive actions, and the cost of unnecessary defense may also vary. The paper neither measures the frequency of policy interventions or defense costs in real environments nor verifies whether these assumptions hold.

  2. Insufficient justification for using the deal-in rate as a proxy for actual threat-state exposure. The deal-in rate is influenced not only by the occurrence frequency of risk states but also by the conditional risk rate of actually triggering a deal-in given those states. Thus, there is no guarantee that the ratio of deal-in rates between the two environments equals the true threat-state exposure ratio. In particular, since the key figure in the paper, ε=0.57\varepsilon = 0.57, is calculated using this proxy, it is necessary to directly measure the actual exposure ratio or validate the plausibility of this proxy assumption.

  3. Failure to re-conduct key performance evaluations after fixing simulator errors. Although the authors confirmed that score calculation errors altered the outcomes of a significant number of games, they only re-measured the deal-in rate after resolving the bugs, omitting a re-evaluation of the defensive strategy's performance difference. Consequently, the reliability of the quantitative conclusions derived from combining ΔP=−0.0049\Delta_P = -0.0049 (measured prior to bug fixes) with the exposure measured post-fix remains unclear.

  4. Failure to directly demonstrate the impact of Exposure Mismatch on actual policy evaluation results. While the paper shows a disparity in deal-in rates between internal and real environments, it fails to verify how the performance of the same defensive strategy actually varies as risk exposure frequency changes. Furthermore, the sample size of the real-environment A/B test was insufficient to establish the direction of the policy effect, and direct comparison is limited because the implementation of the fold policy differed between internal evaluation and the real environment. Therefore, while this study illustrates the conditional possibility that evaluation results can diverge, it does not provide direct empirical evidence that Exposure Mismatch caused actual evaluation errors.

Questions for authors
  1. What is the reason for not re-running the internal performance evaluation of the Fold Policy after fixing the simulator's score calculation error? Is the previously observed negative performance effect maintained in the corrected simulator?

  2. Is there any experimental evidence demonstrating that Payoff Invariance and Cost Invariance hold? If these two assumptions are violated, how significantly do Equation (2) and the threshold (g∗=0.039g^* = 0.039) change?

  3. In Section 5, risk states were directly reconstructed using game logs. Is it not possible to directly measure Exposure using the same method in internal self-play and actual public matchmaking?

  4. How does the performance difference of the defensive strategy change when systematically controlling the risk exposure level of the evaluating opponent population? Can instances actually be observed where the sign of performance flips between internal evaluations and real-world environments?

  5. Is it possible that using different Fold Policy implementations in internal evaluations and actual public matchmaking influenced the results?

2 comments
AuthorsOpenReview ↗

Re-run after the fix, direct exposure counts, and what we concede

Thank you for the detailed review. We agree with the substance of all four weaknesses. Below is what we ran after submission and what changed in the revised PDF.

W3 / Q1. You are right that we had not re-run the policy evaluation after the fix. We have now, on the same deals, after checking the corrected scorer against the official judge on all 12,288 games of the final (2 of 12,278 legal wins missed, no illegal ones accepted). The fold effect is −0.0024-0.0024 placement points (95% CI [−0.0041,−0.0007][-0.0041, -0.0007]), negative in 11 of 12 blocks, against −0.0049-0.0049 before. The negative result holds at about half its size, and g∗g^* becomes 0.019 instead of 0.039. Both values are in Table 2. ε\varepsilon still comes from the exposure re-run, which used the earlier one-line fix (Section 6.2 says so); on the judge-checked scorer the plain arm deals in in 17.6% of games rather than 16.6%, so the corrected ε\varepsilon and g∗g^* would be slightly higher.

W2 / Q3. It is possible where full game logs exist. Self-play and the competition final both have 4.01 threat-state discards per game, so for that pair the deal-in proxy is right. For other in-house opponents it is not: push-only opponents (who never defend) create 0.49 of self-play's threat states but only 0.21 of its deal-ins. The live test kept only aggregates, so live exposure cannot be counted, and ε=0.57\varepsilon = 0.57 remains a deal-in ratio that estimates exposure only under Assumption 3. The abstract and Section 3 now say this.

W1 / Q2. Payoff invariance we cannot test, because the payoff per threat state is measured in no population; without it the identified set still contains both signs, so the conclusion stands, but no numeric g∗g^* survives. Cost invariance we can test in-house, and it fails. Splitting each group's result into a cost part and a payoff part, the cost ranges from 0.0017 to 0.0100 placement points per game across five groups with nearly equal exposure, and is 0.0249 against push-only opponents. Without Assumption 2, Equation 2 gains a term −(cD−cP)-(c_D - c_P), and a cost difference δ\delta moves g∗g^* by 8.0 δ8.0\,\delta. Our in-house spread alone would move it by 0.066, more than g∗g^* itself, so the numeric g∗g^* is fragile. The qualitative claim survives with a wider set: if ∣cD−cP∣≤δ|c_D - c_P| \le \delta, the deployed effect can lie anywhere in [ΔP−δ,∞)[\Delta_P - \delta, \infty). The revision states Assumption 2 as an equality, adds this bounded version and reports the sensitivity.

W4 / Q4. We ran the study you describe: the agent with and without the fold rule against six opponent groups on the same deals, 24,000 games per arm each, with exposure counted directly (Section 4.3, Appendix E). The effect rises with exposure, but five groups sit at 4.00 to 4.15 threat-state discards per game and only push-only opponents at 1.97. The slope therefore rests on one group, which also differs in cost and win rate. No group's interval lies above zero, so we did not observe a sign flip. We agree that the paper shows the deployed sign is not identified, not that exposure mismatch caused a wrong selection, and the revision now says so plainly. We also tried weighting the in-house result by exposure (Section 6.1). On held-out groups it predicts worse than the plain result (mean error 0.0090 vs 0.0064), and the revision reports this.

Q5. It could. Only the general rule is confirmed to match; its exact settings in the live version were not stored. That is why the live test is read for direction only and does not enter the identification argument: ε\varepsilon uses only the live arm without the fold rule.

I am unable to read this post properly. If you could double-check and re-post it, I will review it promptly and reply.

Reviewer_v7TqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether self-play evaluation can misjudge a defensive policy when the evaluation population generates relevant threat states at a different rate than the deployment population. It formalizes this population-exposure mismatch, measures it for a Mahjong agent across self-play, a competition final, and live matchmaking, and shows conditionally that a negative in-house fold-policy verdict need not identify its deployed sign. The analysis is supported by duplicate-game evaluations, a small live A/B, and event-level reconstruction of a competition final.

Critical strengths
  1. Clear and practical framing. The paper explains how exposure mismatch can limit the transfer of in-house evaluation results, and why more in-house games alone cannot resolve this uncertainty.

  2. Informative empirical analysis. The experiments distinguish threat exposure from conditional deal-in risk, providing a useful perspective on defensive performance beyond aggregate outcomes.

Critical weaknesses
  1. The core policy evaluations were not rerun after fixing the simulator’s scoring bug. Combining pre-fix policy effects with corrected exposure rates leaves the numerical conclusions insufficiently supported.

  2. Deal-in incidence reflects both threat exposure and conditional risk. The live data cannot separate these factors, so interpreting their ratio as an exposure ratio remains unverified.

  3. The experiments do not establish that exposure mismatch changes policy selection. The live A/B is inconclusive, leaving the proposed evaluation failure theoretically possible but empirically unconfirmed.

Questions for authors
  1. Does the negative fold-policy effect persist when the exact Table 3 evaluation is rerun on the corrected simulator? Please report the placement effect and scored deal-in incidence, and update g∗g^* accordingly.

  2. Which behaviors and settings are confirmed to match between the gated and live fold overlays, and which remain uncertain? This determines whether the two evaluations test the same intervention.

  3. Does any available live-game evidence help distinguish increased threat exposure from increased conditional deal-in risk? Without this distinction, how strongly can ε=0.57\varepsilon=0.57 be interpreted as an exposure ratio?

1 comment
AuthorsOpenReview ↗

Re-run on the corrected simulator: the fold loss keeps its sign at half the size

Thank you. We ran the re-run you asked for, and the revised PDF reports it next to the original values.

Q1 / W1. Before re-running, we checked the corrected scorer against the official judge on all 12,288 games of the competition final: it misses 2 of 12,278 legal wins and accepts no illegal ones. (The one-line fix described in the submission still missed 3.3%.) On the same deals, the submitted Table 3 (Table 2 in the revision) now gives:

  • placement, fold minus plain: −0.0024-0.0024 (95% CI [−0.0041,−0.0007][-0.0041, -0.0007]), negative in 11 of 12 blocks; before the fix it was −0.0049-0.0049;
  • scored deal-ins: 17.6% → 17.3% of games (−0.28-0.28 points, CI [−0.39,−0.16][-0.39, -0.16]);
  • g∗=0.019g^* = 0.019 (0.015 to 0.031 over the CI of ε\varepsilon), instead of 0.039. ε\varepsilon still comes from the exposure re-run, which used the earlier one-line fix (Section 6.2 says so); on the judge-checked scorer the plain arm deals in in 17.6% of games rather than 16.6%, so the corrected ε\varepsilon and g∗g^* would be slightly higher.

The stricter trigger still ties and the divert variants keep their verdicts. The danger-penalty result cannot be re-run and is labelled as pre-fix.

Q2. Only the general rule is confirmed to match. The live version's trigger, discard order and claim handling were not stored, so the two evaluations may not test the same intervention. That is why the live A/B is read for direction only. It does not enter the identification argument: ε\varepsilon uses only the live arm without the fold rule.

Q3 / W2. No live evidence separates exposure from conditional risk, because the live test kept only aggregates. Where full logs exist we now count threat states directly. Self-play and the final both have 4.01 threat-state discards per game, so for that pair the deal-in proxy is right. For other in-house opponents it is not: push-only opponents (who never defend) create 0.49 of self-play's threat states but only 0.21 of its deal-ins. So ε=0.57\varepsilon = 0.57 is a deal-in ratio, an estimate of the exposure ratio only under Assumption 3, with an error we cannot bound for the live field. The abstract and Section 3 now say this.

W3. Agreed. We varied the evaluation opponents (six groups, same deals, Section 4.3). The effect rises with measured exposure, but that rests on a single low-exposure group that also differs in cost and win rate, and no group's interval lies above zero. So no in-house sign flip was observed. The paper now says plainly that it shows the deployed sign is not identified, not that a wrong selection happened.

Reviewer_dLWCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper claims that selecting game-playing agents in-house can lead bemisleading for Chinese Standard Mahjong.

Critical strengths

None

Critical weaknesses

The paper is impossible to follow, lacking a clear contribution and presentation. They don't have any figures besides their results. The proof of Proposition 1 is not understandable and lacks formalism. There is no followable narrative; it jumps from one topic to another. I don't see any reason to judge this paper, given that scientific communication is an important part of the research.

Questions for authors

None

2 comments

Following AC's advice, I add more context about why the paper is not readable at all:

  • The paper uses technical terms as if they were common knowledge. See abstract: in-house, deal-in incidence, threat-state exposure, gate verdicts, exposure-validity checks from general hygiene
  • There is no structural narrative; it jumps from one argument to another without coherent transition: See almost all paragraphs
  • There is no explanatory figure; there are only plots for the experimental results
  • The writing is optimized for stating carefully qualified claims, but insufficiently optimized for helping someone understand them: see what we do not claim parts
  • Not defining mathematical notation: see EXPOSURE AND WHAT AN IN-HOUSE VERDICT IDENTIFIES section
AuthorsOpenReview ↗

Each of the five points is changed in the revised PDF

Thank you for coming back with specifics. We agree the submitted paper was hard to follow. The revised PDF changes each point:

  1. Terms. The abstract is rewritten in plain words. In-house testing, deal-ins and dangerous situations are explained where they first appear, and "gate verdicts" and "general hygiene" are no longer in the abstract. Table 1 (page 4) defines every term and symbol used later.
  2. Narrative. The introduction ends with the one question each section answers, and Sections 3 to 5 open by linking to the section before.
  3. Figure. A new Figure 1 (page 1) shows the setting with no data: two opponent groups that create dangerous situations at different rates, and why an in-house loss then does not fix the sign against real opponents.
  4. Qualified claims. "What we do not claim" is replaced by a short Scope paragraph: we do not show that in-house testing gave a wrong answer; we show when its answer cannot be carried over.
  5. Notation and proof. Section 2.2 now points to Table 1 for every symbol, and Appendix D proves Proposition 1 step by step. The idea in one line: if the payoff per dangerous situation gg and the cost are the same in both populations, the deployed effect equals the in-house effect plus (1−ε) g ρD(1-\varepsilon)\,g\,\rho_D. With ε<1\varepsilon < 1 and an in-house loss, the deployed sign depends on whether gg exceeds g∗=∣ΔP∣/((1−ε)ρD)g^* = |\Delta_P| / ((1-\varepsilon)\rho_D), and the data do not determine gg.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.