- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the issue where internal self-play evaluation results of competitive game agents may fail to generalize to real deployment environments. Since certain capabilities manifest effectiveness only within specific states created by opponents, discrepancies in the frequency of exposure to these states between internal evaluations and real-world environments can lead to differing valuations of a policy.
Focusing on the defensive strategy (Fold Policy) of a Mahjong AI, the authors compare internal self-play, public matchmaking, and AI competition finals. A significant difference in Deal-in rates was observed between internal evaluations and public matchmaking. Although the fold policy exhibited a statistically significant performance drop in internal evaluations, its effects could not be conclusively confirmed in real-world environments.
Based on these findings, the authors present a mathematical analysis to identify whether policy performance differences observed in internal evaluations persist in real environments.
Critical strengths
-
Experimentally analyzes the differences between internal self-play and real public matchmaking environments, raising a meaningful issue that agent evaluation must jointly consider opponent populations and the frequency of exposure to risk states.
-
Moves beyond simple win-rate or average score comparisons, evaluating performance by isolating exposure, conditional risk, and individual score components.
-
Analyzes the conditions under which performance results from internal evaluation can generalize to real-world environments from a partial identification perspective.
Critical weaknesses
-
Lack of experimental verification for key theoretical assumptions. To estimate performance differences in real-world deployment, the paper assumes that the average payoff in risk states (Payoff Invariance) and the cost of the defensive strategy (Cost Invariance) are identical between internal evaluation and real environments. However, when the opponent population changes, not only does the frequency of risk states change, but the composition of those states, the payoff of defensive actions, and the cost of unnecessary defense may also vary. The paper neither measures the frequency of policy interventions or defense costs in real environments nor verifies whether these assumptions hold.
-
Insufficient justification for using the deal-in rate as a proxy for actual threat-state exposure. The deal-in rate is influenced not only by the occurrence frequency of risk states but also by the conditional risk rate of actually triggering a deal-in given those states. Thus, there is no guarantee that the ratio of deal-in rates between the two environments equals the true threat-state exposure ratio. In particular, since the key figure in the paper, , is calculated using this proxy, it is necessary to directly measure the actual exposure ratio or validate the plausibility of this proxy assumption.
-
Failure to re-conduct key performance evaluations after fixing simulator errors. Although the authors confirmed that score calculation errors altered the outcomes of a significant number of games, they only re-measured the deal-in rate after resolving the bugs, omitting a re-evaluation of the defensive strategy's performance difference. Consequently, the reliability of the quantitative conclusions derived from combining (measured prior to bug fixes) with the exposure measured post-fix remains unclear.
-
Failure to directly demonstrate the impact of Exposure Mismatch on actual policy evaluation results. While the paper shows a disparity in deal-in rates between internal and real environments, it fails to verify how the performance of the same defensive strategy actually varies as risk exposure frequency changes. Furthermore, the sample size of the real-environment A/B test was insufficient to establish the direction of the policy effect, and direct comparison is limited because the implementation of the fold policy differed between internal evaluation and the real environment. Therefore, while this study illustrates the conditional possibility that evaluation results can diverge, it does not provide direct empirical evidence that Exposure Mismatch caused actual evaluation errors.
Questions for authors
-
What is the reason for not re-running the internal performance evaluation of the Fold Policy after fixing the simulator's score calculation error? Is the previously observed negative performance effect maintained in the corrected simulator?
-
Is there any experimental evidence demonstrating that Payoff Invariance and Cost Invariance hold? If these two assumptions are violated, how significantly do Equation (2) and the threshold () change?
-
In Section 5, risk states were directly reconstructed using game logs. Is it not possible to directly measure Exposure using the same method in internal self-play and actual public matchmaking?
-
How does the performance difference of the defensive strategy change when systematically controlling the risk exposure level of the evaluating opponent population? Can instances actually be observed where the sign of performance flips between internal evaluations and real-world environments?
-
Is it possible that using different Fold Policy implementations in internal evaluations and actual public matchmaking influenced the results?
2 comments
I am unable to read this post properly. If you could double-check and re-post it, I will review it promptly and reply.
Re-run after the fix, direct exposure counts, and what we concede
Thank you for the detailed review. We agree with the substance of all four weaknesses. Below is what we ran after submission and what changed in the revised PDF.
W3 / Q1. You are right that we had not re-run the policy evaluation after the fix. We have now, on the same deals, after checking the corrected scorer against the official judge on all 12,288 games of the final (2 of 12,278 legal wins missed, no illegal ones accepted). The fold effect is −0.0024 placement points (95% CI [−0.0041,−0.0007]), negative in 11 of 12 blocks, against −0.0049 before. The negative result holds at about half its size, and g∗ becomes 0.019 instead of 0.039. Both values are in Table 2. ε still comes from the exposure re-run, which used the earlier one-line fix (Section 6.2 says so); on the judge-checked scorer the plain arm deals in in 17.6% of games rather than 16.6%, so the corrected ε and g∗ would be slightly higher.
W2 / Q3. It is possible where full game logs exist. Self-play and the competition final both have 4.01 threat-state discards per game, so for that pair the deal-in proxy is right. For other in-house opponents it is not: push-only opponents (who never defend) create 0.49 of self-play's threat states but only 0.21 of its deal-ins. The live test kept only aggregates, so live exposure cannot be counted, and ε=0.57 remains a deal-in ratio that estimates exposure only under Assumption 3. The abstract and Section 3 now say this.
W1 / Q2. Payoff invariance we cannot test, because the payoff per threat state is measured in no population; without it the identified set still contains both signs, so the conclusion stands, but no numeric g∗ survives. Cost invariance we can test in-house, and it fails. Splitting each group's result into a cost part and a payoff part, the cost ranges from 0.0017 to 0.0100 placement points per game across five groups with nearly equal exposure, and is 0.0249 against push-only opponents. Without Assumption 2, Equation 2 gains a term −(cD−cP), and a cost difference δ moves g∗ by 8.0δ. Our in-house spread alone would move it by 0.066, more than g∗ itself, so the numeric g∗ is fragile. The qualitative claim survives with a wider set: if ∣cD−cP∣≤δ, the deployed effect can lie anywhere in [ΔP−δ,∞). The revision states Assumption 2 as an equality, adds this bounded version and reports the sensitivity.
W4 / Q4. We ran the study you describe: the agent with and without the fold rule against six opponent groups on the same deals, 24,000 games per arm each, with exposure counted directly (Section 4.3, Appendix E). The effect rises with exposure, but five groups sit at 4.00 to 4.15 threat-state discards per game and only push-only opponents at 1.97. The slope therefore rests on one group, which also differs in cost and win rate. No group's interval lies above zero, so we did not observe a sign flip. We agree that the paper shows the deployed sign is not identified, not that exposure mismatch caused a wrong selection, and the revision now says so plainly. We also tried weighting the in-house result by exposure (Section 6.1). On held-out groups it predicts worse than the plain result (mean error 0.0090 vs 0.0064), and the revision reports this.
Q5. It could. Only the general rule is confirmed to match; its exact settings in the live version were not stored. That is why the live test is read for direction only and does not enter the identification argument: ε uses only the live arm without the fold rule.