- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes PGV, a lightweight verification harness for promotion gates in competitive self-play. Its two checks examine whether certification and replication seed ranges are disjoint from selection data and whether evaluation samples structural conditions, such as seats and decks, consistently with training. The evidence comprises four incidents from two deployed systems: a selected mahjong candidate whose reported advantage disappears on fresh evaluation, substantial seed reuse in a nominal replication, a later mahjong improvement that survives disjoint evaluations, and a card-game ladder freeze associated with asymmetric evaluation. A synthetic reproduction finds a smaller directional effect than predicted and falsifies its preregistered magnitude threshold.
Critical strengths
The paper addresses a practically important failure point in self-play systems. Auditing the evaluator's seed allocation and role assignments is inexpensive relative to training, and these implementation errors can be difficult to diagnose from aggregate scores alone.
Critical weaknesses
-
The methodological contribution is limited. Selection-induced optimism is well established in model-selection research, including Cawley and Talbot (2010)[1], and separating tuning and testing seeds is explicitly recommended for RL by Eimer et al. (2023)[2]. The paper acknowledges this background, and its main contribution is to express data separation and balanced evaluation as two checks on a self-play gate's bookkeeping. This is a useful engineering practice, but the paper provides limited new methodological insight into when these checks are sufficient, which failures they cannot detect, or how they improve promotion decisions beyond straightforward evaluation safeguards. I do not find the adaptation of these principles to promotion gates alone a sufficiently substantial contribution.
-
The experiments do not sufficiently establish the effectiveness and generality of PGV. The evidence is limited to four incidents from two systems, without comparison against a simple reserved-seed and balanced-evaluation protocol or controlled measurements of false and missed promotions. In Case C, the results reveal evaluation imbalance but do not establish that it caused the 124-generation freeze. The synthetic experiment fails to reproduce the predicted effect size, while Case D provides limited evidence across independent training runs. Controlled comparisons of the same candidates under the original and corrected protocols, along with broader validation across tasks, would strengthen the evidence for PGV's practical value.
[1]: Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11:2079-2107, 2010.
[2]: Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how to tune them. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 9104-9149, 2023.
Questions for authors
-
What is the principal methodological advance beyond separating selection and evaluation data and balancing structural evaluation conditions? Can you identify a generalizable insight or capability that distinguishes PGV from applying these practices directly to a promotion gate?
-
Can you compare PGV with a reserved-seed and balanced-evaluation protocol under matched evaluation budgets, measuring false and missed promotions using independently assessed candidate strengths? In particular, evaluating the same Boom candidates under both protocols would help test the proposed cause of the freeze, while additional independent training runs and task settings would clarify generality.
Correction: Case D does not hold. Then your two questions
Thank you for the review. One correction first, then your questions.
Correction: Case D does not hold. Section 6 says that our Mahjong simulator did not score some legal wins on a discard, and argues that each verdict stands because it is a comparison within that one simulator. For Case D the verdict did not stand. After submission we replaced the scorer with one checked against the official judge: on the 12,288 games of the competition's final it misses 2 of 12,278 legal wins and accepts no illegal one, where the scorer behind the paper misses 2,576 (21%). We then re-evaluated the two independently trained policies of Table 3 (stored checkpoints, not retrained) against the imitation baseline that both runs started from, each on the same 24 blocks of 2,000 games. The deals (the paper's "walls") were new to training and to every earlier evaluation, and the decision rule (interval above, containing or below 2.500) was fixed before any game was played.
Every interval now contains the 2.500 tie, so by that rule each result is a tie with the baseline. The re-run changed the scorer, the deals and the evaluation code together, and we have not run the old scorer on the new deals, so we cannot say how much of the change comes from each. Either way, the improvement that Case D calls surviving is not supported. The five single-lineage gates in Table 3 used the uncorrected scorer and have not been re-run, so we no longer rely on them. Appendix A's analyses were not re-run; for the two replicas, fan per win is lower than the baseline's, not higher.
What this does to the four cases. Case D was the only one in which a claim passed the check and was presented as real, so the "two-sided validation" that Section 2 names as our contribution is gone. By the paper's account Case D also satisfied both checks, and neither could have flagged it. Cases A and B are statements about seed records and about replication under one scorer. The correction does not undo them, but their placement values were measured with the uncorrected scorer and have not been re-run. Case C concerns a different system. (Our reply to Reviewer hE9C also corrects three descriptions in Sections 4.1 and 4.2, Table 2 and Figure 2; no placement mean or interval changes and the 14 of 24 overlap stands.) We have not uploaded a revised PDF or supplement; the PDF on this page still prints the old Case D.
Weakness 1 / Q1. You are right, and we cannot name an advance beyond the two practices. The first rule is the practice you cite (Section 2 cites both papers), and the second is ordinary balanced evaluation; the paper writes them as tests on the seed and condition records a gate already keeps. Section 2 also says we found no earlier written statement of the replication condition or of the symmetry rule as a check. That describes our search; it is not an advance. Beyond that the paper claimed the two-sided validation, and the correction above removes its second side. The corrected Case D is also a concrete answer to your "which failures they cannot detect": a gate can satisfy both checks and still certify an improvement that does not hold up, for instance when the scorer itself is wrong.
Weakness 2 / Q2. We have not run that comparison, and we have no measurement of false or missed promotions; Section 6 says that four incidents cannot estimate either error rate and that we compare with no alternative safeguard. Applied going forward, the two checks are the protocol you describe. The comparison you ask for is a gate with and without it, at the same budget, against independently known strengths, and we have not run it. On Boom, we have no re-evaluation of the blocked candidates under a mirrored gate with the other settings unchanged, so Case C shows that the gate was imbalanced and does not show that the imbalance caused the freeze. We agree with your reading of the synthetic experiment (an estimated 1.14× against a predicted 3× or more). After the correction, three incidents remain, all found after the fact, in two systems. That is not evidence of generality.
We do not dispute your assessment.