acceptodds
Under review as a conference paper at ICLR 2027

PGV: A Two-Check Verification Harness for Self-Play Promotion Gates, Validated on Four Real Incidents in Deployed Systems

Abstract

Competitive self-play reinforcement learning relies on a promotion gate to decide which candidate policy is actually better. When the gate's own measurement is wrong in a way that is not obviously noise, the loop can silently stall, silently promote a policy that has not improved, or silently reuse data it presents as independent. We introduce PGV, a lightweight verification harness of two offline checks for self-play promotion gates, each a one-line invariant over bookkeeping the gate already keeps: the Disjoint-Reseed Check (a candidate's certifying data, and any data offered as an independent replication, must occupy regions of the seed space disjoint from the selection sweep and from each other) and the Train/Eval Symmetry Check (every structural degree of freedom the evaluation condition exposes - seat, side, deck - must be sampled the way training samples it). Neither check was run in either of two independently built, deployed self-play systems we studied. We validate PGV on four real incidents rather than a synthetic benchmark. The Disjoint-Reseed Check flags two real certification failures - a candidate certified as best of roughly twenty with a +0.0054 placement-point edge that falls to -0.0005 (interval spanning zero) under selection-disjoint fresh evaluation, consistent with a winner's curse, and a seed-stepping bug that made a nominal "independent replication" silently reuse 58.33% of the original run's blocks - and, applied to a different, later improvement claim in the same system, passes: five mutually disjoint evaluations all clear the calibrated tie point, so the check does not drive every claim to null (of two independently trained replicas, one clears on its own; their pooled estimate clears). The Train/Eval Symmetry Check would have flagged the imbalanced gate condition behind a 124-generation freeze of an independently built card-game ladder, diagnosed from a seat-conditional win-rate split (0.14 vs. 0.66); the mahjong gate avoids the identical mismatch by explicit design, and a pre-registered synthetic reproduction shows the mechanism's direction only at larger biases while falsifying its own pre-registered magnitude threshold, reported as falsified. Every number traces to a committed file or a pinned upstream commit, and an evidence plan frozen before the analyses fixed how each claim is narrowed when its own check comes back negative, which happened twice and is reported both times.

Primary area:
reinforcement learning
Keywords:
self-play, reinforcement learning evaluation, promotion gate, winner's curse, verification, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_6dUXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes PGV, a lightweight verification harness for promotion gates in competitive self-play. Its two checks examine whether certification and replication seed ranges are disjoint from selection data and whether evaluation samples structural conditions, such as seats and decks, consistently with training. The evidence comprises four incidents from two deployed systems: a selected mahjong candidate whose reported advantage disappears on fresh evaluation, substantial seed reuse in a nominal replication, a later mahjong improvement that survives disjoint evaluations, and a card-game ladder freeze associated with asymmetric evaluation. A synthetic reproduction finds a smaller directional effect than predicted and falsifies its preregistered magnitude threshold.

Critical strengths

The paper addresses a practically important failure point in self-play systems. Auditing the evaluator's seed allocation and role assignments is inexpensive relative to training, and these implementation errors can be difficult to diagnose from aggregate scores alone.

Critical weaknesses
  1. The methodological contribution is limited. Selection-induced optimism is well established in model-selection research, including Cawley and Talbot (2010)[1], and separating tuning and testing seeds is explicitly recommended for RL by Eimer et al. (2023)[2]. The paper acknowledges this background, and its main contribution is to express data separation and balanced evaluation as two checks on a self-play gate's bookkeeping. This is a useful engineering practice, but the paper provides limited new methodological insight into when these checks are sufficient, which failures they cannot detect, or how they improve promotion decisions beyond straightforward evaluation safeguards. I do not find the adaptation of these principles to promotion gates alone a sufficiently substantial contribution.

  2. The experiments do not sufficiently establish the effectiveness and generality of PGV. The evidence is limited to four incidents from two systems, without comparison against a simple reserved-seed and balanced-evaluation protocol or controlled measurements of false and missed promotions. In Case C, the results reveal evaluation imbalance but do not establish that it caused the 124-generation freeze. The synthetic experiment fails to reproduce the predicted effect size, while Case D provides limited evidence across independent training runs. Controlled comparisons of the same candidates under the original and corrected protocols, along with broader validation across tasks, would strengthen the evidence for PGV's practical value.

[1]: Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11:2079-2107, 2010.

[2]: Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how to tune them. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 9104-9149, 2023.

Questions for authors
  1. What is the principal methodological advance beyond separating selection and evaluation data and balancing structural evaluation conditions? Can you identify a generalizable insight or capability that distinguishes PGV from applying these practices directly to a promotion gate?

  2. Can you compare PGV with a reserved-seed and balanced-evaluation protocol under matched evaluation budgets, measuring false and missed promotions using independently assessed candidate strengths? In particular, evaluating the same Boom candidates under both protocols would help test the proposed cause of the freeze, while additional independent training runs and task settings would clarify generality.

1 comment
AuthorsOpenReview ↗

Correction: Case D does not hold. Then your two questions

Thank you for the review. One correction first, then your questions.

Correction: Case D does not hold. Section 6 says that our Mahjong simulator did not score some legal wins on a discard, and argues that each verdict stands because it is a comparison within that one simulator. For Case D the verdict did not stand. After submission we replaced the scorer with one checked against the official judge: on the 12,288 games of the competition's final it misses 2 of 12,278 legal wins and accepts no illegal one, where the scorer behind the paper misses 2,576 (21%). We then re-evaluated the two independently trained policies of Table 3 (stored checkpoints, not retrained) against the imitation baseline that both runs started from, each on the same 24 blocks of 2,000 games. The deals (the paper's "walls") were new to training and to every earlier evaluation, and the decision rule (interval above, containing or below 2.500) was fixed before any game was played.

  • Replica 1: 2.5036 [2.4986, 2.5087] in Table 3; now 2.4971 [2.4910, 2.5031].
  • Replica 2: 2.5070 [2.5021, 2.5119]; now 2.4970 [2.4928, 2.5011].
  • Pooled: 2.5053 [2.5019, 2.5087] (36 blocks); now 2.4970 [2.4924, 2.5017] (mean of the two per block, same 24 blocks).

Every interval now contains the 2.500 tie, so by that rule each result is a tie with the baseline. The re-run changed the scorer, the deals and the evaluation code together, and we have not run the old scorer on the new deals, so we cannot say how much of the change comes from each. Either way, the improvement that Case D calls surviving is not supported. The five single-lineage gates in Table 3 used the uncorrected scorer and have not been re-run, so we no longer rely on them. Appendix A's analyses were not re-run; for the two replicas, fan per win is lower than the baseline's, not higher.

What this does to the four cases. Case D was the only one in which a claim passed the check and was presented as real, so the "two-sided validation" that Section 2 names as our contribution is gone. By the paper's account Case D also satisfied both checks, and neither could have flagged it. Cases A and B are statements about seed records and about replication under one scorer. The correction does not undo them, but their placement values were measured with the uncorrected scorer and have not been re-run. Case C concerns a different system. (Our reply to Reviewer hE9C also corrects three descriptions in Sections 4.1 and 4.2, Table 2 and Figure 2; no placement mean or interval changes and the 14 of 24 overlap stands.) We have not uploaded a revised PDF or supplement; the PDF on this page still prints the old Case D.

Weakness 1 / Q1. You are right, and we cannot name an advance beyond the two practices. The first rule is the practice you cite (Section 2 cites both papers), and the second is ordinary balanced evaluation; the paper writes them as tests on the seed and condition records a gate already keeps. Section 2 also says we found no earlier written statement of the replication condition or of the symmetry rule as a check. That describes our search; it is not an advance. Beyond that the paper claimed the two-sided validation, and the correction above removes its second side. The corrected Case D is also a concrete answer to your "which failures they cannot detect": a gate can satisfy both checks and still certify an improvement that does not hold up, for instance when the scorer itself is wrong.

Weakness 2 / Q2. We have not run that comparison, and we have no measurement of false or missed promotions; Section 6 says that four incidents cannot estimate either error rate and that we compare with no alternative safeguard. Applied going forward, the two checks are the protocol you describe. The comparison you ask for is a gate with and without it, at the same budget, against independently known strengths, and we have not run it. On Boom, we have no re-evaluation of the blocked candidates under a mirrored gate with the other settings unchanged, so Case C shows that the gate was imbalanced and does not show that the imbalance caused the freeze. We agree with your reading of the synthetic experiment (an estimated 1.14× against a predicted 3× or more). After the correction, three incidents remain, all found after the fact, in two systems. That is not evidence of generality.

We do not dispute your assessment.

Reviewer_CGpFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a lightweight verification framework to identify problems in the promotion gates of self-play RL. It uses some ML rules for this verification (e.g. eval should be similar to training, seeds should be non-overlapping between training and testing, etc.). It reports evaluation of this framework on 4 real incidents.

Critical strengths
  1. The paper tackles an important problem in RL
  2. Evaluation uses real incidents.
Critical weaknesses
  1. The novelty is quite limited, as it uses existing ML practices to make a decision.
  2. The paper does not use any other baselines for comparison.
  3. While the real-case studies are commendable, I am unsure if the conclusions can be generalizable from just 4 case studies on two self-play games (Mahjong and Boom).
  4. The system appears to be offline, but to show real impact it would be good to integrate it during training, in an online manner
Questions for authors
  1. Can the authors evaluate PGV on additional self-play environments or provide evidence that the identified failure modes generalize beyond the two systems studied?
  2. Can the authors integrate PGV into an active self-play training pipeline and measure its effect on incorrect promotions, training stagnation, and final policy performance?
1 comment
AuthorsOpenReview ↗

Correction: Case D does not hold. Your four points are fair

Thank you for the review. One correction first, then your points.

Correction: Case D does not hold. Section 6 says that our Mahjong simulator did not score some legal wins on a discard, and argues that each verdict stands because it is a comparison within that one simulator. For Case D the verdict did not stand. After submission we replaced the scorer with one checked against the official judge: on the 12,288 games of the competition's final it misses 2 of 12,278 legal wins and accepts no illegal one, where the scorer behind the paper misses 2,576 (21%). We then re-evaluated the two independently trained policies of Table 3 (stored checkpoints, not retrained) against the imitation baseline that both runs started from, each on the same 24 blocks of 2,000 games. The deals (the paper's "walls") were new to training and to every earlier evaluation, and the decision rule (interval above, containing or below 2.500) was fixed before any game was played.

  • Replica 1: 2.5036 [2.4986, 2.5087] in Table 3; now 2.4971 [2.4910, 2.5031].
  • Replica 2: 2.5070 [2.5021, 2.5119]; now 2.4970 [2.4928, 2.5011].
  • Pooled: 2.5053 [2.5019, 2.5087] (36 blocks); now 2.4970 [2.4924, 2.5017] (mean of the two per block, same 24 blocks).

Every interval now contains the 2.500 tie, so by that rule each result is a tie with the baseline. The re-run changed the scorer, the deals and the evaluation code together, and we have not run the old scorer on the new deals, so we cannot say how much of the change comes from each. Either way, the improvement that Case D calls surviving is not supported. The five single-lineage gates in Table 3 used the uncorrected scorer and have not been re-run, so we no longer rely on them. Appendix A's analyses were not re-run; for the two replicas, fan per win is lower than the baseline's, not higher.

What this does to the four cases. Case D was the only one in which a claim passed the check and was presented as real, so the "two-sided validation" that Section 2 names as our contribution is gone. By the paper's account Case D also satisfied both checks, and neither could have flagged it. Cases A and B are statements about seed records and about replication under one scorer. The correction does not undo them, but their placement values were measured with the uncorrected scorer and have not been re-run. Case C concerns a different system. (Our reply to Reviewer hE9C also corrects three descriptions in Sections 4.1 and 4.2, Table 2 and Figure 2; no placement mean or interval changes and the 14 of 24 overlap stands.) We have not uploaded a revised PDF or supplement; the PDF on this page still prints the old Case D.

On your four points:

  1. Novelty. Agreed. The first rule is seed separation, which is existing practice, and the second is ordinary balanced evaluation. Section 2 says we found no earlier written statement of the second as a check; that describes our search and is not an advance.
  2. Baselines. There are none. Section 6 says we compare with no alternative safeguard.
  3. Generality / Q1. Two games and, after the correction, three incidents, all found after the fact. We have no additional environment, and we cannot claim that these failure modes are common elsewhere.
  4. Online use / Q2. In Cases A to C the checks were run on stored records, after the fact. Case D's evaluations were laid out in advance to satisfy the first check, and Case D did not hold. We have not put the checks inside a training loop, and we have no measurement of their effect on wrong promotions, stagnation or final strength.

One clarification, because our wording invites the reading in your summary: the first check is not about training seeds versus test seeds. It concerns three sets of evaluation deals: those on which a candidate was picked among many, those used to certify it, and those offered as a replication (Section 3). This does not change your assessment, which we do not dispute.

Reviewer_hE9COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces PGV, an offline auditing harness for self-play policy-promotion pipelines. Its two checks examine whether selection, certification, and claimed-independent evaluations use disjoint seed data, and whether structural evaluation conditions match those used in training. The study reports four cases from two independently built systems: two certification problems and a surviving improvement claim in a mahjong pipeline, and a retrospective diagnosis of a card-game ladder freeze. An auxiliary synthetic experiment investigates condition-induced promotion delay but falsifies its pre-specified magnitude prediction. The contribution is primarily operational and empirical rather than a new training algorithm or statistical theory.

Critical strengths
  1. A practically relevant audit with concrete evidence of data reuse. The paper investigates evaluation failures that aggregate win rates may conceal. Case B gives a specific seed-allocation explanation for 14 of 24 replication blocks being reused, supported by reported matching block scores. The overlap fraction can be checked from the printed seed parameters. This provides a concrete example of why auditing evaluation provenance can be useful.

  2. Transparent treatment of adverse evidence and study limitations. The paper reports the synthetic experiment’s failed magnitude prediction, an independently trained replica that does not individually clear the tie point, and an unresolved ablation caused by unavailable weights. It also distinguishes retrospective analysis from prospective evidence. These disclosures improve the credibility of the limited findings, even though they do not resolve the validation concerns below.

Critical weaknesses
  1. The research gap and contribution are not communicated clearly enough to assess their significance. This is my principal concern. The abstract and Introduction do not provide a clear progression from the precise evaluation problem, through the relevant existing approaches and their remaining limitations, to PGV’s specific advance. The approximately 370-word abstract includes a result sentence exceeding 100 words and combines several cases, statistical qualifications, and provenance details. Compressed expressions such as “one-line invariant over bookkeeping” and “imbalanced gate condition” require substantial interpretation. The Introduction similarly mixes motivation, system descriptions, analysis-plan details, and contribution statements. Its contribution list does exist, but it is embedded in a dense paragraph and mixes completed work with an unperformed ablation. Related Work supplies relevant precedents, including seed separation, but the practical increment beyond correctly applying those principles remains difficult to identify. A new statistical theorem is not necessary for an engineering or empirical contribution; the paper nevertheless needs to establish what reusable capability or substantive knowledge it adds. These problems require restructuring and precise explanations, not merely isolated copyediting.

  2. The checks’ definitions and experimental interpretations leave important validity questions unresolved. The main concerns are the links between an audited bookkeeping property, the intended performance claim, and the evidence used to validate it:

    • Section 3: The definition does not clearly distinguish actually exposed selection seeds from a reserved numerical region. Numerical separation alone is not a statistical independence argument, and unseen seeds within an old region are not automatically selection-contaminated. The symmetry check also needs a specified evaluation target before requiring evaluation to match training: a curriculum distribution and a balanced evaluation distribution can legitimately differ. Finally, mismatch does not by itself imply a skill-independent ceiling. In Figure 4’s finite additive-logit model, the pooled win probability tends to one as skill difference increases. The claimed hard cap therefore requires additional conditions or narrower wording.

    • Section 4: Case A compares a best-of-roughly-20 selected estimate with fresh measurements using normal approximations reconstructed from marginal intervals. The nominal arithmetic is consistent, but Holm correction over three later comparisons does not by itself establish calibration for the preceding selection. The reported decay is consistent with winner’s curse; stronger inferential validation requires a selection-aware analysis or a more limited interpretation. In Case D, pooling 36 evaluation blocks from two trained replicas can estimate their conditional average performance, but does not establish uncertainty or reliability across new training runs. The authors acknowledge the single lineage, the mixed replica outcomes, and post-hoc pooling; the narrow positive-case finding should be preserved without implying stronger training-level replication.

    • Section 5: The real mirrored-deck correction is not explicitly mapped to the toy’s comparison between a pooled positive/negative-bias probability and an unbiased logistic probability. The manuscript also omits key settings needed to interpret the duration result, including the gate budget, decision rule, skill trajectory, cadence, and exact definition of “generations frozen.” A supplementary script is reported, but was not inspected here; the issue is the manuscript’s self-contained experimental specification, not a claim that the implementation is absent. The ratio-of-medians estimate, paired bootstrap interval, and sign test also need their distinct targets stated. A within-bias arm comparison is not the same as an additional effect beyond the zero-bias control; the latter requires an appropriate joint contrast if it is intended. The control’s realized ratio of 1.09 does not itself prove confounding.

These concerns affect the interpretation of the principal validation, rather than only presentation. The retrospective freeze diagnosis and the disclosed synthetic magnitude failure support narrower conclusions than a general prevention or recovery result. I therefore do not find the present exposition and evidence sufficient for acceptance.

Questions for authors

These are clarification requests about the current claims and analyses; I am not asking for a new large-scale study during the response period.

  1. Seed provenance and Case A inference: Does the selection region mean actually observed seed intervals or an entire reserved numerical partition? Were the “same-region fresh” seeds exposed during selection? What null and selection-aware justification support the decisive original-versus-fresh comparison after best-of-roughly-20 selection?

  2. Evaluation target and the ceiling claim: What performance distribution and role/deck weights justify requiring evaluation to match training? Under what explicit assumptions does a mismatch make the promotion threshold unattainable, rather than attenuating improvement over a finite skill range?

  3. Case D’s replication target: Is the pooled interval intended only to describe these two realized policies, or to support expected improvement across independent training runs? How is between-training variability distinguished from evaluation noise in the interpretation?

  4. Synthetic intervention and protocol: Which exact structural-condition violation and correction does each arm represent, and how do these map to the real mirrored-deck operation? Please specify the game budget, Wilson rule, promotion threshold, skill evolution, cadence, incumbent updates, and freeze-duration definition.

  5. Synthetic statistical target: Does the bootstrap interval cover a ratio of arm medians or a median of paired ratios? Is the directional conclusion a fixed-bias arm contrast, or an additional effect relative to the zero-bias control? If the latter, is there a joint seed-level contrast with uncertainty?

2 comments
AuthorsOpenReview ↗

Correction: Case D does not hold. Then your five questions

Thank you for the careful review. One correction first.

Correction: Case D does not hold. Section 6 says our Mahjong simulator did not score some legal wins on a discard, and argues that each verdict stands because it is a comparison within one simulator. For Case D the verdict did not stand. After submission we replaced the scorer with one checked against the official judge (on the 12,288 games of the competition's final it misses 2 of 12,278 legal wins and accepts no illegal one; the paper's scorer misses 2,576) and re-evaluated the two replica policies of Table 3 (stored checkpoints, not retrained) against the imitation baseline that both runs started from, each on the same 24 new blocks of 2,000 games, with the decision rule fixed beforehand. Replica 1 is now 2.4971 [2.4910, 2.5031], Replica 2 2.4970 [2.4928, 2.5011], and their per-block mean 2.4970 [2.4924, 2.5017] (Table 3 pooled, 36 blocks: 2.5053 [2.5019, 2.5087]); all contain the 2.500 tie. Scorer, deals (the paper's "walls") and evaluation code changed together, so we cannot separate them. Either way the improvement is not supported, and the single-lineage gates have not been re-run. No case is left in which a claim passes the check and is shown to be real. The seed facts of Cases A and B do not depend on the scorer, though their placement values come from the old scorer; Case C is another system. No revised PDF or supplement is uploaded.

Weakness 1. Agreed: the abstract is too long, the contribution list includes an unrun ablation, and the gap is never stated. After the correction, two after-the-fact catches and one after-the-fact diagnosis are left.

Q1. You are right: "selection region" in the paper is a numeric range, the wrong object, since unseen seeds in an old range are as good as any others. The condition should be over the deals actually played when the candidate was chosen. Were the same-region seeds exposed? Partly, and Section 4.1's "new wall seeds" is inaccurate. The stored per-block files show that this re-gate played one continuous range of 48,000 deals (seeds 330,000 to 377,999). It contains all 12,000 deals of the original certifying gate; in all, 19,000 of its deals were also played in other stored gates (14,000 by this candidate). So its +0.0017 is a mixture, and the z≈1.5z \approx 1.5 against the certified row compares overlapping samples. Also: the two fresh rows of Table 2 use blocks of 8,000 games, so each has 192,000 games (48,000 is the number of deals); and the original gate's first six blocks start at seeds 330,000 to 335,000, not 350,000 to 355,000 as Section 4.2 and Figure 2 show. These files are not in the supplement, whose summary files carry the same errors. No placement mean or interval changes, and the 14 of 24 overlap stands (shared blocks at seeds 360,000 to 373,000, identical scores). The certified-versus-disjoint test (non-overlapping deals; null: equal means) shows only that the certified value did not reproduce, as winner's curse predicts. It is not selection-aware, and we accept the more limited interpretation.

Q2. The paper gives no evaluation target and needs one. Nothing requires evaluation to match training; the gate should match the condition the promotion decision is about. On the ceiling you are right: the additive model has none. The threshold is unattainable only if the per-role win rates are bounded above whatever the skill, tightly enough that the pooled rate cannot reach it. We measured no such bound. "Regardless of true skill" is wrong as written.

Q3. Only the two realized policies. Two replicas cannot separate between-training variability from evaluation noise, and the pooled interval no longer clears the tie.

Q4. From the supplement's preregistration. Skill difference starts at −0.6-0.6 and gains a normal step (mean 0.01, s.d. 0.03) each generation. Each generation plays 200 games. Biased arm: 100 in each of two roles at σ(Δ−b)\sigma(\Delta-b) and σ(Δ+b)\sigma(\Delta+b), pooled (the original gate, where the two roles held different decks). Mirrored arm: all 200 at σ(Δ)\sigma(\Delta) (the corrected gate, which pooled the two same-deck pairings; one unbiased rate in the toy). Promotion when the Wilson 95% lower bound on the pooled win rate exceeds 0.5. A run stops at its first promotion, so the incumbent is never updated; "generations frozen" counts the generations before it. 40 seeds, each skill path replayed through both arms; the 500-generation cap was never reached.

Q5. The interval is for the ratio of the two arm medians (paired percentile bootstrap over seeds), not a median of paired ratios. The sign test is on per-seed differences within one bias level. So the directional statement is a within-level arm contrast; we computed no joint contrast against the zero-bias control. At b=0b=0 the two arms have the same distribution, so the control's 1.09× is sampling noise in this ratio, and we agree it shows no confounding.

As the authors have acknowledged some of the weaknesses I identified, I believe the paper still requires substantial revisions and further evaluation. Therefore, I will maintain my score of 1.

Reviewer_M5BVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PGV, two checks for auditing self-play promotion gates. One checks for overlap between the seeds used for selection, certification, and replication, while the other checks whether evaluation samples seats, sides, and decks consistently with training. The authors examine four case studies in mahjong and card-game systems. They also test a synthetic version of the freeze mechanism — for which the measured effect falls below their preregistered prediction.

Critical strengths
  • The authors report some negative results, which is useful from a scientific perspective and makes some of reported results easier to interpret.
Critical weaknesses
  • The paper mostly applies established evaluation practices, which doesn't rise to the level of a meaningful contribution. Separating tuning and testing seeds is already recommended in RL [1], and separating selection from held-out reporting is central to SIREN [2]. The cases also have limited practical value as cases A–C are retrospective, cases A and B concern the same selected candidate, and Case D stays within the same system.

  • Section 3 says that asymmetry caps performance "regardless of true skill" but (σ(Δ−b)+σ(Δ+b))/2(\sigma(\Delta-b)+\sigma(\Delta+b))/2 approaches one as Δ\Delta increases. Thus, a sufficiently large skill difference overcomes the bias. In Case C, a corrected self-match near 0.5 doesn't show that any blocked candidate was stronger than the policy it was trying to replace or would have passed a symmetric gate. Finally, the synthetic experiment has a freeze ratio of 1.14x while the prediction was at least 3x. So neither the model nor the reported evaluations establish that the asymmetry caused the 124-generation freeze. The authors acknowledge these limitations in Section 5, but the abstract and Section 3 still overstate what the results show.

[1] Eimer et al. Hyperparameters in Reinforcement Learning and How to Tune Them. ICML 2023.

[2] Xu et al. Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking. arXiv 2026.

Questions for authors
  1. Did you re-evaluate any candidates blocked during the Boom freeze with mirrored decks, keeping the opponent, number of games, and promotion threshold unchanged? If so, did any pass?

  2. What supports the claim that the deck imbalance makes the promotion threshold unreachable? In the synthetic model, a sufficiently large skill difference overcomes the bias. Is there a property of Boom that prevents this but is missing from the model?

2 comments
AuthorsOpenReview ↗

Correction: Case D does not hold. "Regardless of true skill" is wrong as written

Thank you for the review. One correction first, then your points.

Correction: Case D does not hold. Section 6 says that our Mahjong simulator did not score some legal wins on a discard, and argues that each verdict stands because it is a comparison within that one simulator. For Case D the verdict did not stand. After submission we replaced the scorer with one checked against the official judge: on the 12,288 games of the competition's final it misses 2 of 12,278 legal wins and accepts no illegal one, where the scorer behind the paper misses 2,576 (21%). We then re-evaluated the two independently trained policies of Table 3 (stored checkpoints, not retrained) against the imitation baseline that both runs started from, each on the same 24 blocks of 2,000 games. The deals (the paper's "walls") were new to training and to every earlier evaluation, and the decision rule (interval above, containing or below 2.500) was fixed before any game was played.

  • Replica 1: 2.5036 [2.4986, 2.5087] in Table 3; now 2.4971 [2.4910, 2.5031].
  • Replica 2: 2.5070 [2.5021, 2.5119]; now 2.4970 [2.4928, 2.5011].
  • Pooled: 2.5053 [2.5019, 2.5087] (36 blocks); now 2.4970 [2.4924, 2.5017] (mean of the two per block, same 24 blocks).

Every interval now contains the 2.500 tie, so by that rule each result is a tie with the baseline. The re-run changed the scorer, the deals and the evaluation code together, and we have not run the old scorer on the new deals, so we cannot say how much of the change comes from each. Either way, the improvement that Case D calls surviving is not supported. The five single-lineage gates in Table 3 used the uncorrected scorer and have not been re-run, so we no longer rely on them. Appendix A's analyses were not re-run; for the two replicas, fan per win is lower than the baseline's, not higher.

What this does to the four cases. Case D was the only one in which a claim passed the check and was presented as real, so the "two-sided validation" that Section 2 names as our contribution is gone. By the paper's account Case D also satisfied both checks, and neither could have flagged it. Cases A and B are statements about seed records and about replication under one scorer. The correction does not undo them, but their placement values were measured with the uncorrected scorer and have not been re-run. Case C concerns a different system. (Our reply to Reviewer hE9C also corrects three descriptions in Sections 4.1 and 4.2, Table 2 and Figure 2; no placement mean or interval changes and the 14 of 24 overlap stands.) We have not uploaded a revised PDF or supplement; the PDF on this page still prints the old Case D.

First weakness. Agreed on each count. The paper applies established practice, as you say, and Section 2 cites both papers you name. Cases A to C are retrospective. Case B is the replication of Case A's own gate, so the two concern one candidate. Case D came from the same system and, as above, does not hold.

Second weakness. You are right, and Section 3's "regardless of true skill" is wrong as written. In the additive model, (σ(Δ−b)+σ(Δ+b))/2→1(\sigma(\Delta-b)+\sigma(\Delta+b))/2 \to 1 as Δ\Delta grows, so the bias delays a promotion and never caps it; our own Figure 4 shows this. A cap needs something we never showed: limits on the per-role win rates that hold however strong the candidate is, tight enough to keep the pooled rate under the threshold. The abstract's "the imbalanced gate condition behind a 124-generation freeze" likewise states a cause that Section 5 does not establish. What the evidence supports is narrower: the gate was imbalanced during the freeze, and in a simple model an imbalance of that size lengthens the wait for a promotion by an estimated factor of 1.14 (bootstrap interval 1.05 to 1.34), against a predicted 3 or more.

Q1. No. We have no re-evaluation of a blocked candidate with mirrored decks and the same opponent, number of games and threshold. The only mirrored evaluation we can report is the one in Section 5.1 (about 0.53, 95% lower bound 0.41). It did not pass, and as you say it does not show that any blocked candidate was stronger than the policy it tried to replace. The Case C numbers come from a postmortem record, not from stored gate logs (Section 6).

Q2. Nothing we measured supports "unreachable". The 0.14 and 0.66 rates are one snapshot of one candidate. They show a large imbalance at that moment, not a ceiling. A hard per-seat ceiling in Boom is one of the untested alternatives named at the end of Section 5.2. The model has no such property, and we did not test for one.

We do not dispute your assessment.

Thank you to the authors for their response. Given their broad agreement regarding weaknesses, I will maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up: the old scorer on the new deals

In our replies we said that the re-run changed the scorer, the deals and the evaluation code together, and that we had not run the old scorer on the new deals. We have now run it: the same two replicas, the same baseline, the same 24 blocks of 2,000 games on the new deals and the same evaluation code, with only the scorer switched back to the one behind the paper. The decision rule was fixed before any game was played.

Identical policies again give exactly 2.500 on every deal. With the old scorer:

  • Replica 1: 2.5053 [2.5006, 2.5099]; with the judge-checked scorer, 2.4971 [2.4910, 2.5031].
  • Replica 2: 2.5043 [2.4998, 2.5087]; with the judge-checked scorer, 2.4970 [2.4928, 2.5011].
  • Pooled: 2.5048 [2.5008, 2.5087]; with the judge-checked scorer, 2.4970 [2.4924, 2.5017]. Table 3 printed 2.5053 [2.5019, 2.5087] on its own deals.

Paired by block on the same deals, the old scorer adds 0.0077 [0.0043, 0.0112] to the pooled placement.

So the scorer accounts for the change. On the same deals, the scorer that misses legal wins puts the replicas above the 2.500 tie, and the judge-checked scorer makes them a tie: the improvement that Case D reported came from the scorer's error. Under the old scorer, Replica 2's own interval still contains 2.500; the pooled one does not.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.