- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies common-random-number coupling of counterfactual rollouts for policy-gradient learning in two-player imperfect-information games. It defines five admissibility conditions, shows that under marginal preservation the projected gradient noise differs across couplings only through gradient-weighted off-diagonal return covariances, and proposes OSCC-Select, which picks among (independent, root-only, continuation-only, full) coupling using simultaneous Hoeffding lower bounds with an independent fallback. CP-GRPO instantiates full coupling.
Critical strengths
None
Critical weaknesses
- Related work missing.
(a) Training-time variance reduction in imperfect-information games is absent: MCCFR [1] and variance-reduced MCCFR estimators [2–5] are not cited; AIVAT appears only as an evaluation tool.
(b) Imperfect-information policy-gradient work is absent [6–8]. Most importantly, [9] identifies the same bottleneck—return-variance inflation of PPO in IIG self-play caused by stochastic equilibrium policies—and fixes it by averaging over future action draws with a Q-critic (Q-boosting/VRPO). OSCC attacks a complementary noise source (shared exogenous chance across branches) and, by C3, deliberately leaves policy-side action noise uncoupled; the two methods are natural to combine, yet [9] is neither cited nor compared.
(c) [10], a classical CRN-for-policy-search precedent, is missing from the paragraph that claims to list "the closest statistical precedents," and the CFR+ paper [11] is not cited although "exact CFR+" anchors Table 24.
-
No external baseline for either headline. The 55.87% figure is compared only against OSCC-internal arms; there is no antithetic, stratified, or control-variate arm under the same fixed-root estimand and physical-call budget, so the reader cannot tell whether this is the ordinary CRN gain or something specific to the proposed contract. The policy screen compares CP-GRPO only to generic PPO/GRPO and an exact CFR+ anchor; no IIG-specific PG method (VRPO [9] or ACH [7]) is included. Note also that 55.87% is return-contrast variance; the realized update-variance ratio is 0.620 (Table 36), i.e. a 38% reduction, and the abstract reports only the former.
-
The learning claim is not supported by the paper's own evidence. §4.1 calls the three-seed tabular screen "authoritative," and §4.2.2 reports that the independent arm is better on all three seeds. Table 20's five-seed PPO connectivity intervals "include zero." Further, OSCC-Select is never used in any training run, and all four 2×2 arms use complete-action enumeration, so there is no coupled + sampled-action arm. The title's "Imperfect-Information Learning" and the "general learning primitive" framing are therefore not tested.
-
Novelty is overstated relative to the content, and prior art is misdescribed. Prop. 3 is the expansion of tr(GΣG^T) with coupling-invariant diagonals; Thm. 1 is a Bonferroni-corrected Hoeffding bound over four candidates. C2 is the definition of a behavioral strategy under perfect recall; C4's per-event persistent chance addressing overlaps concurrent work on correlated/persistent chance sampling [5]. Table 8's "optimization objective" column lists "paired contrast variance," "scalar return variance," and "trust-region update signal" for prior methods and "gradient noise plus physical cost" for OSCC alone; in those works CRN is a variance-reduction device for an expected-return or policy-gradient estimator, and the TRPO vine procedure explicitly uses common random numbers to reduce the variance of Q-value differences, i.e. it is already CRN-based advantage estimation. Yadav et al. is cited in one sentence although its intermediate-coupling estimator directly anticipates the OSCC-Select candidate set. The standard source for score-function gradient variance reduction [12] is also absent.
-
Internal inconsistencies that undermine the "auditable ledger" positioning. Table 2's reported means (0.01421 / 0.03487) are not the check − bet contrast means implied by §4.2.1's branch means (0.17273 / 0.20437); they track the check-branch means, and the "separate aggregation units" remark does not resolve a 12× discrepancy. §4.3.2 quotes 41.0876 as the 2×2 independent control, but Tables 28/30 give 41.1158; 41.0876 is the Table 23 val./test value; Table 37 gives 41.1087. Appendix C.1 states that "CP-GRPO changes only the coupling assignment in the rollout collector," whereas §3.7 lists complete-action enumeration, frozen returns, and behavior weights as additional changes relative to standard GRPO. Both cannot be true. In Tables 28/30 the 30,720-key independent row reproduces the 100,000-group audit's covariance and variance digit-for-digit (−0.11495, 41.1158), while the CP-GRPO row's variance (18.3269) matches Table 23, not the 100k audit (18.1441); the independent arm's update variance is 0.1835 in Table 22 and 0.1398 in Table 30. Table 38 reports 500 solver queries (498 successful) against Table 21's 5 seeds × 512 held-out roots. "Exact CFR+ reference" has residual exploitability 0.0152 on Leduc (Table 12), 12.8% of CP-GRPO's 0.1184; it is a high-precision iterative solution, not an exact one.
-
The paper writing is terrible, the experimental results are scattered everywhere, and humans can't understand what they're doing. Experiments are limited to Leduc and Kuhn.
[1] M. Lanctot et al. Monte Carlo Sampling for Regret Minimization in Extensive Games. NeurIPS 2009.
[2] R. Gibson et al. Generalized Sampling and Variance in Counterfactual Regret Minimization. AAAI 2012.
[3] M. Schmid et al. Variance Reduction in Monte Carlo Counterfactual Regret Minimization (VR-MCCFR) for Extensive Form Games Using Baselines. AAAI 2019.
[4] T. Davis et al. Low-Variance and Zero-Variance Baselines for Extensive-Form Games. ICML 2020.
[5] B. Li et al. Correlated Chance Sampling for Monte Carlo Counterfactual Regret Minimization. arXiv 2026.
[6] S. Srinivasan et al. Actor-Critic Policy Optimization in Partially Observable Multiagent Environments. NeurIPS 2018.
[7] H. Fu et al. Actor-Critic Policy Optimization in a Large-Scale Imperfect-Information Game. ICLR 2022.
[8] M. Rudolph et al. Reevaluating Policy Gradient Methods for Imperfect-Information Games. ICLR 2026.
[9] Z. Fan, G. Farina. GAE Falls Short in Imperfect-Information Self-Play Reinforcement Learning. NeurIPS 2026.
[10] L. Peshkin, C. R. Shelton. Learning from Scarce Experience. ICML 2002.
[11] O. Tammelin. Solving Large Imperfect Information Games Using CFR+. arXiv 2014.
[12] S. Mohamed et al. Monte Carlo Gradient Estimation in Machine Learning. JMLR 2020.
Questions for authors
-
How does CP-GRPO relate to VRPO/Q-boosting [9], which reduces the policy-side action noise that C3 leaves uncoupled? Can VRPO (or ACH [7]) be added to the policy screen, or combined with CP-GRPO, under the same budget?
-
Which learning experiment is confirmatory: the "authoritative" three-seed tabular screen (favoring independent) or the five-seed policy screen (favoring CP-GRPO)? Please report paired seed-level CIs for both and explain the reversal.
-
Please reconcile the ledger: define the "direct contrast-summary field" in Table 2 and show how its means follow from the branch means; state which independent control (41.0876 / 41.1158 / 41.1087) is authoritative for the 2×2; and clarify whether the four arms of Tables 28/30 share the same 30,720 logical keys given that the independent row reproduces the 100,000-group moments exactly.
-
Does CP-GRPO change only the coupling assignment (App. C.1) or also enumeration, return freezing, and behavior weights (§3.7)? Can you add a coupled + sampled-action arm and a training run in which OSCC-Select chooses the coupling per checkpoint, so that the coupling-only effect and the selector's value are each isolated?
-
Table 8: which cited method actually optimizes "paired contrast variance" or "scalar return variance" as its objective? Would the TRPO vine, which shares the full noise sequence across rollouts, be rejected by C3 (cf. the "shared policy-side draw" row of Table 35)? Please also report δ, M, B, λ and how selections change for λ ∈ {0, 0.1, 1, 10}.