acceptodds
Withdrawn from ICLR 2027

OSCC: Certified Observation-Safe Coupling Optimization for Gradient-Noise Control in Imperfect-Information Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Coupled rollouts can reduce the noise of counterfactual action comparisons, but standard common-random-number constructions are not directly applicable to imperfect-information learning due to information leakage, invalid stochastic synchronization, and mismatch between scalar variance reduction and policy-gradient optimization objectives. We introduce Observation-Safe Counterfactual Coupling (OSCC), a framework that defines valid couplings through marginal preservation, information-state safety, branch-local policy randomness, semantic event alignment, and trace-before-oracle replay. We derive a gradient-aware coupling criterion showing that, under marginal-preserving transformations, policy-gradient noise is governed by policy-Jacobian-weighted off-diagonal return covariance rather than return-contrast variance alone. Based on this analysis, we propose OSCC-Select, a calibration-only selector that evaluates candidate coupling strategies using certified safety constraints, projected gradient-noise reduction, and physical sampling cost, while reverting to independent sampling when improvement cannot be statistically guaranteed. We instantiate the framework with CP-GRPO, a fully coupled rollout design for imperfect-information reinforcement learning. On 100,000 fixed-root Leduc comparisons, CP-GRPO reduces return-contrast variance from 41.1158 to 18.1441 (55.87% reduction) while preserving branch marginals. Extensive analyses across optimization, transfer, safety, noise transfer, and cost surfaces demonstrate how observation-safe coupling improves estimator reliability without violating information constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Policy Optimization, Variance Reduction, Imperfect-Information Learning, Counterfactual Coupling, Common Random Numbers, Gradient Noise Control, Multi-Agent Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Q6PHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies common-random-number coupling of counterfactual rollouts for policy-gradient learning in two-player imperfect-information games. It defines five admissibility conditions, shows that under marginal preservation the projected gradient noise differs across couplings only through gradient-weighted off-diagonal return covariances, and proposes OSCC-Select, which picks among (independent, root-only, continuation-only, full) coupling using simultaneous Hoeffding lower bounds with an independent fallback. CP-GRPO instantiates full coupling.

Critical strengths

None

Critical weaknesses
  1. Related work missing.

(a) Training-time variance reduction in imperfect-information games is absent: MCCFR [1] and variance-reduced MCCFR estimators [2–5] are not cited; AIVAT appears only as an evaluation tool.

(b) Imperfect-information policy-gradient work is absent [6–8]. Most importantly, [9] identifies the same bottleneck—return-variance inflation of PPO in IIG self-play caused by stochastic equilibrium policies—and fixes it by averaging over future action draws with a Q-critic (Q-boosting/VRPO). OSCC attacks a complementary noise source (shared exogenous chance across branches) and, by C3, deliberately leaves policy-side action noise uncoupled; the two methods are natural to combine, yet [9] is neither cited nor compared.

(c) [10], a classical CRN-for-policy-search precedent, is missing from the paragraph that claims to list "the closest statistical precedents," and the CFR+ paper [11] is not cited although "exact CFR+" anchors Table 24.

  1. No external baseline for either headline. The 55.87% figure is compared only against OSCC-internal arms; there is no antithetic, stratified, or control-variate arm under the same fixed-root estimand and physical-call budget, so the reader cannot tell whether this is the ordinary CRN gain or something specific to the proposed contract. The policy screen compares CP-GRPO only to generic PPO/GRPO and an exact CFR+ anchor; no IIG-specific PG method (VRPO [9] or ACH [7]) is included. Note also that 55.87% is return-contrast variance; the realized update-variance ratio is 0.620 (Table 36), i.e. a 38% reduction, and the abstract reports only the former.

  2. The learning claim is not supported by the paper's own evidence. §4.1 calls the three-seed tabular screen "authoritative," and §4.2.2 reports that the independent arm is better on all three seeds. Table 20's five-seed PPO connectivity intervals "include zero." Further, OSCC-Select is never used in any training run, and all four 2×2 arms use complete-action enumeration, so there is no coupled + sampled-action arm. The title's "Imperfect-Information Learning" and the "general learning primitive" framing are therefore not tested.

  3. Novelty is overstated relative to the content, and prior art is misdescribed. Prop. 3 is the expansion of tr(GΣG^T) with coupling-invariant diagonals; Thm. 1 is a Bonferroni-corrected Hoeffding bound over four candidates. C2 is the definition of a behavioral strategy under perfect recall; C4's per-event persistent chance addressing overlaps concurrent work on correlated/persistent chance sampling [5]. Table 8's "optimization objective" column lists "paired contrast variance," "scalar return variance," and "trust-region update signal" for prior methods and "gradient noise plus physical cost" for OSCC alone; in those works CRN is a variance-reduction device for an expected-return or policy-gradient estimator, and the TRPO vine procedure explicitly uses common random numbers to reduce the variance of Q-value differences, i.e. it is already CRN-based advantage estimation. Yadav et al. is cited in one sentence although its intermediate-coupling estimator directly anticipates the OSCC-Select candidate set. The standard source for score-function gradient variance reduction [12] is also absent.

  4. Internal inconsistencies that undermine the "auditable ledger" positioning. Table 2's reported means (0.01421 / 0.03487) are not the check − bet contrast means implied by §4.2.1's branch means (0.17273 / 0.20437); they track the check-branch means, and the "separate aggregation units" remark does not resolve a 12× discrepancy. §4.3.2 quotes 41.0876 as the 2×2 independent control, but Tables 28/30 give 41.1158; 41.0876 is the Table 23 val./test value; Table 37 gives 41.1087. Appendix C.1 states that "CP-GRPO changes only the coupling assignment in the rollout collector," whereas §3.7 lists complete-action enumeration, frozen returns, and behavior weights as additional changes relative to standard GRPO. Both cannot be true. In Tables 28/30 the 30,720-key independent row reproduces the 100,000-group audit's covariance and variance digit-for-digit (−0.11495, 41.1158), while the CP-GRPO row's variance (18.3269) matches Table 23, not the 100k audit (18.1441); the independent arm's update variance is 0.1835 in Table 22 and 0.1398 in Table 30. Table 38 reports 500 solver queries (498 successful) against Table 21's 5 seeds × 512 held-out roots. "Exact CFR+ reference" has residual exploitability 0.0152 on Leduc (Table 12), 12.8% of CP-GRPO's 0.1184; it is a high-precision iterative solution, not an exact one.

  5. The paper writing is terrible, the experimental results are scattered everywhere, and humans can't understand what they're doing. Experiments are limited to Leduc and Kuhn.

[1] M. Lanctot et al. Monte Carlo Sampling for Regret Minimization in Extensive Games. NeurIPS 2009.

[2] R. Gibson et al. Generalized Sampling and Variance in Counterfactual Regret Minimization. AAAI 2012.

[3] M. Schmid et al. Variance Reduction in Monte Carlo Counterfactual Regret Minimization (VR-MCCFR) for Extensive Form Games Using Baselines. AAAI 2019.

[4] T. Davis et al. Low-Variance and Zero-Variance Baselines for Extensive-Form Games. ICML 2020.

[5] B. Li et al. Correlated Chance Sampling for Monte Carlo Counterfactual Regret Minimization. arXiv 2026.

[6] S. Srinivasan et al. Actor-Critic Policy Optimization in Partially Observable Multiagent Environments. NeurIPS 2018.

[7] H. Fu et al. Actor-Critic Policy Optimization in a Large-Scale Imperfect-Information Game. ICLR 2022.

[8] M. Rudolph et al. Reevaluating Policy Gradient Methods for Imperfect-Information Games. ICLR 2026.

[9] Z. Fan, G. Farina. GAE Falls Short in Imperfect-Information Self-Play Reinforcement Learning. NeurIPS 2026.

[10] L. Peshkin, C. R. Shelton. Learning from Scarce Experience. ICML 2002.

[11] O. Tammelin. Solving Large Imperfect Information Games Using CFR+. arXiv 2014.

[12] S. Mohamed et al. Monte Carlo Gradient Estimation in Machine Learning. JMLR 2020.

Questions for authors
  1. How does CP-GRPO relate to VRPO/Q-boosting [9], which reduces the policy-side action noise that C3 leaves uncoupled? Can VRPO (or ACH [7]) be added to the policy screen, or combined with CP-GRPO, under the same budget?

  2. Which learning experiment is confirmatory: the "authoritative" three-seed tabular screen (favoring independent) or the five-seed policy screen (favoring CP-GRPO)? Please report paired seed-level CIs for both and explain the reversal.

  3. Please reconcile the ledger: define the "direct contrast-summary field" in Table 2 and show how its means follow from the branch means; state which independent control (41.0876 / 41.1158 / 41.1087) is authoritative for the 2×2; and clarify whether the four arms of Tables 28/30 share the same 30,720 logical keys given that the independent row reproduces the 100,000-group moments exactly.

  4. Does CP-GRPO change only the coupling assignment (App. C.1) or also enumeration, return freezing, and behavior weights (§3.7)? Can you add a coupled + sampled-action arm and a training run in which OSCC-Select chooses the coupling per checkpoint, so that the coupling-only effect and the selector's value are each isolated?

  5. Table 8: which cited method actually optimizes "paired contrast variance" or "scalar return variance" as its objective? Would the TRPO vine, which shares the full noise sequence across rollouts, be rejected by C3 (cf. the "shared policy-side draw" row of Table 35)? Please also report δ, M, B, λ and how selections change for λ ∈ {0, 0.1, 1, 10}.

Reviewer_at2ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how sharing randomness across counterfactual rollouts affects action comparisons and policy updates in imperfect-information games. The authors introduce observation-safe counterfactual coupling (OSCC), which specifies conditions for preserving each branch’s distribution and restricting the information available to the policy. They then derive how return covariance affects policy-gradient noise and propose OSCC-Select. It uses calibration data to choose among independent sampling, root-only coupling, continuation-only coupling, and full coupling. The selector accounts for sampling cost and falls back to independent sampling when an improvement cannot be certified.

The paper evaluates this framework in Leduc and Kuhn poker, using CP-GRPO as its implementation. The experiments report reduced return-contrast variance and shows that selecting a coupling based on gradient noise can lead to a different choice than selecting based on return variance. The author also examine marginal preservation, information leakage, sampling cost, and downstream policy performance.

Critical strengths

S1. The distinction between return-contrast variance and policy-gradient noise is useful and supported by a concrete experiment. Section 3.5 shows how changes in cross-action return covariance are weighted by the policy gradients. And Section 4.3.4 then illustrates why this matters. It provides a specific reason to consider the update geometry when choosing a coupling.

S2. The experiments include controls that help isolate the effect of coupling. Section 4.3.2 separates root reuse from continuation coupling. The appendix further compares paired and independent rollouts using the same logical update groups and matched budgets and reports both update variance and downstream return. These comparisons provide more direct evidence for the sampling intervention than comparisons against standard PPO or GRPO alone.

Critical weaknesses

W1. My main concern is that the practical benefit of the gradient-aware selector is not sufficiently established. Section 4.3.4 shows that OSCC-Select achieves lower gradient noise than return-variance selection, but does not show whether this difference improves downstream return, exploitability, or learning efficiency. The positive policy results in Appendix D.3, Tables 22–25, primarily concern the fully coupled CP-GRPO implementation. Moreover, Section 4.2.2 reports that independent sampling achieves lower final diagnostic exploitability on all three seeds in a separate multi-iteration experiment. The paper acknowledges that these evaluations use different protocols, but does not adequately explain which differences account for the opposing outcomes.

W2. The contribution beyond established common-random-number methods is not sufficiently clear. Although section 3.3 provides a semantic-event coupling construction and Appendix D demonstrates the detection of deliberately introduced information-boundary violations, these results do not clearly establish what additional guarantees OSCC provides beyond a valid common-random-number implementation that preserves branch marginals and respects the policy’s information constraints.

Questions for authors

Q1. Does OSCC-Select lead to better policy training, beyond reducing the gradient noise ? Please compare its policy performance with fixed full coupling, return-variance-based selection, and independent sampling under the same total budget.

Q2. Section 4.2.2 reports lower final diagnostic exploitability with independent sampling for all three seeds, while Appendix D.3 reports better policy performance with CP-GRPO. Could the authors explain which differences in the training settings or evaluation procedures account for these results?

Reviewer_WrH9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper selects shared randomness across counterfactual rollouts to reduce noise in policy gradients in games with imperfect information. It proposes a coupling framework that preserves observation safety, with a certified selector that falls back to independent sampling, evaluated on Leduc and Kuhn.

Critical strengths
  1. Proposition 3 and the result with three actions in Section 4.3.4 make a useful case for selecting couplings according to the geometry of the policy gradients. Minimizing return variance alone can select the wrong coupling.

  2. The experiments separate root reuse from continuation coupling. I also appreciate that Section 4.2.2 reports a diagnostic across three seeds in which independent sampling achieves lower exploitability, rather than presenting variance reduction as an automatic learning improvement.

Critical weaknesses
  1. I do not see how the certificate in Section 3.6 is instantiated by the proposed algorithm. Eq. (2) defines an objective combining gradient noise and sampling cost, but Theorem 1 starts by assuming access to independent, unbiased, bounded gain scores. The paper needs to show how rollout observations produce those scores for that objective. This is central to calling OSCC-Select a certified method: the theorem establishes a conditional guarantee, but I cannot verify that the reported selector satisfies its conditions.

2, Figure 1 leaves me unsure what the implementation actually does. Its label for the policy input says observations are identical across branches at every decision, while the caption says each branch receives its own observation after histories diverge. This affects the meaning of observation safety, not just the appearance of the diagram. I also find the practical learning conclusion difficult to assess: Section 4.2.2 reports worse exploitability for paired sampling on all three diagnostic seeds, alongside gains in a separate policy experiment with five seeds. Both results matter, but I would need a clearer explanation of the conditions separating them to understand when the proposed coupling improves learning.

Questions for authors
  1. How are the gain scores in Theorem 1 constructed from sampled rollouts for Eq. (2), and what ensures that they are unbiased and bounded? This connection is necessary to assess the claimed certification.

  2. Does Figure 1 describe shared observations throughout a rollout or only a shared root information state? Please reconcile the diagram with its caption. If generative AI contributed to the figure, how was its technical content reviewed?

  3. How should the diagnostic with three seeds and the separate policy experiment with five seeds in Section 4.2.2 be reconciled? Which differences in the learning setup explain when coupling helps, and what does this imply for the practical scope of OSCC-Select’s guarantee?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.