acceptodds
Withdrawn from ICLR 2027

RSD-Poker: Structure-Adaptive and Shift-Robust Risk–Utility Certification for Residual Policies in Imperfect-Information Games

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Imperfect-information reinforcement learning often requires adapting a strong base strategy to new opponents, environments, or deployment constraints, but directly optimizing residual policies can introduce unstable improvements and excessive risk under distribution shifts. We introduce RSD-Poker, a structure-adaptive and shift-robust certification framework for residual policy selection in imperfect-information games. RSD-Poker separates residual policy discovery from deployment-time certification by constructing adaptive evaluation structures, calibrating risk-utility tradeoffs, and selecting policies with statistical guarantees under diverse shift patterns. The framework supports stratified, partition-based, mixture-shift, learned-student, and opponent/solver evaluation endpoints through unified reporting and certification protocols. We evaluate RSD-Poker on imperfect-information poker environments under multiple distribution-shift settings. Compared with baseline residual adaptation strategies, RSD-Poker achieves improved utility while maintaining controlled exploitability risk. The Learned-Group Dual configuration obtains an average utility of 4.4936 with exploitability reduction of -0.0036, certification risk of 0.0078, and coverage of 0.7257. Under robust grouping, the method achieves competitive utility of 4.4827 while further reducing risk to 0.0059 and maintaining coverage of 0.7209. Results across stratified, partition, mixture-shift, learned-student, and opponent/solver endpoints demonstrate that adaptive certification enables reliable residual policy deployment beyond fixed-distribution evaluation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Imperfect-Information Games, Policy Adaptation, Residual Policies, Risk Certification, Distribution Shift, Safe Policy Optimization, Game-Theoretic Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tauyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the trade-offs between risk and utility in residual policies for imperfect-information games. The authors propose RSD-Poker, a structure-adaptive and shift-robust certification framework that combines policy-visible partition, risk–utility certification, and mixture-robust selection. Experiments on poker benchmarks show higher weak-response utility and lower violation rates compared with baseline methods.

Critical strengths
  1. The presentation of this paper is complete. The structure of this manuscript is well-organized.
  2. The topic of this work is interesting and useful for the game field, especially imperfect-information games.
  3. The experiments are thorough and complete.
Critical weaknesses
  1. The author's line of thought in Section 3 is a bit difficult to understand, but this section is very important in the paper. I lost myself many times when reading this section and trying to understand the proposed method. Although the authors provide Fig.1 for explanation, it would be better to give an explanation at the beginning of this section.
  2. I am confused about the relation between imperfect-information settings and the proposed method in this paper. It seems that the authors did not solve the problem caused by imperfect-information settings. I did not find any relevant words for imperfect information, such as "belief".
  3. Several variables lose their definitions, i.e., "pot(s)", "H\mathcal{H}" ,"Qg(s,a)Q_g(s,a)".
  4. I don't understand the relation between dbd_b and dαd_{\alpha}, vbv_b and vαv_{\alpha}. They use the same formulation. But I cannot find dαd_{\alpha} and vαv_{\alpha} after Eq.3 in the manuscript. What are the purpose of dαd_{\alpha} and vαv_{\alpha}?
Questions for authors

Please review my weaknesses.

Reviewer_ebVwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Based on my understanding of the paper, the paper is trying to calibrate a decision in current situation in a Poker game, by using different traces that were stored from the training. However, I am very unsure about this, as the paper does not contain sufficient explanation of the method.

Critical strengths

Figure 1 summarizes the entire algorithm, and I believe it may serve as the backbone of the description of their approach in future versions of the paper, but a sufficient description is currently missing.

Critical weaknesses

The paper feels more like a draft or report than a fully fledged paper. The paper lacks any motivation whatsoever. The whole background required to understand the concepts in the paper is missing. One example of this is line 105: "Neural fictitious self-play offers another route to learned mixed strategies," which does not specify what the "another route" is or how it connects to the authors' work.

It is unclear whether the method applies only to Poker-like games or to other domains, as the authors do not specify this in the paper (other than the algorithm's name). If the paper is specific to Poker, the authors should state this explicitly and explain why it cannot be extended to other domains, or describe the method in the general case, remove "Poker" from the algorithm name, and maybe use Poker only as a running example.

The paper contains many constants without any explanation. For example: Equation (1) and constants 0.92 and 0.08, or Equation (2) with -0.2, 0.02, and 0.55. Similarly, all other equations in the paper are only stated, without any motivation for why they are used, what they accomplish, or whether their choice is arbitrary or grounded in theory.

The experiments show some results, but they are not discussed, so it is unclear to me whether the improvements of some methods are marginal or substantial.

Questions for authors
  1. Could you please provide a summary of the whole approach, which may also be used in the paper to explain your approach?
  2. Could you please discuss the results? What do they show, what are the improvements over prior work, and what are the potential downsides of your approach?
Reviewer_dckHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper attempts to establish a comprehensive methodology encompassing candidate strategy generation, structure-adaptive partitioning, risk and utility certification, and robust deployment under distributional shifts. This research direction is valuable, particularly in its distinction between the average utility of policies and tail degradation risk, which aligns with the research needs of safe reinforcement learning and high-confidence policy improvement.

The proposed RSD-Poker is a substantial and carefully organized framework with meaningful empirical scope. However, its methodological novelty appears primarily incremental, combining established finite-sample risk certification, adaptive grouping, and distributionally robust selection. The paper would be considerably strengthened by a clearer differentiation from prior adaptive group calibration methods, a tighter connection between the partition objective and the certified risk, explicit verification of the utility-boundedness assumptions, and direct evidence of robust gains under deployment mixture shifts.

Critical strengths
  1. This paper has a clear practical motivation, can reasonably distinguish the average utility from the state-level policy degradation risk, and constructs an auditable policy certification process based on independent data partitioning.

  2. The experiment has a relatively comprehensive coverage, including calibration scale, risk budget, residual model, grouping division, distribution shift, multiple training seeds and game-level indicators. The supplementary reproduction protocols and experimental records reflect a relatively high implementation workload.

Critical weaknesses

【Writing】

  1. Insufficient to make a significant contribution: The main thread of the paper is not clear enough, and it is diluted by too many scattered technical modules. Overall, it gives the impression of presenting a complete systematic solution rather than a concentrated breakthrough around a single important scientific issue. In particular, teacher-free students and trace freezing take up quite a lot of space, but the connections between them and the main theoretical innovations are not discussed closely enough.

  2. Figure 1 does not accurately convey the main innovations of the paper, which correspond to "Structure-Adaptive" and "Shift-Robust" in the title. Specifically, how is partitioning learned from structural data? How are different residual candidates selected for different groups? These are precisely the key distinctions of this work compared to simple scalar residual calibration. In addition, the text in the figure is too small, resulting in poor clarity when viewed in a standard PDF reader.

  3. The abstract provides an excessive amount of experimental information, which instead weakens the expression of the core mechanism. It should focus more on the research question, the limitations of existing methods, and the uniqueness of the proposed solution based on the research question.

  4. Several experimental results in Section 4 lack sufficient causal explanations. For example, Table 1 shows that Learned-Group Dual reduces the violation rate while enhancing utility. However, the main text mainly repeats the values and fails to fully answer: Is this improvement due to a more reasonable group division or a more favorable allocation of candidate strategies? This is more meaningful than simply listing indicators.

  5. Some equations in the method section are not numbered.

  6. There are too many subsections and the content is relatively short, making the overall structure very fragmented and not conducive to reading.

【Novelty】

  1. The technological innovation of Residual candidate bank is relatively weak. The author constructs a finite candidate set by freezing multiple residual directions and scales. This design can avoid the dependence on a single residual scale, but the method itself still belongs to the construction and selection of the finite candidate strategy family. Meanwhile, The exponential tilt operation is not a new strategy update mechanism. Its essence is closely related to existing ideas such as KL regularization strategy update and exponential weight update. The paper does not propose new residual learning objectives or new strategy representation architectures.

  2. Dual risk-Utility certification is theoretically correct, but not theoretically novel enough. Section 3.4 respectively uses: Clopper-Pearson upper confidence bound to control the state-level violation probability; Normalization utility of confidence bound control under Hoeffding; Joint confidence allocation handles multiple candidate strategies and groups. From the perspective of probability theory, this is a natural and principle-reasonable design, but these tools themselves already have mature theories. Therefore, the existing proofs are more like the validation of the effectiveness of specific strategy selection frameworks rather than the results of new statistical learning theories.

  3. The structural division objective is defined in Section 3.3. This design reflects a trade-off. If it is too coarse, risk differences cannot be identified; if it is too fine, the confidence bound will be more conservative. To my knowledge, partition learning itself has related work [1], which has obvious similarities to the main partitioning idea of this article. The feature of this article lies in applying this division to the risk-utility selection of residual strategies and adding penalties for the support of calibration samples. However, the author needs to clearly answer: Compared with the existing adaptive group calibration, apart from changing the application scenarios and adding regularization terms, what new method mechanisms are introduced in this article exactly? At present, the Related Work section has not fully established this difference.

【Experiment】

  1. The key contributions lack a strong enough direct comparison: The author has compared manual grouping, random grouping, and learned partitions without authentication penalties. These ablations are necessary but not sufficient to establish the methodological advantages of certificate-aware partitioning. Especially considering the existing adaptive partition methods based on intra-group homogeneity, the authors should further compare competitive risk perception partitioning strategies. The current results do not fully demonstrate that this classification objective is superior to the existing and more direct risk perception structure learning methods, which will weaken the most core innovation contribution.

  2. The key experimental persuasiveness of Shift-Robust is insufficient (see Table 17, Table 20, Table 24) : The title of the paper highlights "Shif-robust", but the existing results do not fully demonstrate the advantages of Robust optimization under the variation of the actual group ratio. The most convincing experiments are still lacking: Let different strategies be tested under the same deployment mixed distribution that clearly belongs to Q_{\rho } , especially comparing the real risk-utility performance of non-robust and robust Selectors on the same offset, and observing the relationship between the worst-case test results and the certificates. Otherwise, what the author mainly proves is that robust optimization can change the risk-utility trade-off, rather than its significant advantage under the actual mixture shift. This is the key evidence that needs to be supplemented for title-level contributions.

  3. The main experiment lacked statistical significance: For Table 1, the violation rate of Learned-Group Dual is 0.0078, while that of Robust Group is 0.0059. Based on the 512 test states provided in the appendix, the difference on the main test set may correspond to only one additional violation sample, which does not directly prove that the true violation probability of Robust Group is significantly lower than that of Learned-Group Dual. It would be better if at least supplement the confidence interval of the paired difference and report robustness across multiple independent test samples or training seeds, so as to provide a more direct statistical test for the comparison between Learned-Group Dual and Robust Group.

References [1] Gil, Natalia Martinez, et al. "Identifying Homogeneous and Interpretable Groups for Conformal Prediction." UAI. 2024.

Questions for authors

see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.