acceptodds
Withdrawn from ICLR 2027

RIVER: Recursive Self-Improvement of LLM Poker Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agent skills, natural-language instructions written by human experts, are increasingly used to extend large language model (LLM) agents without retraining, and have recently been applied to strategic games such as poker. For a recent expert-written skill for heads-up no-limit Texas hold'em (PokerSkill), we find that the return varies considerably across base models, across reasoning efforts of a single model, and even across requests for more detailed analysis before acting that leave the encoded expertise unchanged. More reasoning thus does not guarantee better play, and the value of a skill depends on how each model executes it. Since we cannot know in advance how a model will apply fixed human expertise, this raises the question of whether a model can start from human expertise and iteratively derive a poker skill suited to itself. Poker makes such self-improvement difficult, since single-hand outcomes are highly variable with no verifiable reward for individual decisions, and the decision space is too large for hand-written advice to cover. We introduce (\riverexpansion), in which the LLM that plays poker recursively improves its own skill library. Each update reflects retrospectively on three complementary evidence streams, namely the human expertise encoded in the original skill, the model's execution of its current skill, and a comparative analysis of returns across skill versions. We further factorize the skill by decision factor and aggregate evidence from single decisions to similar states, condensing the reasons behind them into updates of the skill. Across four configurations spanning three model families, derives for every model a skill that reaches a positive mean return against the Slumbot poker bot and exceeds the expert-written skill by 383 to 526 milli-big-blinds per hand.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, poker agent, self-improvement

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_55LTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RIVER, a recursive self-improvement framework for LLM poker agents. The main motivation is that the effectiveness of a fixed expert-written poker skill can vary substantially across models and reasoning configurations. RIVER therefore allows a frozen LLM to iteratively refine its own natural-language skill library using human expertise from PokerSkill, analysis of the model's own skill execution, and observed returns across skill versions. The method further factorizes poker strategy into several decision factors and uses hierarchical induction to aggregate evidence across similar states. Experiments against Slumbot show improved observed returns over the original PokerSkill across four model configurations, together with ablation and qualitative analyses of the learned skills.

Critical strengths
  1. Clear motivation and intuitive problem formulation. The paper identifies an interesting practical issue: the same expert-written skill may be interpreted and executed differently by different LLMs, motivating model-specific skill adaptation.

  2. Well-structured and easy to follow. The paper is generally clearly written, and the method is presented in a logical progression. The figures, particularly the overview of the RIVER pipeline, are visually clear and helpful for understanding the framework. I won't feel messy while reading it.

  3. Interesting system design. Combining factorized skills, multiple reflection signals, hierarchical aggregation, and version selection provides a coherent framework for adapting natural-language agent skills without modifying model weights.

Critical weaknesses

W1. The main performance claim is difficult to support under the current high-variance evaluation protocol.

Each skill version is evaluated using only 5,000 raw HUNL hands, even though heads-up no-limit hold'em is known to have extremely high return variance. This sample size appears particularly small for estimating long-run poker strength without variance reduction. Prior HUNL evaluations have required substantially larger samples or specialized variance-reduction techniques to obtain reliable estimates. For example, Burch et al. (2018) discuss an 80,000-hand Claudico match with a margin exceeding 9 bb/100 that was still close to the boundary of statistical significance. As another order-of-magnitude reference, the raw DeepStack human evaluation reported a 95% confidence interval of approximately ±220\pm 220 mbb/game over 44,852 hands. Under the usual 1/N1/\sqrt{N} scaling, this would correspond to uncertainty on the order of ±660\pm 660 mbb/game, or roughly ±66\pm 66 bb/100, at 5,000 hands. The exact variance in the present setup may of course differ, but the paper does not report empirical standard errors or confidence intervals that would allow this uncertainty to be assessed.

This issue is further amplified by the version-selection procedure. Figure 3 shows substantial fluctuations in the raw returns of successive skill versions, rather than a stable improvement trajectory. The monotonic curve corresponds to the best observed version so far, not to the performance of successive versions themselves. Since RIVER evaluates ten skill versions on 5,000 hands each and selects the version with the largest observed return, the selected score is subject to an optimistic best-of-NN bias when poker returns are highly noisy. The same noisy rollout is then reused as the final result in Table 2. Therefore, the reported improvements of approximately 38 to 53 bb/100 over PokerSkill are difficult to interpret as estimates of the selected policies' true expected returns without a fresh post-selection evaluation.

I would strongly prefer the authors to freeze the selected S⋆S^\star and evaluate it on a substantially larger fresh set of hands, with confidence intervals or repeated independent evaluation batches. Multiple independent RIVER runs would also help separate genuine improvement of the skill-generation procedure from stochastic skill rewriting and poker variance. If inference cost is the main constraint, I would consider a narrower but statistically better-powered study to be more convincing, for example using fewer model configurations or less expensive models while allocating substantially more samples to independent evaluation. The central claim concerns whether RIVER reliably improves a model-specific skill, rather than whether the method works specifically with the most expensive frontier models, so statistical reliability seems more important here than maximizing model scale.

A more direct alternative would be to evaluate the learned policies in a realistic poker environment with independently generated hands, such as a licensed research platform or a controlled poker server that permits automated agents. This would provide a more realistic estimate of long-run performance while avoiding the restrictions that commercial real-money platforms typically impose on bot play.

W2. The evaluation does not clearly distinguish general poker improvement from adaptation to Slumbot.

The learned refinements in Section 4.4 are plausible, but poker decisions are highly opponent-dependent. For example, bluffing less may be profitable against an opponent that overcalls, while the opposite adjustment may be better against an opponent that overfolds. This concern is especially important because Slumbot is used both as a source of reference decisions and as the evaluation opponent. Therefore, the learned rules may reflect specialization to Slumbot rather than generally stronger poker strategy.

Testing against additional opponents with different tendencies, or using solver-based evaluation, would help clarify this distinction. GTO- or solver-based analysis is standard in poker, and given that the learned policies are expressed in relatively clear natural-language rules, it should be feasible to compare representative RIVER refinements against solver-preferred actions or expected values.

A further concern is that the paper does not evaluate the learned skills on existing poker benchmarks that could provide complementary evidence beyond the noisy Slumbot returns. PokerBench, for example, contains 11,000 pre-flop and post-flop poker scenarios and evaluates decision quality using solver-grounded targets. The benchmark was also validated by showing that models with higher PokerBench scores tend to achieve higher win rates in actual games. Evaluating RIVER before and after skill refinement on PokerBench would therefore provide a useful indication of whether the learned rules improve general poker decision quality rather than merely adapting to Slumbot.

A complementary option is the Heads-Up Poker benchmark in Kaggle Game Arena, which evaluates LLMs through direct HUNL play at a much larger scale. Its initial benchmark uses 900,000 total hands and duplicate poker, where the same deals are replayed with player positions reversed to substantially reduce the influence of card luck. Compared with the current evaluation of only 5,000 raw hands per version, these existing benchmarks illustrate that substantially stronger evaluation protocols are available. Using either solver-grounded decision benchmarks or large-scale cross-opponent gameplay would make it much easier to determine whether RIVER learns generally stronger poker skills or primarily specializes to Slumbot. With these two flaws, they majorly reflect the score I give.

W3. The ablation study is limited to a single model configuration.

The component ablations in Section 4.3 are conducted only on Gemini 3.7 Flash with low reasoning effort. While the results suggest that the expertise stream, hierarchical induction, and return-based selection are useful in this setting, it is unclear whether the same relative contributions hold across the other model families and reasoning configurations considered in the main experiments. A small number of additional ablations on another model would make the component analysis more convincing. If inference cost is a concern, I would consider ablations on only two representative models sufficient, as this would still provide substantially stronger evidence of cross-model robustness.

W4. The algorithm used too much space and is not so useful for the paper. Algorithm 1 largely repeats the workflow already explained in Figure 2 and the main text. I think it could be removed or substantially shortened. The saved space could instead be used for a compact table summarizing the three evidence streams, their inputs, and their roles in the update process, which would be more useful for understanding the method.

The flaws are obvious and is difficult to be fixed in rebuttal phase. I will consider the rebuttal result but I now lean forward a relatively clear rejection. References

[1] Burch et al., AIVAT: A New Variance Reduction Technique for Agent Evaluation in Imperfect Information Games, AAAI 2018.

[2] Zhuang et al., PokerBench: Training Large Language Models to Become Professional Poker Players, AAAI 2025.

[3] Kaggle Game Arena and Google DeepMind, Evaluating LLMs through Heads Up No Limit Texas Hold Em, 2026.

Questions for authors

I try to put all problems here, but please mainly answer the questions mentioned in the weakness section.

Q1. What is the empirical uncertainty of the reported poker returns?

Could the authors report the per-hand return variance, standard errors, or confidence intervals for the results in Tables 1 and 2? In particular, I would like to know whether the differences between PokerSkill and RIVER remain statistically distinguishable under the observed variance of this specific Slumbot setup, rather than relying on variance estimates from prior HUNL studies.

Q2. Was the selected best skill independently re-evaluated after version selection?

My understanding is that each version is evaluated on 5,000 hands, the version with the highest observed return is selected, and the same rollout return is reported as its final performance. Is this interpretation correct? If the selected S⋆S^\star was evaluated again on fresh hands that were not used for version selection, please report those results. If not, how do the authors account for the optimistic bias introduced by selecting the maximum among several noisy evaluations?

Q3. How stable is the complete RIVER procedure across independent runs?

Were multiple independent RIVER runs performed for any of the four model configurations, including independent skill rewrites and fresh poker hands? Figure 3 shows substantial variation across successive versions, which could arise from both poker variance and stochastic skill generation. A distribution over independently repeated RIVER runs would materially affect my confidence that the improvement is attributable to the procedure rather than to a favorable single trajectory.

Q4. How much of the improvement transfers beyond Slumbot?

Slumbot appears to serve both as the source of reference decisions and as the opponent used for optimization and final evaluation. Have the learned skills been evaluated against any additional opponents with substantially different tendencies? If not, what evidence supports interpreting the improvement as stronger poker play rather than specialization to Slumbot?

Q5. Can the learned strategy refinements in Section 4.4 be validated independently of game returns?

For the qualitative examples in Section 4.4, such as reduced stabbing, increased checking with medium-strength hands, and blocker-conditioned river bluffing, do the authors have solver-based EV comparisons, exploitability analysis, or results on a solver-grounded benchmark such as PokerBench? Such evidence would help determine whether these refinements are strategically stronger or simply plausible and effective against the particular opponent used in the experiments.

Q6. Do the ablation conclusions replicate on another model configuration?

The component ablations are reported only for Gemini 3.7 Flash with low reasoning effort. Were any of these ablations run on another model family or reasoning setting? Even one additional configuration would be useful for determining whether the relative importance of expertise, hierarchical induction, and return-based selection is a property of RIVER rather than of this particular model configuration.

Q7. What evidence supports the claim that the learned skills are model-specific rather than simply better prompts?

A central motivation is that different models execute the same expert-written skill differently and should therefore learn skills suited to themselves. Have the authors cross-evaluated a skill learned by one model using another model, for example by applying the Gemini-learned skill to DeepSeek or GPT-6 Luna? A cross-model transfer experiment would directly test whether the learned improvements are genuinely model-specific, which seems central to the paper's framing.

Reviewer_aYFYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies recursive self-improvement of LLM poker agents based on PokerSkill (the best LLM poker agent today).

Critical strengths

No

Critical weaknesses

Although I really like the baseline of this paper, PokerSkill, and believe in the great significance of PokerSkill in Poker LLM Agent, I think RIVER has indeed been done very poorly.

  1. The author's implementation of the PokerSkill paper is wrong. I am also very interested in the PokerSkill and try to improve it. I found that in the open-source version of PokerSkill, the skill library seems not to be publicly available, but only provides a battle interface and API call with GTOWizard, and does not provide a battle interface with Slumbot. In addition, the paper performs evolutionary operations on PokerSkill's skill library, but considering that the text of PokerSkill's skill library is not publicly available, unless the author is the author of PokerSkill or the source code is obtained from the author of PokerSkill, I believe that the implementation of the paper is clearly inconsistent with the description. Section 1 and Paragraph 2 of the paper mention 'We evaluate the released PokerSkill', but do not mention anywhere how the PokerSkill skill library was obtained. The paper also did not provide any verifiable code repository. The author needs to explain how they obtained the expert skill library in PokerSkill.

  2. All results are all wrong without any bar or variance reduction with only 5000 hands. The paper mentions in Section 2, Paragraph 1 that it follows the evaluation method of PokerSkill and uses 5000 hands. This is misleading. The reason why PokerSkill can be evaluated with only 5000 hands is because of its opponent, GTOWizard,AIVAT variance reduction technique was adopted to provide meaningful results within 5000 hands. The target of this paper is Slumbot, and Slumbot does not provide variance reduction techniques (this point is also explicitly mentioned in the appendix of the paper). Moreover, all the results in the paper have no error bars, and poker is a game with high randomness. If variance reduction techniques are not used, the result of 5000 hands is meaningless. I think you need to engage in battles with GTOWizard like PokerSkill, rather than reporting meaningless results with Slumbot. I think the paper needs to remove any inappropriate statements that say “our method exceeds PokerSkill”.

  3. The evaluation method in the paper are completely wrong. The experiment in the paper is completely wrong! The paper reports the so-called 'Best version so far' as the result of its method, RIVER, which is equivalent to taking the optimal value in 10 epochs of 5000 games. Do you know any previous published work uses such wrong evaluation methods? If you insist on using the evaluation method in the paper, train at least 20 runs, each run using 20 epochs, and each epoch using at least 100000 hands against Slumbot. And after fixing the optimal epoch, each run will conduct a test report on the results of 500000 hands against Slumbot.

  4. The method of the paper is completely wrong. In fact, I have tried methods similar to distillation PokerSkill and LLM, but the similar method as in the paper did not work at all in my attempts. The novelty is limited as well. The paper seems just a kind of PokerSkill+self-evo. Can you give me a reason why this paper is worth being accepted by ICLR?

  5. The motivation is weird and may wrong. There is no evidence to suggest a direct relationship between PokerSkill's performance and level of thinking. And as far as I know, if you play Poker directly with a big model, there is a positive correlation between performance and level of thinking. This conclusion is clearly contrary to the observations in the article. I suggest either deleting the motivation or conducting more convincing experiments to verify it.

  6. The writing of the paper is very poor and sometimes wrong. The paper does not even briefly introduce its baseline, the basic method of PokerSkill. The method diagram of the paper is simply incomprehensible to readers who are not familiar with Poker.

Overall, I believe that the motivation, methodology, and evaluation presented in this paper are completely flawed. My score is 0 (strong reject).

Questions for authors
  1. How did you obtain PokerSkill's expert skill library?

  2. Can you provide the complete code?

  3. Can you provide the results against GTOWizard Benchmark?

Reviewer_r624OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies self-improvement of LLM agent skills for heads-up no-limit Texas hold’em. Specifically, the authors argue that an expert-written skill (PokerSkill) yields different returns across models, reasoning efforts, and analysis formats, so a fixed skill is not suited to every model. To address this, they propose RIVER, where the playing model iteratively rewrites a skill library factorized by six decision factors, using three evidence streams (expert guidance anchored by reference decisions, audits of its own execution, and version-level return analysis), with hierarchical induction and version selection. Experiments on four configurations across three model families show that the best RIVER version reaches a positive return against Slumbot and exceeds PokerSkill by 383–526 mbb/hand.

Critical strengths
  1. The idea is straightfoward and easy to follow.
  2. The problem is interesting and underexplored.
  3. The experimental results seem good compared with baselines.
Critical weaknesses
  1. The motivation is unclear. I do not understand the goal of this paper. If the goal is to develop a stronger poker player. I think equilibrium solvers like CFR are already strong enough and consistently beat LLM-based players, and beating Slumbot does not show that LLM skills are competitive with them. If the goal is general self-improvement under noisy, non-verifiable rewards, the paper evaluates only one domain with a poker-specific design and does not compoare with other harness self-evolution methods (e.g., GEPA, ACE). In either case, the significance of the results is not well established.
  2. The results are not convincing. First, the PokerSkill baseline is weak in this setting: it loses to Slumbot in every configuration (Table 1), so outperforming it is not meaningful. Second, Table 2 reports the best of ten versions, each evaluated only once on 5,000 noisy hands and without confidence intervals, so the reported gains do not convincingly show that the method is effective.
  3. The paradigm is bounded by the evolver’s own capability. The same model both plays and rewrites the skill (Section 3.1, Appendix D), and the external information it receives is limited to the PokerSkill text, Slumbot’s actions, and noisy returns. If the evolver does not already have the knowledge to recognize a good strategy, self-improvement cannot discover it, so the ceiling of RIVER is set by the base model rather than by the method.
Questions for authors

Please see my weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.