- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RIVER, a recursive self-improvement framework for LLM poker agents. The main motivation is that the effectiveness of a fixed expert-written poker skill can vary substantially across models and reasoning configurations. RIVER therefore allows a frozen LLM to iteratively refine its own natural-language skill library using human expertise from PokerSkill, analysis of the model's own skill execution, and observed returns across skill versions. The method further factorizes poker strategy into several decision factors and uses hierarchical induction to aggregate evidence across similar states. Experiments against Slumbot show improved observed returns over the original PokerSkill across four model configurations, together with ablation and qualitative analyses of the learned skills.
Critical strengths
-
Clear motivation and intuitive problem formulation. The paper identifies an interesting practical issue: the same expert-written skill may be interpreted and executed differently by different LLMs, motivating model-specific skill adaptation.
-
Well-structured and easy to follow. The paper is generally clearly written, and the method is presented in a logical progression. The figures, particularly the overview of the RIVER pipeline, are visually clear and helpful for understanding the framework. I won't feel messy while reading it.
-
Interesting system design. Combining factorized skills, multiple reflection signals, hierarchical aggregation, and version selection provides a coherent framework for adapting natural-language agent skills without modifying model weights.
Critical weaknesses
W1. The main performance claim is difficult to support under the current high-variance evaluation protocol.
Each skill version is evaluated using only 5,000 raw HUNL hands, even though heads-up no-limit hold'em is known to have extremely high return variance. This sample size appears particularly small for estimating long-run poker strength without variance reduction. Prior HUNL evaluations have required substantially larger samples or specialized variance-reduction techniques to obtain reliable estimates. For example, Burch et al. (2018) discuss an 80,000-hand Claudico match with a margin exceeding 9 bb/100 that was still close to the boundary of statistical significance. As another order-of-magnitude reference, the raw DeepStack human evaluation reported a 95% confidence interval of approximately mbb/game over 44,852 hands. Under the usual scaling, this would correspond to uncertainty on the order of mbb/game, or roughly bb/100, at 5,000 hands. The exact variance in the present setup may of course differ, but the paper does not report empirical standard errors or confidence intervals that would allow this uncertainty to be assessed.
This issue is further amplified by the version-selection procedure. Figure 3 shows substantial fluctuations in the raw returns of successive skill versions, rather than a stable improvement trajectory. The monotonic curve corresponds to the best observed version so far, not to the performance of successive versions themselves. Since RIVER evaluates ten skill versions on 5,000 hands each and selects the version with the largest observed return, the selected score is subject to an optimistic best-of- bias when poker returns are highly noisy. The same noisy rollout is then reused as the final result in Table 2. Therefore, the reported improvements of approximately 38 to 53 bb/100 over PokerSkill are difficult to interpret as estimates of the selected policies' true expected returns without a fresh post-selection evaluation.
I would strongly prefer the authors to freeze the selected and evaluate it on a substantially larger fresh set of hands, with confidence intervals or repeated independent evaluation batches. Multiple independent RIVER runs would also help separate genuine improvement of the skill-generation procedure from stochastic skill rewriting and poker variance. If inference cost is the main constraint, I would consider a narrower but statistically better-powered study to be more convincing, for example using fewer model configurations or less expensive models while allocating substantially more samples to independent evaluation. The central claim concerns whether RIVER reliably improves a model-specific skill, rather than whether the method works specifically with the most expensive frontier models, so statistical reliability seems more important here than maximizing model scale.
A more direct alternative would be to evaluate the learned policies in a realistic poker environment with independently generated hands, such as a licensed research platform or a controlled poker server that permits automated agents. This would provide a more realistic estimate of long-run performance while avoiding the restrictions that commercial real-money platforms typically impose on bot play.
W2. The evaluation does not clearly distinguish general poker improvement from adaptation to Slumbot.
The learned refinements in Section 4.4 are plausible, but poker decisions are highly opponent-dependent. For example, bluffing less may be profitable against an opponent that overcalls, while the opposite adjustment may be better against an opponent that overfolds. This concern is especially important because Slumbot is used both as a source of reference decisions and as the evaluation opponent. Therefore, the learned rules may reflect specialization to Slumbot rather than generally stronger poker strategy.
Testing against additional opponents with different tendencies, or using solver-based evaluation, would help clarify this distinction. GTO- or solver-based analysis is standard in poker, and given that the learned policies are expressed in relatively clear natural-language rules, it should be feasible to compare representative RIVER refinements against solver-preferred actions or expected values.
A further concern is that the paper does not evaluate the learned skills on existing poker benchmarks that could provide complementary evidence beyond the noisy Slumbot returns. PokerBench, for example, contains 11,000 pre-flop and post-flop poker scenarios and evaluates decision quality using solver-grounded targets. The benchmark was also validated by showing that models with higher PokerBench scores tend to achieve higher win rates in actual games. Evaluating RIVER before and after skill refinement on PokerBench would therefore provide a useful indication of whether the learned rules improve general poker decision quality rather than merely adapting to Slumbot.
A complementary option is the Heads-Up Poker benchmark in Kaggle Game Arena, which evaluates LLMs through direct HUNL play at a much larger scale. Its initial benchmark uses 900,000 total hands and duplicate poker, where the same deals are replayed with player positions reversed to substantially reduce the influence of card luck. Compared with the current evaluation of only 5,000 raw hands per version, these existing benchmarks illustrate that substantially stronger evaluation protocols are available. Using either solver-grounded decision benchmarks or large-scale cross-opponent gameplay would make it much easier to determine whether RIVER learns generally stronger poker skills or primarily specializes to Slumbot. With these two flaws, they majorly reflect the score I give.
W3. The ablation study is limited to a single model configuration.
The component ablations in Section 4.3 are conducted only on Gemini 3.7 Flash with low reasoning effort. While the results suggest that the expertise stream, hierarchical induction, and return-based selection are useful in this setting, it is unclear whether the same relative contributions hold across the other model families and reasoning configurations considered in the main experiments. A small number of additional ablations on another model would make the component analysis more convincing. If inference cost is a concern, I would consider ablations on only two representative models sufficient, as this would still provide substantially stronger evidence of cross-model robustness.
W4. The algorithm used too much space and is not so useful for the paper. Algorithm 1 largely repeats the workflow already explained in Figure 2 and the main text. I think it could be removed or substantially shortened. The saved space could instead be used for a compact table summarizing the three evidence streams, their inputs, and their roles in the update process, which would be more useful for understanding the method.
The flaws are obvious and is difficult to be fixed in rebuttal phase. I will consider the rebuttal result but I now lean forward a relatively clear rejection. References
[1] Burch et al., AIVAT: A New Variance Reduction Technique for Agent Evaluation in Imperfect Information Games, AAAI 2018.
[2] Zhuang et al., PokerBench: Training Large Language Models to Become Professional Poker Players, AAAI 2025.
[3] Kaggle Game Arena and Google DeepMind, Evaluating LLMs through Heads Up No Limit Texas Hold Em, 2026.
Questions for authors
I try to put all problems here, but please mainly answer the questions mentioned in the weakness section.
Q1. What is the empirical uncertainty of the reported poker returns?
Could the authors report the per-hand return variance, standard errors, or confidence intervals for the results in Tables 1 and 2? In particular, I would like to know whether the differences between PokerSkill and RIVER remain statistically distinguishable under the observed variance of this specific Slumbot setup, rather than relying on variance estimates from prior HUNL studies.
Q2. Was the selected best skill independently re-evaluated after version selection?
My understanding is that each version is evaluated on 5,000 hands, the version with the highest observed return is selected, and the same rollout return is reported as its final performance. Is this interpretation correct? If the selected was evaluated again on fresh hands that were not used for version selection, please report those results. If not, how do the authors account for the optimistic bias introduced by selecting the maximum among several noisy evaluations?
Q3. How stable is the complete RIVER procedure across independent runs?
Were multiple independent RIVER runs performed for any of the four model configurations, including independent skill rewrites and fresh poker hands? Figure 3 shows substantial variation across successive versions, which could arise from both poker variance and stochastic skill generation. A distribution over independently repeated RIVER runs would materially affect my confidence that the improvement is attributable to the procedure rather than to a favorable single trajectory.
Q4. How much of the improvement transfers beyond Slumbot?
Slumbot appears to serve both as the source of reference decisions and as the opponent used for optimization and final evaluation. Have the learned skills been evaluated against any additional opponents with substantially different tendencies? If not, what evidence supports interpreting the improvement as stronger poker play rather than specialization to Slumbot?
Q5. Can the learned strategy refinements in Section 4.4 be validated independently of game returns?
For the qualitative examples in Section 4.4, such as reduced stabbing, increased checking with medium-strength hands, and blocker-conditioned river bluffing, do the authors have solver-based EV comparisons, exploitability analysis, or results on a solver-grounded benchmark such as PokerBench? Such evidence would help determine whether these refinements are strategically stronger or simply plausible and effective against the particular opponent used in the experiments.
Q6. Do the ablation conclusions replicate on another model configuration?
The component ablations are reported only for Gemini 3.7 Flash with low reasoning effort. Were any of these ablations run on another model family or reasoning setting? Even one additional configuration would be useful for determining whether the relative importance of expertise, hierarchical induction, and return-based selection is a property of RIVER rather than of this particular model configuration.
Q7. What evidence supports the claim that the learned skills are model-specific rather than simply better prompts?
A central motivation is that different models execute the same expert-written skill differently and should therefore learn skills suited to themselves. Have the authors cross-evaluated a skill learned by one model using another model, for example by applying the Gemini-learned skill to DeepSeek or GPT-6 Luna? A cross-model transfer experiment would directly test whether the learned improvements are genuinely model-specific, which seems central to the paper's framing.