- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies LLM finetuning for long-horizon game tasks with imperfect information. The authors argue that existing LLM agent training methods fail to account for how an action's quality depend on earlier moves, and propose the TatRL-Evolution framework to address this. The framework consists of an RL algorithm trained on complete multi-turn self-play trajectories, and an LLM-driven tool that iteratively revises training scripts based on execution feedback. The authors evaluate their framework on eight card games against zero-shot LLMs, SFT finetuned models, and the specialized GuanDan agent DanZero. The script-revision tool is evaluated separately by training-job success rates.
Critical strengths
The paper evaluates its proposed framework with broad experiments that covers eight card games and four model backbones from three LLM families. The comparisons include zero-shot LLMs, SFT finetuned models and a specialized game agent, and the reported gains are large. The prompt templates and hyperparameters are documented, which supports reproducibility.
Critical weaknesses
- The manuscript looks incomplete. Section 2 has unresolved cross-references. Several appendix sections are empty (such as Appendix J, which Section 4.2 cites for detailed results).
- The theoretical framework is hard to follow and does not connect to the proposed method. For example, Eq. (1) is not well-defined; Definitions 3.1 and 3.4, Observation 3.2, and Assumption 3.3 are not used anywhere in the algorithm or experiments.
- The "closed-loop world model evolution" in the title and abstract is overclaimed. No component of the proposed method learns, uses or evaluates a model of environment dynamics.
- The proposed method has limited novelty. The core RL is a GRPO-style algorithm. The remaining components appear to be existing techniques: the dual-clip bound with c = 3 is from [1], the asymmetric clipping is from DAPO [2], the ScalingInter-RL is from [3], and the opponent sampling from a checkpoint pool is from [4].
[1] Ye, Deheng, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu et al. "Mastering complex control in moba games with deep reinforcement learning." In Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 04, pp. 6672-6679. 2020.
[2] Yu, Qiying, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai et al. "Dapo: An open-source llm reinforcement learning system at scale." Advances in Neural Information Processing Systems 38 (2026): 113222-113244.
[3] Xi, Zhiheng, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng et al. "Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning." arXiv preprint arXiv:2509.08755 (2025).
[4] Vinyals, Oriol, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi et al. "Grandmaster level in StarCraft II using multi-agent reinforcement learning." nature 575, no. 7782 (2019): 350-354.
Questions for authors
- How is the proposed framework compared with existing multi-turn RL methods, such as [5] which the paper discusses as prior work? Comparing the framework with recent work on long-horizon RL (such as [6] and [7]) would also be helpful.
- Can the authors clarify why GPT-4o's GuanDan win rate vs. rule-based is 1.9% in Table 1 but 3.2% in Table 2? And why the full-method DouDizhu result @128 is 91.2% in Table 3 but 93.1% in Table 10?
- Several ablated components (adaptive staleness, loss-weighted sampling, task advantage normalization, etc.) are not described in the method section. Could the authors explain them?
- Could the authors clarify the difference or relationship between your proposed ScalingInter-RL approach with the one proposed in [3]?
[3] Xi, Zhiheng, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng et al. "Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning." arXiv preprint arXiv:2509.08755 (2025).
[5] Zhang, Hanchen, Xiao Liu, Bowen Lv, Xueqiao Sun, Bohao Jing, Iat Long Iong, Zhenyu Hou et al. "AgentRL: Scaling agentic reinforcement learning with a multi-turn, multi-task framework." arXiv preprint arXiv:2510.04206 (2025).
[6] Hou, Zhenyu, Yujiang Li, Jie Tang, and Yuxiao Dong. "Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning." arXiv preprint arXiv:2607.07508 (2026).
[7] Shi, Chengshuai, Wenzhe Li, Xinran Liang, Yizhou Lu, Wenjia Yang, Ruirong Feng, Seth Karten et al. "Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning." arXiv preprint arXiv:2605.00347 (2026).