acceptodds
Under review as a conference paper at ICLR 2027

TatRL: Closed-Loop World Model Evolution for Long-Horizon LLM Planning

Abstract

Large language models learn implicit world models through next-token prediction, yet leveraging this capacity for long-horizon planning remains challenging. Single-step methods, including supervised fine-tuning and turn-level optimization, cannot capture this structure, as they implicitly assume that the quality of each action is independent of the trajectory that generated it. This assumption holds for single-step tasks such as mathematical reasoning, but breaks down in long-horizon sequential tasks where actions are optimal only as a consequence of preceding moves. To address this, we introduce closed-loop world model evolution, a framework that treats entire occupancy trajectories as the unit of both generation and evaluation. Building on the connection between trajectory-level evaluation and multi-scale world modeling, we develop a progressive temporal scaling algorithm that trains within a self-play framework at increasingly long horizons, analogous to varying the planning horizon in geometric horizon models. We evaluate our approach on eight complex long-horizon tasks from the DouDizhu, GuanDan, and poker domains and find that trajectory-level evaluation is both necessary and sufficient for learning compositionally robust policies, achieving significant improvements across all benchmarks. Moreover, we achieve 99.2% action-sequence consistency while using 20 less demonstration data than single-step baselines, highlighting the sample efficiency of closed-loop world model evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
evolving systems, LLM agents, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HVZtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies LLM finetuning for long-horizon game tasks with imperfect information. The authors argue that existing LLM agent training methods fail to account for how an action's quality depend on earlier moves, and propose the TatRL-Evolution framework to address this. The framework consists of an RL algorithm trained on complete multi-turn self-play trajectories, and an LLM-driven tool that iteratively revises training scripts based on execution feedback. The authors evaluate their framework on eight card games against zero-shot LLMs, SFT finetuned models, and the specialized GuanDan agent DanZero. The script-revision tool is evaluated separately by training-job success rates.

Critical strengths

The paper evaluates its proposed framework with broad experiments that covers eight card games and four model backbones from three LLM families. The comparisons include zero-shot LLMs, SFT finetuned models and a specialized game agent, and the reported gains are large. The prompt templates and hyperparameters are documented, which supports reproducibility.

Critical weaknesses
  1. The manuscript looks incomplete. Section 2 has unresolved cross-references. Several appendix sections are empty (such as Appendix J, which Section 4.2 cites for detailed results).
  2. The theoretical framework is hard to follow and does not connect to the proposed method. For example, Eq. (1) is not well-defined; Definitions 3.1 and 3.4, Observation 3.2, and Assumption 3.3 are not used anywhere in the algorithm or experiments.
  3. The "closed-loop world model evolution" in the title and abstract is overclaimed. No component of the proposed method learns, uses or evaluates a model of environment dynamics.
  4. The proposed method has limited novelty. The core RL is a GRPO-style algorithm. The remaining components appear to be existing techniques: the dual-clip bound with c = 3 is from [1], the asymmetric clipping is from DAPO [2], the ScalingInter-RL is from [3], and the opponent sampling from a checkpoint pool is from [4].

[1] Ye, Deheng, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu et al. "Mastering complex control in moba games with deep reinforcement learning." In Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 04, pp. 6672-6679. 2020.

[2] Yu, Qiying, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai et al. "Dapo: An open-source llm reinforcement learning system at scale." Advances in Neural Information Processing Systems 38 (2026): 113222-113244.

[3] Xi, Zhiheng, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng et al. "Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning." arXiv preprint arXiv:2509.08755 (2025).

[4] Vinyals, Oriol, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi et al. "Grandmaster level in StarCraft II using multi-agent reinforcement learning." nature 575, no. 7782 (2019): 350-354.

Questions for authors
  1. How is the proposed framework compared with existing multi-turn RL methods, such as [5] which the paper discusses as prior work? Comparing the framework with recent work on long-horizon RL (such as [6] and [7]) would also be helpful.
  2. Can the authors clarify why GPT-4o's GuanDan win rate vs. rule-based is 1.9% in Table 1 but 3.2% in Table 2? And why the full-method DouDizhu result @128 is 91.2% in Table 3 but 93.1% in Table 10?
  3. Several ablated components (adaptive staleness, loss-weighted sampling, task advantage normalization, etc.) are not described in the method section. Could the authors explain them?
  4. Could the authors clarify the difference or relationship between your proposed ScalingInter-RL approach with the one proposed in [3]?

[3] Xi, Zhiheng, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng et al. "Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning." arXiv preprint arXiv:2509.08755 (2025).

[5] Zhang, Hanchen, Xiao Liu, Bowen Lv, Xueqiao Sun, Bohao Jing, Iat Long Iong, Zhenyu Hou et al. "AgentRL: Scaling agentic reinforcement learning with a multi-turn, multi-task framework." arXiv preprint arXiv:2510.04206 (2025).

[6] Hou, Zhenyu, Yujiang Li, Jie Tang, and Yuxiao Dong. "Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning." arXiv preprint arXiv:2607.07508 (2026).

[7] Shi, Chengshuai, Wenzhe Li, Xinran Liang, Yizhou Lu, Wenjia Yang, Ruirong Feng, Seth Karten et al. "Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning." arXiv preprint arXiv:2605.00347 (2026).

Reviewer_HT4bOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackles the problem of training LLMs to play long-horizon games. They introduce a training methodology that consists of GRPO with some minor modifications and self-play. They show that the approach allows a 14B model to perform better than SFT-only training and GPT-4o, Sonnet 4, and DeepSeek-R1.

Critical strengths

The final results show effective fine-tuning of the model.

Critical weaknesses

The paper's text lacks a coherent story and is mostly incomprehensible. This is, above all, visible by comparing the conclusion and the abstract. The conclusion appears to be describing a completely different paper than what the abstract describes. The focus and the story change a lot, there are empty sections in the appendix, and numerous vague terminologies.

In the following, I provide some specific notes.

The paper fails to rigorously define many parts of its arguments. As an example, many parts of eq. 1 are questionable. The advantage AkA_k is not defined, I would assume this is the standard one from GRPO, but that wouldn't match the subscript k. Also, ρk\rho_k is not defined, so I assume it is the importance sampling ratio. A derivative operation seems to be missing from the third term. Lastly, I don't understand the inequality symbol between the vectors here.

In the continuation, lines 159-163 are very vague. I don't understand how ties can be problematic (is it the normaliztion?). The decomposablity assumption is also not defined well: The paper doesn't define "the advantage of a full occupancy trajectory", so I cannot judge how reasonable the assumption is. The operations "removing actions" and "recombining the remaining steps into a refined policy" also are not clear.

Reward r_t in line 174 is not defined.

I understand defining "derivable" formally could have been difficult, but an intuitive one is not given either.

Assumption 3.3 comes before defining modules, and its last sentence is meaningless. Even by the end of section, the definition of module is not clear.

The curriculum inspired by geometric horizon models that was mentioned in the abstract and introduction is never introduced.

Unclear contribution. From Section 3.1 seems like the whole method is found through an automated research loop. This by itself could be the topic of the paper. However, the paper mostly focuses on the training methodology that is used.

From what I understand, the RL method is introducing a different clipping threshold (c) for negative advantage tokens compared to positive ones (ϵ\epsilon). Also, the groups where all returns are equal are not normalized. The role of turn-level loss is not mentioned.

Lack of RL-finetuned baselines. The experiments only compare the tuned model with zero-shot performance of other models. I couldn't find how, for example, using vanilla GRPO compares with their modified RL training.

Minor:

  • Multiple missing links like Figure ?? or Section ??
Questions for authors

I don't understand the author's claim that single-step methods evaluate actions in isolation and not in the trajectory's context they are in (e.g. mentioned in lines 131-134). Both SFT and RL losses the authors mention in the related work put the loss on the token's probability the model selects conditioned on all the past tokens. This conditioning exactly allows evaluating the actions in the context they have appeared.

Reviewer_a8c5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TatRL-Evolution, a framework for training LLM agents on long-horizon decision-making tasks. It combines supervised fine-tuning with reinforcement learning through self-play. The main approach, ScalingInter-RL, gradually increases the interaction horizon during training to help agents learn from longer trajectories. The framework uses terminal game outcomes to compute trajectory-level advantages, discounted returns for turn-level credit assignment, and clipped policy optimization to update the model. It also incorporates cross-policy sampling, allowing agents to train against opponents with different skill levels, and an automated system for detecting and fixing training failures. The authors evaluate the framework on eight card games using several LLM backbones, including Qwen, Llama, and GLM. The results show improvements over zero-shot LLMs, SFT-only baselines, and specialized game agents. The paper also reports ablation studies on the training components, improved action format compliance, and reduced supervised data requirements through self-play reinforcement learning.

Critical strengths
  1. The paper evaluates eight complex card games across different model architectures and reports substantial improvements over zero-shot LLMs, SFT-only models, and specialized game AI systems.

  2. The method addresses several practical challenges in long-horizon agent RL by combining progressive curriculum learning, cross-policy sampling, trajectory-level rewards, and policy optimization.

Critical weaknesses
  1. The proposed framework mainly combines existing RL techniques. Progressive Temporal Scaling appears to be the main contribution, but it is essentially a form of curriculum learning rather than a fundamentally new method.

  2. I am not convinced by the paper's claims about world model evolution. Although the authors emphasize implicit world models in LLMs, the method relies on game simulators for state transitions and uses terminal rewards to optimize the LLM policy. There is no explicit world model training objective or evaluation of state transition prediction. The paper would be more accurately framed as long-horizon policy optimization.

  3. The proposed credit assignment method mainly uses standard discounted terminal rewards. It does not directly estimate how much each action contributes to the final outcome.

  4. The caption of Table 1 states that GuanDan results are win rates against rule-based opponents. However, the API model results exactly match the vs. DanZero results in Table 2, rather than the vs. Rule results. This inconsistency needs clarification.

  5. Several references in the paper still appear as Figure ?? or Section ??.

Questions for authors
  1. Table 3 shows that removing ScalingInter-RL reduces performance. However, the ablation setting is not sufficiently clear: What horizon is used when ScalingInter-RL is removed? Does the baseline train on full game trajectories from the beginning? Do both settings use the same number of environment interactions and comparable training compute?

  2. The method uses terminal rewards. If a trajectory reaches the maximum horizon before the game ends, how is its reward calculated?

  3. How is the final advantage used for policy optimization calculated? Does the method use Grouped Outcome Advantage (GOA), Turn-Level Advantage, or a combination of both?

  4. Theoretical justification. Equation (1) needs further clarification. Is it intended to establish a policy gradient estimate, a performance bound, or a policy improvement guarantee? The authors should revisit the formulation and provide a complete derivation. If the claimed guarantee cannot be established, it should be removed or revised accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.