acceptodds
Withdrawn from ICLR 2027

Long-Horizon Textual World Modeling through Structured Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models must predict how an environment evolves under sequences of actions, enabling agents to compare possible futures and reason about counterfactual actions before acting. Long-horizon prediction is commonly obtained by recursively applying a one-step transition model, but intermediate errors can compound over time. Multi-step dynamics models instead condition on a sequence of future actions and predict their consequences directly, but become harder to learn as horizon grows: the model must track interacting state changes across the trajectory, endpoint supervision provides weak credit assignment, and intermediate predictions can remain plausible while losing information needed for later states. We show that these challenges can be addressed by casting the internal evolution of a multi-step transition as structured reasoning over textual world states: reasoning over sparse state changes reduces the burden of state tracking, a predictive-gain objective rewards the learned state for improving over a matched predictor that conditions on raw history instead, and intermediate predictive rewards supervise each state along the trajectory. Because these intermediate states are explicit textual representations of the world, they provide semantically meaningful targets that can be inspected, scored, and corrected during training. Across ScienceWorld, Jericho, and CEO-Bench, our approach achieves the strongest average long-horizon performance against recursive and non-recursive baselines that condition directly on raw history, with gains increasing at longer horizons: at horizon 10, it exceeds the strongest such baseline by 12.3% relative on ScienceWorld and 9.3% relative on CEO-Bench, and error on Jericho drops by nearly 30%. In a controlled counterfactual study, our model is also the only one with statistically significant sensitivity to future actions (p<0.001), while baselines show null or anti-correlated sensitivity once ground-truth geometry is accounted for. These results suggest that making a multi-step transition's internal trajectory explicit and supervisable through structured reasoning is a better abstraction for long-horizon textual world modeling than recursively composing one-step predictions.

Primary area:
learning on time series and dynamical systems
Keywords:
world models, reasoning, language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aV9XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a method to do long-horizon world modeling. The problem requires the model to predict the final state given a sequence of actions. The key idea is to predict "the change of state" at each step instead of the full intermediate state. The experiments show that this method performs better than several baselines on multiple environments.

Critical strengths
  1. The method is clear and simple.
  2. The results are on multiple benchmarks and there are a lot of baselines.
Critical weaknesses
  1. The comparison with the baselines are not fair. The method has access to data that is over-generated (more than one data point is generated from one original data point) by GPT5.2, filtered by quality, and it uses SFT+GRPO. The baselines seem to only use unfiltered data and do not use GRPO.
Questions for authors
  1. Can you provide result of the best baseline (latent-direct) with the same training paradigm of your methods?
Reviewer_BXe9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a framework in which a language model generates incremental updates to world states in action order within a single generation pass. These updates are then replayed programmatically to reconstruct the corresponding states, while rewards are used to improve the accuracy of both intermediate and final predictions.

Critical strengths
  1. The intermediate states are inspectable, making them easier to supervise than free-form textual reasoning.
  2. The experimental evaluation is relatively comprehensive.
Critical weaknesses
  1. The writing is not sufficiently reader-friendly.
  2. The dependence on errors arising during the generation process still needs to be adequately addressed.
  3. The novelty of the proposed method needs to be more clearly justified.
Questions for authors

1. The writing quality of the paper needs improvement. The paper introduces numerous concepts and terminologies, while the connection between the unified framework and its concrete implementation is not sufficiently intuitive. I had to spend considerable effort understanding the proposed method.

2. Why is the state representation assumed to be sparse? What happens if the state changes are dense? In fact, certain actions may lead to extensive, system-wide environmental changes, potentially even triggering butterfly effects. Would the proposed method still be applicable in such scenarios?

3. The proposed method does not fundamentally eliminate error dependencies during generation. It still relies on autoregressive generation, meaning that later state updates remain dependent on previously generated updates. This issue needs to be reasonably explained and discussed.

4. The authors aim to demonstrate that "non-recursive structured trajectories are better suited for long-horizon world modeling than recursive prediction." However, the actual comparison may only establish that "sparse updates combined with programmatic state preservation outperform repeatedly regenerating the entire state." These are not necessarily equivalent claims, and the experimental evidence does not seem sufficient to support the broader conclusion.

5. The overall method appears to be largely interpretable as a combination of teacher distillation and reinforcement learning, both of which are well-established techniques. The main difference seems to lie in how information is injected and how the generation process is organized. From this perspective, the authors need to provide a stronger justification for the methodological novelty of their approach.

Reviewer_b8NgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this work, the authors propose a method for world modelling over long horizons. They train a model to create a new structured text state from the trajectory history, predict changes from actions/edits applied to the state. They first train on filtered teacher trajectories from a strong model (GPT-5.2), then use RL to reward accurate intermediate and final observation predictions in the trajectory. The results on ScienceWorld, Jericho, CEO-Bench show some improvements in long-horizon state prediction.

Critical strengths

None

Critical weaknesses
  • It is unclear how much of the improvements in state prediction come from simply rewriting the previous full observation as a text state vs additional world model training. Multiple ablations exist in the paper, but it is hard to understand what hypothesis each ablation tests.
  • The reward function in sec 5.2 is hard to follow. What does each term contribute?
  • Would replacing a baseline world model with latent-SR improve task success or environment return under the same planner and inference budget? For ex. keep the planner fixed, change the world model, and measure task success or environment return.
Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.