acceptodds
Withdrawn from ICLR 2027

Does Learning to Predict the World Help Agents Act? Auditing World-Model Post-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting how an environment will change before acting is a natural route to better decision making for agents. Recent post-training methods therefore require agents to predict the next observation and turn that prediction into a reward or a direct supervision signal, which is called world model. Existing next-observation training methods help the agent to learn the environmental content. However, they additionally involve an optimization process, which may introduce several effects other than learning to predict the world. Consequently, **where the performance gain comes from during the training process remains an open question**. We answer this research question through replacing true next-observation targets with in-distribution mismatched observations during the training process. Across two interactive text environments, mismatched targets lower prediction accuracy by 15.3–61.6% relative to ground-truth targets, yet retain substantial task gains over the base model. Compared with the base model, trained models consider more candidate actions and exhibit less looping. We also introduce a setting that replaces prediction-based rewards with independent random signals. This training expands task coverage (pass@64) even when the reward carries no environment information. We also generalize this finding to VisualWebArena, where random-reward training raises pass@64 by 14.3% relative to the base model, without observation-matching rewards or an external multimodal teacher for reward construction.

Primary area:
reinforcement learning
Keywords:
agents, world models, reinforcement learning, agent post-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fgF2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether task improvements from world-model post-training depend on correct next-observation targets. It compares ground-truth targets, mismatched observations, and independent random rewards under GRPO on ALFWorld and ScienceWorld, with an additional self-distillation comparison on ALFWorld. Evaluation measures prediction accuracy, single-attempt success, and task coverage over 64 attempts. Mismatched targets substantially reduce prediction accuracy while retaining substantial task gains, and random-reward training also improves coverage. Behavioral analyses identify increased consideration of alternative actions and reduced looping after training. An extension to VisualWebArena demonstrates coverage improvements from random-reward training in a visual browsing environment.

Critical strengths

A well-designed audit of an important attribution problem.

The mismatched-target and random-reward controls directly test whether correct prediction supervision is necessary for task gains, while separate prediction evaluation checks that target corruption actually changes predictive performance. This provides a useful methodology for assessing what world-model post-training contributes to agent performance.

A consequential empirical finding supported across multiple settings.

The separation between prediction accuracy and task performance appears in both text environments and in the self-distillation experiment. The random-reward coverage improvement also extends to VisualWebArena. These results provide concrete evidence that task gains alone are insufficient to establish improved environment prediction as their source, an important consideration for interpreting and designing future agent-training methods.

Critical weaknesses

The broader conclusions about world-model learning are not sufficiently supported

The experiments show that substantial task gains can be retained without correct prediction targets, but it remains unclear whether this finding extends beyond the chosen training methods. Further comparisons across different ways of learning and using environment dynamics are needed to support broader conclusions about the contribution of prediction learning to agent performance.

Questions for authors

1. Does the same finding hold for directly supervised prediction training?

Why does the study not include a directly supervised next-state prediction baseline, such as Implicit World Modeling (IWM) [1]? With GT and MIS prediction targets followed by identical action fine-tuning, would MIS still retain most of the gains achieved by GT?

2. What motivates the implementation choices and training duration?

The original RWML collects ALFWorld trajectories using the model being trained, whereas this paper uses DeepSeek-V4-Flash. What motivates this change, and are the original easy-example subsampling and reward-text preprocessing procedures retained? The public RWML script uses entropy and KL coefficients of 0 and 0.001, respectively, whereas Table 6 uses 0.001 and 0.01. What motivates these adjustments?

COMAP combines ground-truth next-state supervision with distillation from an EMA-updated teacher, whereas this paper uses a fixed teacher and only the distillation loss. Why was this implementation chosen?

All conditions are trained for one epoch and evaluated at the final checkpoint. Why was this training duration selected? Do the relative performances of GT, MIS, and COIN persist when the training budget is extended equally across conditions?

3. How many independent training runs underlie Table 2?

The confidence intervals for the pass@k curves are obtained by resampling test tasks and do not capture training variability. How large are the GT–MIS differences relative to variation across training runs? Do the gains from COIN recur across different training seeds?

4. Could annotation errors affect the action-revision findings in Table 3?

Appendix C.2 reports that an independent review confirmed 40 of 60 steps labeled as action revision. Are the false positives concentrated in particular training conditions? Could these errors change the reported trend that post-training increases action revision?

Reference

[1] https://arxiv.org/abs/2510.08558

Reviewer_vuBaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors analyze if predicting transitions in a world model correctly is necessary for performing better quality decision-making. They work in the setting where the world model receives a reward for correctly predicting the next observation in the environment and is trained using reinforcement learning. The paper sets up a control study wherein a base world model is trained with rewards associated with true next observation, mismatched observation and random sampling. Across textual and visual domains, the paper establishes that it is not really necessary to use the rewards coming from true transitions to train the world model for performing better decision-making.

Critical strengths

Paper demonstrates that using rewards that do not contain environment information can also lead to strong performance improvement in decision-making.

Critical weaknesses
  1. The quality of presentation is seems low.
  • The paper does not formalize the setup properly and keeps using colloquial language for technical terms. e.g. I come from RL background and I find the description "actions change environment" breaking the stationarity of the environment.

  • In general, the results are put together without any coherent thread tying them together in the theme of the paper. e.g. On ScienceWorld-No-list (Fig 2-c), the results are not clearly distinguishable. The text referring to that figure does not mention or explain this phenomenon.

  • The writing does not make it clear why the topic of study is important for studying.

  1. There are no error bars for the result tables in the main text.
  2. If not predicting next observation correctly, then what makes the world models perform good decision-making is left open.
Questions for authors
  • Please add more evidence for increasing statistical significance of the results.
  • Authors can also try adding a paragraph or two describing the formal setup, define standard terminology to refer to through out the draft.
  • Can you please mention if the setup is stationary environment? If it is, then can you please change the language to "transition in environment" than "change to environment"?
  • There are short forms like RWML never elaborated upon.
  • Authors should reveal the details of AI usage along the lines of ICLR guidelines. The current statement only talks about training data collection and writing help. Please mention if AI was used in setting up the code, literature survey and ideation.
  • What are your hypotheses for world model's decision making performance if not good prediction of the state transitions?

To me, it feels like the paper needs significant work. It enquires a very reasonable question about world models and if polished well, this can provide significant insights to other researchers working in this domain.

Reviewer_RQqqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • An interesting ablation study, with a new finding that the training procedure itself might lend benefit to agents independently of the agents' ability to predict the environment.
  • I think the most surprising finding of the paper is that substantial task gains, more specifically, coverage, are observed even with random rewards and mismatched targets.
  • The biggest gains in performance are observed for the single-attempt selection setting.

Some of my issues with the paper: Mostly, the writing makes the paper difficult to follow; specifically, since this is an empirical paper with a new finding, the prose does not spend enough effort on explaining said observation or giving a prescriptive training procedure, except for the multi-modal example. This paper is terminology-heavy; however, terminology is either not properly defined/motivated or used before it is defined. Numerical details should support the story; but in their case it interrupts the narrative. The prose frequently reports numerical results that are better suited in a table. Finally, the authors do a poor job of distinguishing behavioral observation from explanations. For example, does broader action search lead to reduced looping? A lot is left to the reader’s imagination.

Critical strengths

S1: problem identification (conceptual) - The general direction is useful; that is, we can gain agent task performance for free by randomisation, without a world model. The general novelty is however limited see W1. S2: Interesting empirical result - Random reward training improves task coverage. Also, multiple environments and multiple training procedures give nice empirical coverage.

Critical weaknesses

W1: This is an empirical paper with a specific and narrow focus on LLM-agent world post-training. The standard result that better world prediction does not imply better action has been established in model based RL (Wei et al. 2024 - A Unified view on Solving Objective Mismatch in Model-based RL). Also, specifically for COIN (Shao et al - Spuriour Rewards: Rethinking Training Signals in RLVR). The conceptual findings are therefore less novel than suggested despite its specific setting and GT-MIS-COIN experiments.

W2: Little to no identification of mechanisms responsible for the performance gains. The authors distinguish content dependent improvements from what tthey call “optimization effects”. However we are never made aware what they optimization effects actually are. There is no causal attribution to why agents are actually improving.

Questions for authors

Q1: What exactly constitutes an optimization effect? Q2: Why does random-reward training/incorrect supervision improve performance? Q3: Why are action revision, entity reference, multiple candidate generation good behavioral indicators for succesful decision making? Q4: How robust are the COIN results across independent training runs? Q5: How does the VisualWebArena compare to correct world-model supervision?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.