acceptodds
Under review as a conference paper at ICLR 2027

How To Train Your World Model: Fine-tuning vs RAG for LM-based World Modeling

Abstract

World models (WMs) simulate the transition dynamics of environments, enabling agents to plan over the consequences of their actions. In text-based environments, fine-tuning a Language Model (LM) to serve as a WM has emerged as a dominant paradigm. However, despite the widespread success of non-parametric approaches such as Retrieval Augmented Generation (RAG), retrieval over past experience for LM-based world modelling remains underexplored. We conduct a systematic evaluation across five diverse environments spanning embodied, web navigation and social settings, comparing fine-tuning and RAG-based approaches for LM-based world modelling. Our study reveals that fine-tuning often outperforms RAG, with fine-tuned WMs enabling agents to obtain higher rewards on 15/20 settings. However, there are several cases where RAG proves superior, and there are often cases where simple prompting baselines match or outperform finetuning. While both construction paradigms benefit from additional and more diverse exploration, RAG-based approaches prove more data-efficient, while fine-tuning approaches disproportionately benefit from scaling the amount of experience collected. With a focus on RAG-based WMs, we devise a procedure that uses counterfactual intervention to estimate the error rate of the retrieval stage, and show that retrievers consistently surface suboptimal transitions from the experience buffer. Hoping to address this failing, we study a variety of query reformulation strategies, demonstrating that a hierarchical approach outperforms the traditional retrieval pipeline. Finally, we compose our findings into a hybrid world modelling system that parametrically captures core environment dynamics, while learning to rely on retrieval from an actively maintained memory store. Our hybrid system consistently outperforms other methods across multiple environments and models, showcasing the robustness of the approach and the applicability of our findings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Models, Retrieval, Memory, LLM, Language Model, RAG

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gXNrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides a broad empirical comparison of fine-tuned and retrieval-based LM world models, showing that fine-tuning is usually stronger while RAG is more sample-efficient and highly environment-dependent. It proposes hierarchical retrieval combined with fine-tuning, which improves downstream utility across evaluated settings.

Critical strengths
  1. Direct comparisons between fine-tuning and retrieval over self-collected experience are useful for LM-based world modeling, and the experiments cover several types of text environments rather than relying on a single benchmark.
  2. This is appropriate because a more textually faithful prediction does not necessarily lead to better action selection. The reported results also usefully show that neither FT nor RAG is uniformly preferable across environments.
Critical weaknesses
  1. The Hybrid gain is not attributed. Hybrid adds retrieval at inference and fine-tunes on retrieval-augmented inputs, and the paper does not specify which retriever it uses; no ablation isolates these factors. The 17/20 result therefore does not show that learned parametric/non-parametric fusion is responsible. This matters because Hybrid is the paper’s main proposed system.
  2. The oracle-normalized aggregate is fragile. The oracle only provides true next observations for five policy-proposed actions, and an imperfect LM value estimate still selects the action, so it is not a strict upper bound; Chameleon’s -4.50 normalized utilities also suggest a small denominator. This matters because the 77% vs. 60% headline depends on this normalization.
Questions for authors
  1. Report results for hierarchical retrieval without fine-tuning and for an FT model given the same retrieved transitions at inference time, using the same backbone, buffer, and evaluation budget. These comparisons are necessary to identify the source of the Hybrid improvement.
  2. Provide the raw utility, ground-truth-WM utility, number of evaluation episodes, and number of independent runs for each setting. Also report whether the 77% versus 60% result remains under aggregation based on raw rewards or after excluding settings with near-zero oracle utility.
  3. Clarify the implemented Chameleon reward. Appendix A states that the evaluated player receives −1-1 when accused, but also states that this outcome cannot occur because the player is never assigned the Chameleon role. An incorrect accusation of that player appears possible.
  4. Typo. ("Three players is in any case" should be "are")
Reviewer_wgtSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper compares fine-tuning and retrieval-based approaches for language-model world modeling across five text-based environments. It also investigates data collection and retrieval strategies, and proposes a hybrid model trained with retrieved transitions as context.

Critical strengths

The evaluation includes both next-observation prediction and downstream task reward. Including zero-shot and few-shot baselines is useful, particularly since they sometimes outperform the adapted models.

Critical weaknesses
  1. The contribution is limited. The hybrid method combines supervised fine-tuning with retrieved examples, while the retrieval modification first matches observations and then actions. The paper reports improvements, but does not establish a sufficiently clear insight into when or why these choices work. It also lacks a controlled comparison separating the benefits of hierarchical retrieval, retrieval-conditioned training, and simply adding retrieval to a fine-tuned model.

  2. The evidence is not sufficient to support the broad conclusions. Fine-tuning hyperparameters, default buffer sizes, and evaluation sample counts are not adequately documented, making the comparison difficult to assess. Some claims also overstate the plotted results: Section 4.1 says RAG universally peaks at an exploration ratio of 50%, whereas Figure 2 shows a different peak for Sotopia. These issues matter because the paper’s main contribution rests on the reliability of its empirical comparisons.

Questions for authors
  1. How does the hybrid compare with a normally fine-tuned model given retrieved examples only at inference? What is the separate contribution of hierarchical retrieval?

  2. What training budgets, hyperparameter searches, buffer sizes, and evaluation sample counts were used for each method?

  3. Which findings remain consistent across models and environments, and which are specific to the settings shown in the ablations?

Reviewer_ogfvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares RAG-based world models with fine-tuning-based world models. It also presents a hybrid paradigm which combines the parametric and RAG-based approaches. The paper evaluates these world model methods on five diverse datasets spanning embodied, web navigation, and social settings. Fine-tuned world models consistently outperform RAG-based world models except on the Chameleon dataset.
The proposed hybrid world model further outperforms the fine-tuned world models in most settings.

Critical strengths
  1. The paper is the first to make an apples-to-apples comparison between RAG-based and fine-tuning-based world models across various domains. The results would benefit future work on building world models.

  2. The proposed hybrid world models combine the advantages of the RAG-based and fine-tuning-based world models and outperform both in most scenarios.

Critical weaknesses
  1. The paper lacks so many details in its experimental settings. Please see my questions below. Without knowing such information, it is hard to know how sound the experiments are.

  2. The fidelity score design raises some concerns. 1. Gemma-4-31B is used as the judge. But it was also used as the agent policy model, and possibly the world model, but the paper didn’t state what the backbone models for the world models are. This raises concerns about whether there is bias. 2. The world model prediction is free-form text generation. It’s unclear whether the free-form text generation is the desired the behaviors of world models, comparing to slot filling. The BLEU score may not be a good metric.

  3. The paper does not reveal the absolute rewards or scores on each benchmark, which makes it impossible to compare them with prior work. The paper would benefit from revealing both absolute scores using each benchmark’s official metrics and the reported relative reward.

  4. Small things that can be improved: Table 17 is rotated. The content needs to be adjusted for readability. Figure 3’s resolution is very different from that of other parts of the paper.

Questions for authors
  1. What are the backbone models for the world models? Are they the same as the policy models in all settings?
  2. What are the world model fine-tuning settings? How large is the training dataset in each domain? The fine-tuned world model shows the greatest benefit on WebShop and ALFWorld, but there is no analysis of the two domains when it comes to the data scale analysis in Figure 2.
  3. What is the judge model in each benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.