acceptodds
Under review as a conference paper at ICLR 2027

Towards a Belief-Based World Model for LLM Agents

Abstract

Large language models (LLMs) are being used as policies for autonomous decision-making and planning in many domains. Despite their strong reasoning capabilities, LLMs struggle with long-horizon tasks, especially under partial observability. World models are a promising way to enhance policy performance, both during training and inference. During inference, agents currently use world models to simulate the consequences of candidate actions before choosing an action, which can improve decision-making. However, we argue that simulation alone is an incomplete interface for decision-making under partial observability: simulation doesn't adequately capture uncertainty about the current state, which agents may need for accurate decision-making. We address this limitation with Belief World Models (BWMs), which maintain a belief that LLMs can query to access information on what is known and uncertain about the current state. Before developing methods to learn accurate BWMs, this paper focuses on a more fundamental question: does exposing a world model's belief directly to an LLM policy improve decision-making? Our results show that giving LLM agents access to beliefs improves task performance under partial observability, while remaining complementary to existing simulation-based world models. Code to be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world model, belief, llm agents, uncertainty

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Uy7aOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Belief-Based World Models (BB-WMs), which allow LLM agents to query explicit beliefs about the current environment state, complementing conventional simulation-based world models. The authors evaluate this approach on ALFWorld, ScienceWorld, and BabyAI using hand-crafted belief models and action-validity checkers. Experiments show improved task performance and efficiency across multiple LLMs and agent frameworks.

Critical strengths

The paper highlights the importance of explicitly representing current-state uncertainty for LLM decision-making under partial observability. The proposed belief-query interface is intuitive and practically useful. Experiments across multiple benchmarks, LLMs, and agent frameworks demonstrate the benefits of belief access. The memory-only and non-uniform prior ablations further support the usefulness of explicit belief information.

Critical weaknesses
  1. The proposed approach relies heavily on environment-specific priors and rule-based belief updates derived from game-engine knowledge. Although the authors investigate non-uniform priors, the provided priors still match the underlying environment distributions. It remains unclear whether the improvements persist when beliefs are inaccurate or misspecified, limiting the generalizability of the conclusions.
  2. The paper presents a useful interface for exposing belief information to LLM agents, and the ablation studies demonstrate its effectiveness. However, the implemented BB-WMs rely heavily on benchmark-specific priors and relatively simple rule-based belief updates. It remains unclear to what extent the improvements stem from the proposed belief-query mechanism itself, rather than from providing additional structured environment knowledge. This makes the incremental contribution of the proposed approach difficult to assess.
Questions for authors
  1. How sensitive is BB-WM to inaccurate belief distributions? Could the authors evaluate robustness under perturbed priors, incorrect probability rankings, or errors in the candidate-state support?
  2. Does the Memory-only baseline in Fig. 4 have access to the same environment-specific placement constraints used by BB-WM to construct its belief priors (e.g., which receptacles an egg can appear in according to the ALFWorld game rules)? If not, could the authors compare against a Memory baseline provided with the same prior knowledge? This would help distinguish whether the improvements come primarily from additional environment-specific knowledge or from explicitly maintaining and querying updated beliefs.
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer Uy7a

We thank the reviewer for their review, which highlights the proposed belief-query interface as intuitive and practically useful. We address the raised weaknesses below:

Inaccurate beliefs (Weakness #1)

Conceptually, an inaccurate world model will undoubtedly adversely affect the policy, given the policy is relying on the world model's responses to inform what actions to take. The implemented BB-WMs are error-free belief world models (specifically, the current state uncertainty and belief update components). This is by-design, and the implemented BB-WMs are likely an upper-bound on the gains of such methods for simple scenarios. On the other hand, for complex scenarios, it is infeasible to hand-specify error-free BB-WMs, and instead, it is likely more accurate to learn them from data (environment observation and action data). Thus, we view this work as a necessary first step to demonstrate the benefits of BB-WMs and to motivate work into generalized learning of BB-WMs.

We empirically validate three variants on the same setting as Section 5.3 (non-uniform priors). Each variant has increasingly more error:

  1. Belief-Perturbed: We perturb the world model's numerical probability values away from the ground-truth prior, but the overall ordered rankings are unchanged. Thus, the order and support are accurate.
  2. Belief-Shuffled: We randomly re-assign probability values across the candidate-state support. Thus, the order is changed, but the overall support is still accurate.
  3. Belief-Dropped: We drop the true receptacle from the distribution, and re-normalize. Thus, the support is inaccurate in the most severe way.

As an example, an unseen apple (that is actually in the fridge) may have a distribution over the support as follows:

ReceptacleOriginalPerturbedShuffledDropped
fridge0.5400.5110.1370
cabinet0.2270.2290.0960.493
drawer0.1370.1480.5400.298
countertop0.0960.1110.2270.209

Results can be seen here. tl;dr -- Belief-Dropped results in severe performance degradation, and the other variants generally perform on-par or worse than Belief. Thus, it is critical for future world models to accurately model the candidate-state support; there is some tolerance for errors in the associated probability values.

Reliance on prior knowledge (Weakness #2 & Question #2)

The implemented BB-WMs intentionally rely on benchmark-specific priors. Since neither the Memory baseline or the Original agent baseline had access to the object-placement prior available to Belief. To make a fair comparison, we introduce two additional baselines: (1) Original-Prior and (2) Memory-Prior

In Original-Prior, the policy gets the same placement constraints as the world model (in the initial prompt). Since Original-Prior has no world model, the policy must implicitly reason about and update its beliefs as actions are taken and new observations come. We experiment on Llama & Qwen on ALFWorld & ScienceWorld. Results can be seen here. For Llama, Original-Prior does not perform substantially better than Original; for Qwen on ALFWorld, Original-Prior outperforms Original. In no case is Original-Prior better than Belief. We draw the following conclusion: LLMs, even when given access to the same prior knowledge and when the belief updates are very simple, may not be best suited for updating beliefs while acting. We believe world models are better positioned to do this (for reasons described in lines 80-90). This reinforces the strength of our work: unlike current world models that only simulate the future (e.g., WALL-E), our world models maintain and update a belief about the current state.

In Memory-Prior, we take the same Original-Prior baseline, but give it access to a memory-only world model. We similarly study Llama & Qwen on ALFWorld & ScienceWorld. Results can be seen here. On ALFWorld: for Llama, we observe a severe degradation in capabilities (underperforming Original), and for Qwen, performs is somewhat better than Memory, but worse than Belief. On ScienceWorld, Memory-Prior does not outperform Belief across any configuration. Similar to before, these results favor maintaining and updating explicit beliefs in the world model. By making the prior knowledge accessible via an agent query, the agent can access this knowledge when needed.

We will add the Original-Prior results above the current Section 5.2, and we will update the current Section 5.2 (Memory vs. Belief) with the Memory-Prior results.

If our response has addressed your concerns, please consider raising your score. Otherwise, we are happy to engage further during the discussion period!

Thank you for the detailed response and the additional experiments. The new results on sensitivity to inaccurate belief priors, together with the comparisons against Original-Prior and Memory-Prior, help address my main concerns. I encourage the authors to incorporate these two sets of experimental results and the corresponding discussions into the revised manuscript to make the findings accessible to readers.

Given these clarifications and additional evidence, I am raising my rating from 2 to 3.

We thank the reviewer for their response and are glad that we have addressed the reviewer's main concerns. We will include the results and discussions above in the revised manuscript.

Reviewer_M2NmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper evaluates whether LLM agents gain by querying directly the belief held by a world model on the current state. The authors construct hand-made belief trackers on ALFWorld, ScienceWorld, and BabyAI, which together with a WALL-E style action validity checker are evaluated with three different LLMs and two different agent frameworks, showing that belief access helps the performance of tasks and decreases environment steps. Further experiments compare belief access to state memory and test the impact of non-uniform priors on object locations.

Critical strengths
  1. Hand-made belief construction is a reasonable option in the current experiment. It allows authors to check whether frozen LLMs are able to use belief access without mixing with mistakes of belief estimation.

  2. The comparison of Belief, WALL-E, and BB-WM is quite useful. For example, on BabyAI, Sonnet gains a lot from belief access, whereas WALL-E does not help much.

  3. The memory-only ablation in Section 5.2 is necessary because belief access also gives access to the state and search history.

Critical weaknesses
  1. Novelty is limited because of existing studies on explicit beliefs for LLM agents such as BeliefMem [1], ABBEL [2], and Agent-BRACE [3]. The new contribution is the integration of belief into WALL-E, but these two pieces do not work together in terms of the belief being used to predict future states while WALL-E verifies the actions only.

  2. The Memory baseline in Figure 4 does not have access to the object-placement prior that is available in Belief.

  3. In BabyAI, the agent has a possibility to query a spatial map, its location and orientation, as well as the objects it had seen before. This should already be quite helpful in the task of navigating, but the memory-only ablation does not apply to BabyAI.

  4. Numerical probability and probability ranking are mixed in Figure 5. Belief-NoProb discards numerical probability and reorders the candidates as well; hence, it can be an effect of just ranking that gives improvement.

Questions for authors
  1. What is the performance of a prior-only baseline on ALFWorld in which the agent gets the initial list of valid receptacle locations without further belief updates?

  2. How much of the improvement on BabyAI remains if the agent can query the spatial map and observed state, but not beliefs over unseen locations?

[1] Belief Memory: Agent Memory Under Partial Observability.

[2] ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interaction.

[3] Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty.

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer M2Nm

We thank the reviewer for their review, which highlights it's soundness with useful and necessary comparisons/experiments. We address the weaknesses raised by the reviewer below:

Novelty with respect to prior works (Weakness #1)

We respectfully disagree that the "new contribution is the integration of belief into WALL-E". We feel that this characterization misses the conceptual strengths. Unlike prior works, we conceptually motivate why a world model needs beliefs (line 43-79 and Fig. 1), which is a significant departure from the simulation-only paradigm of current world models (see lines 192-194 and Fig. 2). The referenced papers are not world models and are discussed in lines 135-150. Thus, this work is the first to add and empirically evaluate explicit belief tracking and updates with world models.

Experimentally, we integrate beliefs into WALL-E as a simple instantiation of BB-WMs in order to test our research question. Doing this allows us to show that the benefits of (1) belief estimation and (2) foresight from simulation are complementary. Furthermore, prior works seldom study efficiency, whereas we show that adding beliefs directly improves the agent's efficiency (i.e., agent takes better actions).

We agree that belief-based world models will be strengthened by having the "two pieces ... work together". This is included in the limitations and future work mentioned in the Conclusion (lines 482-485).

Memory baseline lacks prior (Weakness #2 & Question #1)

Neither the Memory baseline or the Original agent baseline had access to the object-placement prior available to Belief. To make a fair comparison, we introduce two additional baselines: (1) Original-Prior and (2) Memory-Prior

In Original-Prior, the policy gets the same placement constraints as the world model (in the initial prompt). Since Original-Prior has no world model, the policy must implicitly reason about and update its beliefs as actions are taken and new observations come. We experiment on Llama & Qwen on ALFWorld & ScienceWorld. Results can be seen here. Results tl;dr -- Original-Prior performs similarly to Original, and in no case is it better to our Belief variant. Since Original-Prior has no world model, the policy must implicitly reason about and update its beliefs as actions are taken and new observations come. We draw the following conclusion: LLM policies, even when given access to the same prior knowledge, may not be best suited for updating beliefs while acting. We believe world models are better positioned to do this (for reasons described in lines 80-90). This reinforces the strength of our work: unlike current world models that only simulate the future (e.g., WALL-E), our world models maintain and update a belief about the current state. We will add a new section in the main paper (above Section 5.2) with these results and discussion.

In Memory-Prior, we take the same Original-Prior baseline, but give it access to a memory-only world model. We similarly study Llama & Qwen on ALFWorld & ScienceWorld. Results can be seen here. Results tl;dr -- Memory-Prior never performs better than our Belief variant. On ALFWorld: for Llama, we observe a severe degradation in capabilities (underperforming Original), and for Qwen, performs is somewhat better than Memory, but worse than Belief. On ScienceWorld, Memory-Prior does not outperform Belief across any configuration. Similar to before, these results favor representing prior knowledge in the world model. By making the prior knowledge accessible via an agent query, the agent can access this knowledge when needed.

We will add the Original-Prior results above the current Section 5.2, and we will update the current Section 5.2 (Memory vs. Belief) with the Memory-Prior results.

AuthorsOpenReview ↗

Part 2 of Author Response

BabyAI memory baseline (Weakness #3 & Question #2)

We implement the Memory baseline for BabyAI such that the spatial map does not show the full grid layout and which spots are unknown; instead, these spots are blank. Aside from this change, the Memory world model is the same as the Belief case. As an example, the Memory spatial map is:

      #
      . B
# . . ^

where ^ is the agent location, B denotes a box, . denotes an empty cell, and # denotes a wall. The corresponding Belief map is:

# # # # # # # #
# . . . B . . #
# . . ^ . . . #
# ? ? . ? ? ? #
# ? ? ? ? ? ? #
# ? ? ? ? ? ? #
# ? ? ? ? ? ? #
# # # # # # # #

where ? additionally denotes an unobserved grid cell.

Results can be seen here. tl;dr -- all models benefit more from belief than memory, but the benefits are marginal for stronger models. Llama fails to make effective use of Memory information. On Qwen, Memory rises quicker, but plateaus sooner than Belief. Because of the sparser map, we find that Memory allows the Qwen agent to quickly solve the task if the object is seen, but unlike Belief, it does not help Qwen explore unseen grid cells. Sonnet benefits from Memory information, but consistenly performs slightly worse than Belief, indicating that there is some benefit (even to strong LLMs) in world models that maintain explicit uncertainty over the state.

We will add this result in the Appendix, and reference it in the main text in Section 5.2.

Confound of numerical probability & rankings (Weakness #4)

We introduce Belief-OrderedRank, which keeps the order but discards numerical probabilities and evaluate it in the same setting as Section 5.3. Results can be seen here.

Generally, we see that Belief-OrderedRank and Belief perform similarly (with the exception of Llama ReflAct). This adds some nuance to our discussion in lines 453-457: it is true that probability information improves search efficiency, but this information need not necessarily be numerical. Ordered rankings can also provide the same information to the agent.

We will update Section 5.3 with the results and discussion accordingly.

If our response has addressed your concerns, please consider raising your score. Otherwise, we are happy to engage further during the discussion period!

AuthorsOpenReview ↗

Checking In

Dear Reviewer M2Nm,

As we near the end of the discussion period, we wanted to check in on our response above. If our response has addressed your concerns, please consider raising your score. Otherwise, we are happy to engage further during the discussion period.

Reviewer_iKVsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discuss the function of world models beyond next state simulation. The paper argues that although simulation works well for pragmatic actions, the world model can also provide the belief of the environment for epistemic actions, which can reduce the uncertainty of the environment for the policy model. A Belief-Based World Model (BB-WM) support both simulation for next state as well as answering policy model’s questions.
The authors conducted experiments on three text-based games ALFWorld, Science World, and BabyAI with 3 LLMs and 2 agents frameworks.

BB-WM generally improves over the base agents with belief-only or simulation-only world models interfaces. Moreover, the belief-only interface outperforms the simulation-only interface and a memory-only interface in most settings.

Critical strengths
  1. The paper discusses a limitation of existing world model interface which solely simulates the future states and propose an useful extension of world models by allowing them to answer policy models’ questions about environment beliefs in addition to simulation.
  2. The results on various text-based games demonstrates the benefits of this belief-based interface for world models.
Critical weaknesses
  1. The evaluation focuses on simulated environments where policy models may not have the same prior knowledge as the belief tracking world models, and the simulation interface is too simple. For example, in ALFWorld, a cup can appear in different compatible places, and the belief tracker is given these placement constraints. But the simulation interface only checks whether an action such as picking up a cup from a table is valid, rather than predicting the full next state. Such settings may overstate the importance of belief and understate the importance of simulation. The paper would benefit from enhancing the simulation baseline with full state prediction and conducting experiments on other domains such as computer-use agents, web search agents, coding agents to provide a comprehensive understanding of world model interfaces.
  2. The paper also lacks a baseline that requires the policy model to explicitly reason about beliefs. This is necessary to understand whether we should use world models or policy models for beliefs.
Questions for authors

Do policy models have the same knowledge as the belief tracking world models in the experiments?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer iKVs

We thank the reviewer for their review, which highlights our work as a useful extension of world models. We address the weaknesses raised by the reviewer below:

Simulation limited to action validity (Weakness #1)

The paper intentionally restricted the simulation baseline to action validity. Prior work observed that the full next observation may not be helpful to the policy [1,2].

We confirm this with an experiment that enhances the simulation baseline with full-state prediction. Specifically, the world model responds with the next environment observation, and the policy is given the option to re-plan or execute the action. We call this variant NextObs, and show results here for ALFWorld on Llama and Qwen. Results tl;dr -- NextObs underperforms even the no world model setting (Original).

Discussion: The results indicate that the policy cannot effectively make use of foresight in terms of next observation simulation. Instead, providing targeted information (e.g., action valid/invalid) is more helpful. This supports our conceptual shift to modeling current-state uncertainty via the BB-WM, as this may be more useful to a policy than foresight from next-state simulation. We will add this experiment in the Appendix and reference it in the main text (after Section 5.3).

[1] Siyu Zhou et al., WALL-E: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents, NeurIPS 2025. [2] Cheng Qian et al., Current Agents Fail to Leverage World Model as Tool for Foresight, ACL 2026.

Policy models lack same prior knowledge (Weakness #2, Question #1)

The policy model (Original in our experiments) does not have access to the same prior knowledge as our world model. To make a fair comparison, we introduce an additional baseline, Original-Prior, where the policy gets the same placement constraints as the world model (in the initial prompt).

We experiment on Llama & Qwen on ALFWorld & ScienceWorld. Results can be seen here. Results tl;dr -- Original-Prior performs similarly to Original, and in no case is it better to our Belief variant. Since Original-Prior has no world model, the policy must implicitly reason about and update its beliefs as actions are taken and new observations come. We draw the following conclusion: LLM policies, even when given access to the same prior knowledge, may not be best suited for updating beliefs while acting. We believe world models are better positioned to do this (for reasons described in lines 80-90). This reinforces the strength of our work: unlike current world models that only simulate the future (e.g., WALL-E), our world models maintain and update a belief about the current state. We will add a new section in the main paper (above Section 5.2) with these results and discussion.

Experiments on other domains (Weakness #1)

We agree that extending to other domains like computer-use and coding are important world-model applications. However, we believe that belief-based world models (BB-WMs) for these domains will require learned deep neural nets (i.e., these domains are too complex for hand-specified world models). In order to motivate efforts for creating such models, this work deliberately abstracted away the seperate challenge of learning BB-WMs by focusing on simpler environments. This allowed us to empirically test the benefits of BB-WMs for agent planning & decision-making.

We will expand upon the limitations mentioned in the Conclusion (line 482) to highlight the importance of such settings for gaining a comprehensive understanding of world model interfaces.

If our response has addressed your concerns, please consider raising your score. Otherwise, we are happy to engage further during the discussion period!

AuthorsOpenReview ↗

Checking In

Dear Reviewer iKVs,

As we near the end of the discussion period, we wanted to check in on our response above. If our response has addressed your concerns, please consider raising your score. Otherwise, we are happy to engage further during the discussion period.

Thank you for your response. It addressed most of my concern. I have raised my scores.

We thank the reviewer for their response and are glad that we have addressed most of the reviewer's concerns. We will include the results and discussions above in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank the reviewers for the effort they've put into their reviews. In particular, reviewers note that:

  • BB-WMs are a useful extension (iKVs) to world models
  • the proposed method is intuitive and practically useful (Uy7a)
  • the paper is generally clear, informative, and sound with fixable concerns.

We respond to all weaknesses and questions. All reviewers had concerns about privileged prior knowledge leading to unfair comparisons. We address this with a controlled study showing that agents do not use the same prior information as effectively as world models, highlighting the importance of capturing this information in a world model instead.

If our responses have addressed your concerns, please consider raising your score. Otherwise, we are happy to engage further during the discussion period!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.