- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Belief-Based World Models (BB-WMs), which allow LLM agents to query explicit beliefs about the current environment state, complementing conventional simulation-based world models. The authors evaluate this approach on ALFWorld, ScienceWorld, and BabyAI using hand-crafted belief models and action-validity checkers. Experiments show improved task performance and efficiency across multiple LLMs and agent frameworks.
Critical strengths
The paper highlights the importance of explicitly representing current-state uncertainty for LLM decision-making under partial observability. The proposed belief-query interface is intuitive and practically useful. Experiments across multiple benchmarks, LLMs, and agent frameworks demonstrate the benefits of belief access. The memory-only and non-uniform prior ablations further support the usefulness of explicit belief information.
Critical weaknesses
- The proposed approach relies heavily on environment-specific priors and rule-based belief updates derived from game-engine knowledge. Although the authors investigate non-uniform priors, the provided priors still match the underlying environment distributions. It remains unclear whether the improvements persist when beliefs are inaccurate or misspecified, limiting the generalizability of the conclusions.
- The paper presents a useful interface for exposing belief information to LLM agents, and the ablation studies demonstrate its effectiveness. However, the implemented BB-WMs rely heavily on benchmark-specific priors and relatively simple rule-based belief updates. It remains unclear to what extent the improvements stem from the proposed belief-query mechanism itself, rather than from providing additional structured environment knowledge. This makes the incremental contribution of the proposed approach difficult to assess.
Questions for authors
- How sensitive is BB-WM to inaccurate belief distributions? Could the authors evaluate robustness under perturbed priors, incorrect probability rankings, or errors in the candidate-state support?
- Does the Memory-only baseline in Fig. 4 have access to the same environment-specific placement constraints used by BB-WM to construct its belief priors (e.g., which receptacles an egg can appear in according to the ALFWorld game rules)? If not, could the authors compare against a Memory baseline provided with the same prior knowledge? This would help distinguish whether the improvements come primarily from additional environment-specific knowledge or from explicitly maintaining and querying updated beliefs.
3 comments
Thank you for the detailed response and the additional experiments. The new results on sensitivity to inaccurate belief priors, together with the comparisons against Original-Prior and Memory-Prior, help address my main concerns. I encourage the authors to incorporate these two sets of experimental results and the corresponding discussions into the revised manuscript to make the findings accessible to readers.
Given these clarifications and additional evidence, I am raising my rating from 2 to 3.
We thank the reviewer for their response and are glad that we have addressed the reviewer's main concerns. We will include the results and discussions above in the revised manuscript.
Author Response to Reviewer Uy7a
We thank the reviewer for their review, which highlights the proposed belief-query interface as intuitive and practically useful. We address the raised weaknesses below:
Inaccurate beliefs (Weakness #1)
Conceptually, an inaccurate world model will undoubtedly adversely affect the policy, given the policy is relying on the world model's responses to inform what actions to take. The implemented BB-WMs are error-free belief world models (specifically, the current state uncertainty and belief update components). This is by-design, and the implemented BB-WMs are likely an upper-bound on the gains of such methods for simple scenarios. On the other hand, for complex scenarios, it is infeasible to hand-specify error-free BB-WMs, and instead, it is likely more accurate to learn them from data (environment observation and action data). Thus, we view this work as a necessary first step to demonstrate the benefits of BB-WMs and to motivate work into generalized learning of BB-WMs.
We empirically validate three variants on the same setting as Section 5.3 (non-uniform priors). Each variant has increasingly more error:
As an example, an unseen apple (that is actually in the fridge) may have a distribution over the support as follows:
Results can be seen here. tl;dr -- Belief-Dropped results in severe performance degradation, and the other variants generally perform on-par or worse than Belief. Thus, it is critical for future world models to accurately model the candidate-state support; there is some tolerance for errors in the associated probability values.
Reliance on prior knowledge (Weakness #2 & Question #2)
The implemented BB-WMs intentionally rely on benchmark-specific priors. Since neither the Memory baseline or the Original agent baseline had access to the object-placement prior available to Belief. To make a fair comparison, we introduce two additional baselines: (1) Original-Prior and (2) Memory-Prior
In Original-Prior, the policy gets the same placement constraints as the world model (in the initial prompt). Since Original-Prior has no world model, the policy must implicitly reason about and update its beliefs as actions are taken and new observations come. We experiment on Llama & Qwen on ALFWorld & ScienceWorld. Results can be seen here. For Llama, Original-Prior does not perform substantially better than Original; for Qwen on ALFWorld, Original-Prior outperforms Original. In no case is Original-Prior better than Belief. We draw the following conclusion: LLMs, even when given access to the same prior knowledge and when the belief updates are very simple, may not be best suited for updating beliefs while acting. We believe world models are better positioned to do this (for reasons described in lines 80-90). This reinforces the strength of our work: unlike current world models that only simulate the future (e.g., WALL-E), our world models maintain and update a belief about the current state.
In Memory-Prior, we take the same Original-Prior baseline, but give it access to a memory-only world model. We similarly study Llama & Qwen on ALFWorld & ScienceWorld. Results can be seen here. On ALFWorld: for Llama, we observe a severe degradation in capabilities (underperforming Original), and for Qwen, performs is somewhat better than Memory, but worse than Belief. On ScienceWorld, Memory-Prior does not outperform Belief across any configuration. Similar to before, these results favor maintaining and updating explicit beliefs in the world model. By making the prior knowledge accessible via an agent query, the agent can access this knowledge when needed.
We will add the Original-Prior results above the current Section 5.2, and we will update the current Section 5.2 (Memory vs. Belief) with the Memory-Prior results.
If our response has addressed your concerns, please consider raising your score. Otherwise, we are happy to engage further during the discussion period!