acceptodds
Withdrawn from ICLR 2027

3D-Belief: Embodied Belief Inference via Generative 3D World Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in visual generative models have highlighted the promise of learning generative world models. However, most existing approaches frame world modeling as novel-view synthesis or future-frame prediction, emphasizing visual realism rather than the structured uncertainty required by embodied agents acting under partial observability. In this work, we propose a different perspective: world modeling as embodied belief inference in 3D space. From this view, a world model should not merely render what may be seen, but maintain and update an agent’s belief about the unobserved 3D world as new observations are acquired. We identify several key capabilities for such models, including spatially consistent scene memory, multi-hypothesis belief sampling, sequential belief updating, and semantically informed prediction of unseen regions. We instantiate these ideas in 3D-Belief, a generative world model that infers and updates explicit, actionable 3D beliefs from partial observations. Unlike prior approaches, 3D-Belief updates 3D scene hypotheses by retaining observed geometry, incorporating new evidence, and resampling unseen regions, enabling embodied agents to reason and act under partial observability. We evaluate 3D-Belief on 2D visual quality for scene memory and unobserved-scene imagination, object- and scene-level 3D imagination using our proposed 3D-CORE benchmark, and challenging object navigation tasks in both simulation and the real world. Experiments show that 3D-Belief improves 2D and 3D imagination quality and downstream embodied task performance compared to state-of-the-art methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D World Models, Embodied AI, Belief Inference, Scene Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pNQ5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

3D-Belief frames embodied world modeling under partial observability as sequential 3D belief inference using 3D Gaussian Splatting (3DGS) primitives with semantic features. It combines a U-ViT backbone, an MVS cost-volume depth head, and a semantic head, trained on image pairs via forward-rendering diffusion. At inference, it accumulates observed Gaussians via concatenation and resamples unseen regions for navigation. While evaluated across view synthesis, 3D-CORE, and navigation, the high-level belief-update framing diverges from the actual pipeline, alongside missing technical formulations and evaluation confounders.

Critical strengths
  1. Practical 3D-Semantic Integration for Navigation: Pairing explicit 3DGS generation with open-vocabulary features enables trajectory evaluation without costly per-step VLM queries, achieving 59.17% SR at 2.14 s/step with zero VLM tokens in simulation (Table 6).
  2. Broad Empirical Scope: Evaluations span 2D synthesis on AI2-THOR and RealEstate10K (Table 1), 3D completion on 3D-CORE (Table 2), spatial QA on SAT-Real (Table 11), and navigation in simulation and on a Stretch robot (Table 4).
Critical weaknesses
  1. Gap Between POMDP Formulation and Test-Time Concatenation: The paper frames the method as sequential Bayesian belief updating, but the network is trained strictly on isolated image pairs and does not condition on accumulated 3D states at test time. Instead, observed Gaussians are simply concatenated without deduplication or posterior refinement (leading to linear memory growth and drift), while prior unseen hypotheses are discarded at each step. The claims regarding belief updating and structured uncertainty should be revised to reflect rendering-guided primitive concatenation.
  2. Underspecified Mechanisms and Incremental Architecture: The model combines existing components (DFM, DFoT, MVSplat) with several unaddressed technical gaps: (i) how MVS plane-sweep matching operates under diffusion noise or zero frustum overlap; (ii) undocumented supervisory losses and regularization cited in appendices but missing from the main text; and (iii) missing mathematical definitions for trajectory scoring and key ablations (Table 7).
  3. Confounded Evaluation Protocols: Training involves a composite dataset and depth supervision not matched by the baselines, and real-world navigation relies on test-time depth sensors for scale alignment alongside a permissive 4.0 m success radius that spans most of emulator/test rooms. Controlled ablations under matched data and standard 1.0 m thresholds are needed.
Questions for authors
  1. What is the performance of 3D-Belief when trained strictly under the same data and supervision modalities as the baselines?
  2. How does the MVS cost-volume module compute valid features under heavy diffusion noise or non-overlapping viewpoints?
  3. Could the authors provide the missing mathematical formulations for the trajectory scoring function and key ablations like "w/o geometry"?
  4. What are the navigation success rates under a standard 1.0 m threshold instead of the 4.0 m radius?
Reviewer_NnkoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents 3D-Belief, a generative world model that maintains observed 3D Gaussians and resamples hypotheses for unobserved regions as new observations arrive. The representation incorporates geometry, appearance, and semantic features, supporting rendering and object navigation. Experiments evaluate visual synthesis, 3D completion, and navigation in simulation and the real world.

Critical strengths
  1. The motivation is compelling: embodied agents operating under partial observability should preserve previously observed information while reasoning about unseen regions, with both geometric and semantic understanding.
  2. The integration of scene memory, generative imagination, and semantic querying provides a coherent framework for embodied navigation.
  3. The evaluation spans perception, imagination, and downstream navigation, with promising improvements over the evaluated baselines.
Critical weaknesses
  1. Limited technical novelty. The method builds closely on existing components, including MVSplat-style geometry prediction, DFM-style diffusion with rendering, and semantic features embedded in 3D representations. Gaussian memory for online exploration has also been studied in EmbodiedOcc [1]. The paper should clarify what the proposed memory and imagination update adds beyond combining these methods.

  2. Insufficient evaluation of long-term spatial memory. The “scene memory” experiment in Table 1 mainly tests view interpolation between two observations, while the completion tasks mainly test imagination. The paper lacks a direct evaluation of spatial memory consistency and persistence over long input sequences. This is a central concern: stable memory is important for navigation, where incorrect decisions may require agents to backtrack through previously explored areas.

  3. Limited semantic evaluation. Appendix C.6 evaluates open-vocabulary localization, but lacks comparisons with related methods that jointly perform 3D reconstruction and semantic understanding, such as IGGT [2].

References

[1] Yuqi Wu et al. EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understanding. ICCV, 2025.

[2] Hao Li et al. IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction. ICLR, 2026.

Questions for authors

Please address the weaknesses above.

Reviewer_ztKbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The. paper proposed 3D-belief, a generative model that keep explicit semantic 3D gaussian scene. In doing so, it supports rendering, open-vocabulary query, and planing. The authors conduct comprehensive experiments, including visual prediction, real-world navigation, demonstrating the effectiveness of the paper.

Critical strengths

The pipeline supports both visual rendering, semantic query and geometry representation. It shows potential to support embodied agents, e.g., improving several navigation baseline by a large margin, demonstrating the effectiveness of the methods.

Critical weaknesses

Although the method claimed it can be used in sim and real-world navigtation, I think it is still contrained to be applied in real world, as this method is limited to handle static objects and scens, and the performance of dynamic GS is still unsatisfactory. On 3D-CORE, the only reported baseline is DFoT-VGGT. Its errors combine video-generation inconsistency, geometry lifting, and reconstruction errors. This establishes an advantage over that particular pipeline but does not establish an advantage over direct generative 3D approaches discussed in the related work.

Questions for authors

See the weakness part and provide more evidence to supprot the effectiveness of the method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.