acceptodds
Under review as a conference paper at ICLR 2027

AMOR: A Memory-guided Omnidirectional RGB-D Diffusion Model for Geometry-Consistent Next-View Synthesis

Abstract

Visual world models aim to support viewpoint exploration while preserving a coherent environment. Repeatedly extending video clips can compound geometric drift and visual degradation. For static scenes, panoramas cover more space from sparse viewpoints, but each generated view must be internally consistent in appearance and depth, and anchored to prior observations. We present AMOR, a memory-guided paradigm that progressively expands a persistent, explicit 3D scene through joint omnidirectional RGB-D generation at sparse viewpoints. Surface-guided memory readout conditions generation on target-aligned evidence from previously observed surfaces, while conservative updates integrate geometrically compatible predictions into the shared 3D scene. Beyond generating new views, AMOR’s joint modeling of appearance and geometry also enables depth estimation conditioned on observed RGB. Conservative memory updates reduce final-map geometric accuracy error by 21% relative to direct fusion. For novel-view synthesis, AMOR achieves the lowest LPIPS and FID on each of three public benchmarks and the highest average PSNR and SSIM. For depth estimation from observed RGB, it achieves the lowest RMSE and highest δ₁ on TartanGround among the evaluated methods.

Primary area:
generative models
Keywords:
World Models, Omnidirectional RGB-D Generation, Diffusion Models, Geometric Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CQqfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AMOR, a memory‑guided omnidirectional RGB‑D diffusion framework for static scenes. Taking an initial RGB panorama, its camera pose, and a specified target camera pose as inputs, the model progressively generates panoramic color images and radial depth for novel views, and fuses credible outputs into an explicitly maintained 3D scene memory.

The method consists of three core modules: ① A separate single‑channel depth prediction branch to avoid depth numerical distortion introduced by RGB‑VAE encoding. ② Surface‑guided memory readout, which performs ray‑intersection rendering on existing 3D surfels to generate conditioning panoramas. ③ Conservative memory update, which conducts scale alignment and geometric validation, filters invalid predictions, and mitigates geometric drift.

Experiments are conducted on the Holo360D, Replica, and TartanGround datasets, with comparisons against Pano2Room, HY‑World 2.0, Matrix‑3D, and PanoWorld. AMOR achieves better LPIPS and FID, as well as competitive PSNR and SSIM, and obtains promising results on the TartanGround depth task. Ablation studies verify the effectiveness of surface‑guided readout and conservative memory update. A fixed‑prediction replay experiment demonstrates the usefulness of the memory‑fusion strategy.

Critical strengths
  1. It builds a complete generation‑fusion closed‑loop. Instead of decoupling novel‑view synthesis and 3D memory update, it establishes a full pipeline of “memory readout ‑ RGB‑D generation ‑ validated memory update”. It specifically addresses the problem of error accumulation under continuous scene generation, and bears practical significance for the task of incremental roaming over static scenes.
  2. The component design is targeted and supported by reasonably‑designed controlled ablation evidence. The separate single‑channel depth prediction branch avoids depth distortion caused by the VAE. Ablations show that surface‑guided readout outperforms naive point projection. In the fixed‑prediction replay experiment, diffusion generation outputs are fully fixed; only the memory‑fusion policy is modified, yet map precision is improved. This proves that performance gains originate from the memory strategy itself.
Critical weaknesses

Experimental validation is insufficient. Omissions exist across multiple experimental sections, which may fail to fully support the paper’s claims. 

  1. The authors note that baselines have inconsistent output resolutions and comparisons are performed after resizing all outputs to the same dimension, which may introduce unfairness. It would be valuable to conduct validation experiments to quantify the losses brought by resizing, for instance by resizing the authors’ own model outputs to different resolutions. 
  2. Anti‑drift performance via trajectory revisit is validated on only one single scene. Additional quantitative or qualitative results on more scenes are desirable. 
  3. The paper mentions that conservative update improves precision at the cost of reduced completeness. Though this precision‑completeness trade‑off is acknowledged, there lacks analysis quantifying the magnitude of this drop and whether such degradation is acceptable. 
  4. Ablation studies should be supplemented with sensitivity‑analysis experiments for key hyper‑parameters.
Questions for authors
  1. Memory initialization relies on initial depth outputs from MoGe‑2 followed by a scale‑alignment procedure. Please clarify whether this scale‑alignment step depends on ground‑truth (GT) depth from the dataset, or whether GT depth is entirely unnecessary for real‑world deployment. Also please confirm whether all compared baselines have access to equivalent scale‑related inputs. This implementation detail affects real‑world applicability of the method as well as the fairness among baseline comparisons.
  2. The paper acknowledges inconsistent native output resolutions among baselines; comparisons are made after resizing outputs to identical resolution, which may cause unfairness. Please perform validation experiments to quantify the impact introduced by resizing, for example by resizing outputs of your own model to different resolutions, or adopt alternative approaches to verify whether such unfairness severely affects experimental outcomes and whether reported performance rankings still hold. If such experiments cannot be completed, please explicitly state this evaluation limitation in the main text.
  3. Anti‑drift validation via trajectory revisit is performed only on the Hospital scene. Meanwhile, ablations for core‑module effectiveness are conducted on a small subset of merely 9 scenes. Are there revisit results from more scenes or longer trajectories to examine whether errors keep accumulating under successive updates? If additional revisit results are unavailable, please explicitly state the validated scope of this anti‑drift effect in the manuscript to help readers understand the conditions under which this conclusion applies.
  4. As shown in Sec.4.4 of the paper, conservative update improves precision while degrading completeness. The paper acknowledges this precision‑completeness trade‑off, yet experimental analysis quantifying the magnitude of completeness degradation and whether this degradation is acceptable is missing. Please add such analysis.
  5. Please supplement sensitivity‑analysis ablation experiments for key hyper‑parameters.
  6. In the depth estimation evaluation under RGB-only observation (Figure 7b), AMOR achieves better RMSE, while DepthMaster obtains a lower AbsRel. Analysis for this observation should be supplemented.
Reviewer_93MEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AMOR addresses progressive panoramic novel-view synthesis through a joint RGB-D diffusion model conditioned on a persistent geometric memory. It introduces surface-guided memory readout and conservative integration of generated geometry to reduce inconsistent feedback. Evaluation covers panoramic synthesis, depth estimation, memory maintenance, and revisiting previously observed regions, with ablations examining integration and RGB readout.

Critical strengths
  • The method targets a consequential failure mode of progressive generation.
  • The ablations provide useful evidence beyond final image quality.
Critical weaknesses
  • The geometric precision–completeness trade-off is not sufficiently quantified. The conservative policy improves prediction-to-reference accuracy, but the paper acknowledges that Direct achieves a higher F-score. Without corresponding completeness/recall results and comparisons across admission settings, it remains unclear whether the method improves the overall trade-off or primarily selects a smaller, safer subset of geometry.
  • The readout ablation does not fully establish the advantage over competitive point-based alternatives. It compares surface-guided readout against single-pixel point projection using the same model trained with the proposed conditioning. The substantial gain establishes the importance of the deployed readout, but can also reflect improved coverage and an inference-time conditioning distribution shift.
Questions for authors
  • What are completeness, precision, recall, and F-score for Direct and Ours in both sequential generation and fixed-sequence replay? How much does retained geometry differ?
  • How does surface-guided readout compare with a depth-buffered, footprint-aware point renderer? Does the advantage persist when the alternative conditioning is also used during training?
  • AMOR uses different training dataset compared to the baselines with overlap with the evaluation dataset. Does the improvement in Table1 mainly come from this?
Reviewer_1kD8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AMOR progressively constructs a static 3D scene through panoramic RGB-D generation conditioned on geometric memory. Building on Modality Forcing, it jointly generates appearance and scalar depth, renders remembered surfaces into target-aligned conditions, and selectively integrates geometrically compatible predictions. Evaluation covers 32 scenes and 128 novel-view targets across three datasets, alongside depth-estimation and memory experiments. The evaluation uses known camera poses and source ground-truth depth for initial scale calibration. AMOR reports strong perceptual novel-view quality, while conservative updates improve map accuracy and appearance retention at an acknowledged completeness cost.

Critical strengths
  • Consistent perceptual improvements across distinct datasets. AMOR achieves the lowest LPIPS and FID on each of Holo360D, Replica, and TartanGround, with dataset-average LPIPS improving from HY-World 2.0’s 0.3588 to 0.2894 and FID from 87.97 to 66.17 (Table 1). Average PSNR and SSIM also improve, although individual dataset rankings vary. Including all eighteen Replica scenes, which are not used for training, strengthens the evidence beyond evaluation within the training datasets.

  • Useful controls support the memory mechanisms. The readout experiment fixes initial memory, target poses, depth conditions, and sampling seed, and changing RGB readout improves PSNR from 14.36 to 17.14 dB (Table 4). The fixed-prediction replay additionally applies both update policies to identical raw RGB and relative-depth predictions: map accuracy error remains lower for AMOR, 0.533 m versus 0.653 m (Appendix F.1). These experiments provide meaningful evidence beyond differences in generated inputs, although replay retains policy-specific alignment.

Critical weaknesses
  • This paper seems like an extension of Modality Forcing paper. In terms of architecture wise, I do not see any improvement but this is like an engineering effort to adapt Modality Forcing from regular to panorama images.

  • The scene memory updates seems useful but the techniques are not new. Similar video-based world models or camera-controlled video generation methods [1,2] have done this. I think the paper did not discussed those works and more comparisons are need to highlight the differences.

  1. Shen, T., Bahmani, S., He, K., Srinivasan, S. G., Cao, T., Ren, J., ... & Ren, X. (2026). Lyra 2.0: Explorable generative 3d worlds. arXiv preprint arXiv:2604.13036.
  2. Lee, D., Huang, C. H. P., Chen, X., Ye, J. C., Ceylan, D., & Jeong, H. (2026, September). Memory-v2v: Memory-augmented video-to-video diffusion for consistent multi-turn editing. In European Conference on Computer Vision (pp. 75-93). Cham: Springer Nature Switzerland.
Questions for authors
  • Existing-result clarification and limited analysis: Please report completeness, recall/F-score, and admitted geometric coverage for Direct and AMOR in both sequential generation and fixed-prediction replay. Does the accuracy advantage persist at comparable coverage or recall? Please also specify the numerical admission thresholds and scale-support inheritance limit governing this tradeoff.

  • Limited validation request: How does surface-guided readout compare with finite-footprint, depth-buffered point splatting on the existing fixed-memory evaluation, under comparable coverage? Can the contribution of the surface-orientation and depth-continuity checks be distinguished from conditioning density and training-distribution match?

  • Comparison with recent memory-guided video generation method as described above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.