acceptodds
Withdrawn from ICLR 2027

OccGSGen: Persistent Gaussian World Generation through Occupancy-Constrained State Updates

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Controllable driving scene generation can provide diverse visual observations for training and evaluating autonomous driving systems. A central challenge is to generate newly revealed regions while maintaining consistent geometry and appearance across cameras and over time. We present OccGSGen, a framework for generating persistent Gaussian worlds from semantic occupancy, camera poses, and tracked object motion. The framework builds a shared 3D scene by integrating appearance from a video diffusion model through incremental updates constrained by occupancy, visibility, and surface correspondence. A persistent surface memory separates appearance from rendering primitives, allowing additional Gaussians to extend surface coverage without regenerating the appearance of known surfaces. We further supervise newly stored appearance through later observations of the same surfaces. Together, these mechanisms allow the generated world to expand while preserving appearance for reuse across views and time. Experiments on nuScenes demonstrate the effectiveness of OccGSGen in generating driving scenes with consistent geometry and persistent appearance.

Primary area:
generative models
Keywords:
3D Scene Generation, 3D Gaussian Splatting, Generative World Models, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pLYpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OccGSGen, a framework that generates a persistent Gaussian world from semantic occupancy, calibrated camera poses, and tracked actor motion, without reference RGB at inference. The saved world uses Gaussians to store geometry and rendering attributes, and an appearance atlas stores surface appearance independently of Gaussian primitives. A deterministic composer applies occupancy-constrained updates; Gaussian acceptance is decided by correspondence, visibility, depth/alpha, and projected support. Static surfaces are queried in world coordinates and actors in object-local coordinates, with appearance read by patch interpolation. Future read supervision trains initial appearance predictions through later atlas reads. An optional video completion module fills unsupported regions in rendered videos without modifying the saved world or continuation history. Experiments use nuScenes and Occ3D occupancy with three forward-facing cameras.

Critical strengths
  1. This work addresses an important and timely problem: controllable Gaussian generation for driving scenes. This is a relevant and practically significant setting, and framing the method around persistent world construction under several conditions, such as camera and ego-motion, is well motivated.
  2. The proposed structure-driven Gaussian acceptance rule is a solid technical contribution. Combining correspondence, visibility, depth/alpha, and projected support gives a principled way to control primitive growth, and the ablation provides useful evidence: removing the support check increases the Gaussian count by 1.41×1.41\times without affect FID/FVD.
  3. The optional video completion module is practically valuable. By filling unsupported regions, including areas outside the finite occupancy volume, it substantially extends the usable range of synthesized camera views. This improves the practical utility of the rendered outputs while aligning with the geometry of the Gaussian world.
Critical weaknesses
  1. The comparison in Table 2 is not well aligned with the generative setting. Most Table 2 baselines are reconstruction-oriented methods. This work is a controllable generative framework. MagicDrive3D appears to be the most direct baseline: it also outputs Gaussians and uses fewer conditions, yet it is compared in Table 1 rather than Table 2. The authors should reposition MagicDrive3D in the novel-view comparison or explicitly justify why it is excluded from Table 2.
  2. The occupancy condition is assumed available, but its acquisition is not discussed. Occupancy is often difficult to obtain in practice. The authors should discuss practical and simple occupancy acquisition options, such as off-the-shelf occupancy predictors or camera/LiDAR-based estimation pipelines, and clarify how such imperfect occupancy would affect the method.
  3. The effectiveness of the generated data is not validated. For controllability, the authors are encouraged to use the benchmark proposed in arXiv:2507.01735 and evaluate control accuracy with perception models. In addition, the paper should provide evidence that training with the synthetic data improves downstream model performance, rather than only reporting rendering/video metrics.
  4. Diversity of generated scenes is not evaluated. The paper reports fidelity and consistency but does not provide diversity results. The authors should add metrics or analyses on scene, appearance, and actor-layout diversity to demonstrate that the model does not merely reproduce limited modes.
Questions for authors
  1. How easily can the proposed framework be extended to other driving datasets such as Waymo? The authors are encouraged to briefly discuss the expected effort and, if possible, provide preliminary evidence of cross-dataset generalization.
Reviewer_fRHROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper constructs a persistent Gaussian driving world from semantic occupancy, camera poses, and tracked actor motion, using occupancy-constrained Gaussian admission, an append-only surface appearance atlas, and future-read supervision. The separation between rendering support and surface appearance is sensible, and the results are promising. However, the central persistence claim is evaluated only after one continuation block, while the main comparisons involve different backbones, controls, completion systems, and protocols.

Critical strengths

S1. The method decouples geometry updates from appearance updates. This limits unbounded growth of the Gaussian set and reduces appearance drift when the same surface is observed again.

S2. The experimental evaluation is broad. The paper compares original-view video generation and novel-view synthesis, and also measures revisit consistency when previously observed regions are seen again.

Critical weaknesses

W1. The evaluation settings of Tables 1 and 2 need to be specified, including how many views and how many frames are used for each reported score.

W2. The central long-term persistence claim is not adequately evaluated. Please report how many blocks the method can update recursively, together with the corresponding metrics.

W3. The definition of revisit in Figure 4 and Table 3 is unclear. Please describe how these revisit trajectories are constructed, and show the full revisit sequence in the visualizations.

Questions for authors

Q1. How sensitive are the results to the fixed camera-update order?

Q2. How many continuation blocks can the model process before the atlas capacity is reached, and what is the observed capacity-failure rate?

Reviewer_NQzBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents OccGSGen, which incrementally constructs a Gaussian world from semantic occupancy, camera poses, and tracked actor motion. It separates Gaussian admission from appearance assignment, reuses surface appearance through an atlas, and trains initial appearance predictions through later memory reads. The formulation is coherent, and the internal ablations support appearance reuse and future-read supervision. My main concern is that the persistence evaluation covers only one continuation block, without numerical coverage statistics or measurements of sustained world growth.

Critical strengths
  • Separating additional rendering support from new appearance assignment addresses a concrete problem in incremental Gaussian construction: an already represented surface may require more primitives without requiring a new appearance. The independent-appearance ablation supports this distinction, increasing revisit RGB error from 0.0021 to 0.044 while holding geometry and opacity fixed.
  • Future-read supervision provides a practical way to train initial appearance predictions for subsequent reuse. Removing this loss worsens raw-render FID/FVD from 24.06/98.45 to 28.71/127.62 at a lateral shift of ±2 m. These results support its usefulness within the proposed system.
Critical weaknesses
  • The incremental value of the proposed appearance-memory design needs stronger evidence. GEN3C incrementally updates a 3D cache for continued generation; Texture-GS separates Gaussian geometry from surface texture; and Self Gradient Forcing uses future losses to supervise historical memory writing. These establish precedents for the broader principles, while the occupancy-constrained surface atlas and its supervision differ in representation and operation. The key experimental question is whether the proposed surface patches and interpolation improve over simpler appearance reuse within the same pipeline. The independent-appearance baseline establishes the benefit of reuse, but leaves this comparison unresolved.
  • The evaluation provides limited evidence for sustained persistence. Appendix B specifies an initial block followed by one continuation block for the ablations. Figure 4 provides short-term qualitative examples, but there is no quantitative analysis of repeated continuation, appearance drift, dynamic-actor persistence, or atlas growth and capacity failures. Section 3.3 limits its preservation property to stored appearance under fixed coordinates, identities, and lookup rules, and distinguishes this from rendered consistency. Whether correspondence, coverage, actor attachment, and rendered consistency remain reliable over extended rollouts is therefore an empirical question that the current evaluation does not resolve.
  • The persistence measurements do not sufficiently characterize the quality–coverage tradeoff. Revisit RGB compares earlier and updated renders on static surfaces with valid colors in both states. It can remain low despite implausible or spatially misplaced appearance, and queries that lose valid rendering support do not enter this error. Appendix B states that coverage, depth conflicts, and retained-query fractions are checked, but their numerical results are not reported. The independent-appearance comparison fixes geometry and opacity, so its consistency improvement is informative; coverage statistics are particularly important for variants that change geometry. The class-grouped distributional future-read loss accommodates multiple plausible appearances for the same controls, but its effect on correspondence-specific fidelity within a generated world remains unmeasured.
  • The main video-quality comparisons do not isolate the benefit of the world-update mechanism. Tables 1 and 2 report completed videos produced with an independent diffusion module. The paper identifies these as final-system comparisons and explains that raw and completed FID/FVD use different evaluation regions, while published baselines retain different inputs and protocols. A comparison using the same prior, controls, and evaluation protocol is still needed to determine the persistent representation's contribution relative to simpler world-update alternatives. As reported in Section 4.5, completion decreases OCC-IoU from 97.1% to 95.3% and increases Cross RGB error from 0.035 to 0.050 on common OCC queries. This quantifies an output-alignment tradeoff; the saved world itself is unchanged.
Questions for authors
  • How does the atlas compare with simpler appearance-reuse strategies, such as nearest-neighbor color transfer, under the same backbone, proposals, training budget, and evaluation protocol? What benefit specifically requires surface patches and interpolation?
  • Can the authors evaluate several successive continuation blocks and repeated revisits, reporting static and dynamic consistency, Gaussian and atlas growth, memory consumption, and capacity failures as functions of rollout length?
  • Can the authors report the coverage, depth-conflict, and retained-query statistics described in Appendix B for every ablation? Can correspondence-based evaluation within each generated world distinguish consistent, plausible surface appearance from consistently misplaced or implausible appearance?
  • Since OCC-IoU and Cross RGB already use common OCC queries, can raw and completed FID/FVD also be evaluated with identical OCC masks and preprocessing? Can a baseline with the same prior and controls isolate the benefit of the persistent-world mechanism separately from video completion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.