acceptodds
Withdrawn from ICLR 2027

SCENORBIT: Grounding Scene Programs for Consistent Multiview Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A controllable world generator must turn a requested scene into mutually consistent observations across cameras and time. Current systems distribute this responsibility across language interfaces, geometric controllers, and visual generators, leaving the agreement between them to emerge indirectly. We introduce SCENORBIT, which treats a structured scene program as a common reference for these components. A language agent specifies the scene, a geometric renderer grounds its entities in each camera, and a visual critic supervises their agreement. Shared compression and positional binding connect the rendered controls to the video representation before generation. This design supports precise scene editing while preserving the relationships between views. On nuScenes, the reported evaluation achieves 0.132 epipolar photometric error and 21.55 BEV detection mAP, with 32-frame, six-camera generation in 9.6 seconds. Training a detector on 20,000 generated clips yields 44.0 NDS, while combining real and generated data reaches 45.7 NDS. Component studies connect these gains to geometric grounding, aligned conditioning, and critic feedback.

Primary area:
generative models
Keywords:
multiview video generation, world models, structured scene programs, controllable generation, latent representation, geometric grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LS9mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes using scene programs for multi-view driving video generation. Given a text prompt, a Director first generates a world script that describes the global scene state, ego trajectory, entity list, map information, and other scene attributes. A Cartographer then projects the 3D scene into a 2D layout for each camera view. The resulting layouts, together with the geometric information, are compressed using Camera Interleaving and a Shared 3D VAE, after which the model generates the final multi-view videos.

Critical strengths
  1. Explicit 3D scene modeling helps enhance consistency across multiple views.
  2. The scene program partially disentangles the 3D scene representation, providing strong editability and interpretability.
Critical weaknesses
  1. When the Director generates the world script, how does it ensure that the generated trajectories are consistent with the text description? For example, if the prompt specifies that the ego vehicle or a target vehicle should make a turn, how is the generated trajectory constrained to reflect this turning behavior? In addition, how does the method ensure that the generated scene entities cover all entities present in the reference image without omissions?

  2. VistaGEN [1] also introduces an evaluator to assess generated multi-view driving videos and refine them accordingly. What are the advantages and novel aspects of the proposed Auditor compared with VistaGEN’s evaluator?

  3. In Table 2, the results of different methods are reported using different numbers of frames and different resolutions. I do not think these results are directly comparable, because metrics such as FID and FVD are sensitive to these factors. The authors should standardize the evaluation settings before comparison. For methods generating fewer frames, the resolution should at least be matched; for methods generating more frames, the videos should be temporally sampled so that both the resolution and sequence length are aligned.

  4. Most prior multi-view driving video generation methods (such as [2,3,4]) use cross-view attention to enable information exchange across views. Why does this work instead adopt Camera Interleaving together with 3D convolutions for cross-view interaction, rather than directly using cross-view attention?

  5. RGB and geometry use a shared VAE and the same positional indices. However, since the VAE performs both spatial and temporal compression, how does the method ensure that RGB and geometry remain accurately aligned in latent space? In other words, how is it guaranteed that corresponding tokens from the two modalities actually encode the same spatial regions?

  6. For View-Time Permutation, does the ordering of cameras within the same time step affect the results? Is the camera order fixed, or can it be arbitrarily permuted?

  7. At the beginning of Auditor training, draft videos are used for critic training. How exactly are these draft videos generated?

  8. The paper lacks a comparison with Cosmos-Predict2.5/Auto/Multiview [2] (the post-trained version for autonomous driving), which should be included as a baseline.

[1] Chen L H, Cheng K, Liu Y, et al. VistaGEN: Consistent Driving Video Generation with Fine-Grained Control Using Multiview Visual-Language Reasoning[J]. arXiv preprint arXiv:2603.28353, 2026.

[2] Ali A, Bai J, Bala M, et al. World simulation with video foundation models for physical ai[J]. arXiv preprint arXiv:2511.00062, 2025.

[3] Gao R, Chen K, Xiao B, et al. Magicdrive-v2: High-resolution long video generation for autonomous driving with adaptive control[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 28135-28144.

[4] Chen R, Wu Z, Liu Y, et al. Unimlvg: Unified framework for multi-view long video generation with comprehensive control capabilities for autonomous driving[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 25453-25463.

Questions for authors

Please see the "Critical Weaknesses" section

Reviewer_S8AMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCENORBIT is a scene programming interface that parses language instructions into structured scenes, renders them into calibrated geometric layouts, and allows the layouts and RGB tokens to share the same set of view-time serialization and patch coordinates, thereby achieving accurate geometric grounding and cross-camera consistency in multi-view video generation.

Critical strengths

The paper is well organized and clearly written. The authors also provide detailed experimental analyses to validate the proposed method.

Critical weaknesses
  1. Limited evaluation scope. All experiments are conducted on nuScenes at 1280×880, with six cameras and 32 frames. There is no validation on other datasets, resolutions, camera configurations, or frame lengths, so the generality of the method remains unclear.

  2. Outdated baselines. The paper does not compare against recent methods; among the listed baselines, only one was published in 2025. The comparison should include more recent multi-view driving video generation methods to better position the contribution.

  3. Training cost not reported. The method requires 800k training iterations, but the paper does not report overall training cost, such as GPU-hours, memory usage, or data scale. This makes it difficult to assess the practical cost of the approach.

Questions for authors

Please see the weaknesses.

Reviewer_YRVSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a controllable world generator based on a structured scene program that serves as a common reference for the components of LLMs, geometric controllers, and visual generators. However, the explanation is incoherent about what the exact input and output are, why each component is needed, how training is integrated into the agentic design and why.

The abstract contains claims not backed by the paper (support for scene editing which is not discussed anywhere), and an improvement in a non-established metric presented only in appendix (epipolar photometric error, later called EPC). 3 out of 4 claimed contributions describe an elaborate technical designs without implications from it, which are not scientific findings on their own. The paper introduces formulas with many unexplained variables and heavy notation, uses extensive jargon that obstructs the meaning (Sec. 3, 4), and jumps into elaborate technical details without any motivation.

The results are hard to interpret:

  • Quantitative evaluation involves more custom, unintroduced metrics (e.g. PC, OFC).
  • Qualitative Figures 2, 3 show only 1 baseline, don’t elaborate on the setup and show key frame inputs not mentioned in the main text.

Overall, I suggest rejection due to unclear scientific contribution and missing crucial clarifications at all levels: motivation, presentation (text, figures), and results.

Critical strengths

In theory, targets a relevant problem of controllable world generation, but it's far from enough for acceptance. The very low clarity makes potential strengths hard to identify.

Critical weaknesses

Missing critical method details and weak presentation.

  • Inputs, outputs and contributions are unclear. Table 1 lists the Director’s input as “Request, optional reference”, which could be any modality.
  • Custom jargon is used heavily without explanation (e.g. positional binding, physical instant, accidental transition, stride-aware support propagation, geometric attention odds, intermediate grounding, “sparse geometry fixes support”, etc).
  • Core Equations 1, 2, 3 use undefined symbols (q,Π,Eϕ,zgeo,m(q,q’),δ,Wδ,x~,yqq, Π, E_{\phi}, z^{geo}, m(q, q’), \delta, W_{\delta}, \tilde{x}, y_q, the kernel support K\mathcal{K} which is said to enforce unexplained causal ordering, etc), and unexplained operators (Eq. 3). Eq. 6 uses undefined losses.
  • Method Fig.1 contains components never introduced, e.g. View-Block-Aware Padding and World Script Json.

Problematic results.

  • Metrics such as EPC, PC and OFC are never explained in the main text.
  • Qualitative results are very difficult to interpret. Fig. 2 is advertised as “illustrating the appearance and structural agreement assessed by EPC/OFC", then compares to a single baseline and claims its “photometric drift and ghosting”. However, the results between models look like an incomparable, different scene. The input and the expected result from each angle are not clarified. Fig. 3 illustrates key frame inputs never mentioned in the method Sec. 3.
Questions for authors
  • Please outline the pipeline in Fig. 1 in plain notation: inputs and outputs of each component at training and inference, and the reasoning behind each design.
  • Please define missing notations, variables, and metrics in the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.