acceptodds
Under review as a conference paper at ICLR 2027

CoMO4D: Temporally Consistent Multi-Object 4D Generation from a Single Monocular Video

Abstract

Generating temporally consistent multi-object 4D data from a monocular video is fundamental to dynamic scene modeling, yet existing methods often struggle to retain semantic object organization and maintain spatial and temporal consistency under occlusion. We propose CoMO4D, a structured 4D generation framework that explicitly represents each persistent semantic object with a single shared canonical geometry and a sequence of frame-wise spatial states, thereby describing both the stable object structure and its time-varying spatial evolution throughout the video. CoMO4D combines monocular depth estimates with semantic cues to construct object-level point-cloud sequences. At its core is a spatiotemporal completion mechanism that integrates semantic-region and inter-object relational reasoning, cross-frame temporal reasoning, and adaptive gated fusion to aggregate complementary observations for canonical geometry generation and consistent frame-wise object-state prediction. We further construct CoScene4D, a CARLA-based 4D dataset for structured semantic, geometric, spatial, and temporal supervision. CoMO4D is trained using CARLA-based CoScene4D together with nuScenes, while quantitative evaluation is conducted on the held-out V2X-Sep benchmark. The resulting performance demonstrates strong cross-domain transfer for temporally consistent multi-object 4D generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Object 4D Generation, Dynamic Scene Modeling, Spatiotemporal Completion, Cross-Domain Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_n3ZROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoMO4D reconstructs dynamic rigid objects from a monocular video using a shared canonical geometry for each tracked object and frame-wise translation, rotation, and scale. The framework combines intra-frame object and semantic-region relations with cross-frame observations of the same object through adaptive gated fusion. The proposed modules are trained while the underlying encoders and decoders remain frozen. The paper also introduces the CARLA-based CoScene4D dataset and evaluates transfer to V2X-Sep, Waymo, and PandaSet.

Critical strengths
  1. The shared canonical geometry provides a useful object-centric representation of dynamic scenes. Combining spatial context with identity-aligned temporal observations is a reasonable approach to recovering object states under partial visibility and occlusion.

2.Evaluation on three external datasets provides evidence of cross-domain transfer. The pose results and module ablations support the benefits of relational and temporal modeling, and the comparison with frame-wise SAM3D shows a substantial reduction in mesh-generation time.

Critical weaknesses

Unclear source of appearance improvements and insufficient implementation details. Figure 5 suggests that CoMO4D produces better object geometry and appearance than SAM3D. However, the source of this improvement is unclear: the geometry and appearance components are frozen, and training uses only translation, rotation, and scale losses. Although improved conditioning could benefit a frozen decoder, the paper does not explain sufficiently how the proposed modules produce these gains or distinguish them from improvements in object placement. Moreover, the sources, checkpoints, and pretraining of the frozen components, as well as several preprocessing choices, are not specified. These omissions make the method and its appearance improvements difficult to assess and reproduce.

Questions for authors
  1. Figure 5 shows improved object appearance over SAM3D. How do the trainable modules affect the frozen geometry and appearance generation components when supervision is limited to object-state losses? Please specify the underlying models and checkpoints, and, if possible, compare appearance with object poses held fixed.

  2. Please introduce the SAM3D Forloop protocol in the main text, correct its reference, and provide the missing CUPID citation. Is CUPID also applied independently to each frame and object, or does it use cross-frame information?

  3. I could not find an AI Use Statement in the submission. Please add this required section in accordance with the ICLR 2027 author guidelines.

Reviewer_a3nDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CoMO4D, an object-centric framework for 4D reconstruction from monocular video. It represents each tracked dynamic rigid object using shared canonical geometry and frame-wise spatial states. The method constructs object-level point clouds from estimated monocular depth and instance masks, and introduces a Relation Module to capture intra-frame object–scene and object–object relationships, alongside a Temporal Module that aggregates observations of the same object across frames. These features are combined through adaptive gated fusion to predict object geometry and spatial states.

The authors also introduce CoScene4D, a CARLA-based dataset providing structured geometric, semantic, spatial, and temporal supervision. The model is trained on CoScene4D and nuScenes and evaluated on three traffic-scene datasets: V2X-Sep, Waymo, and PandaSet. The reported results show improvements in pose estimation and foreground appearance over several baselines, while ablations demonstrate the contributions of the Relation and Temporal Modules.

Critical strengths

1. A reasonable object-centric formulation with complementary relational and temporal modeling.

The proposed representation separates persistent object geometry from time-varying spatial states, which is suitable for reconstructing dynamic rigid objects. The combination of intra-frame spatial relationships and identity-aligned cross-frame observations is well motivated for addressing partial observations and pose ambiguity.

2. Promising pose estimation results.

The method reports improvements in translation, rotation, scale, and relative-translation errors on three held-out traffic datasets. The introduction of CoScene4D also provides structured annotations that may be useful for object-centric dynamic-scene research.

Critical weaknesses

1. Limited methodological novelty and a mismatch between the broad 4D generation framing and the evaluated setting.

Although the paper is framed as multi-object 4D generation, the proposed method primarily performs observation-conditioned reconstruction of tracked rigid objects from monocular videos. Its main methodological components, including object-centric canonical representations, relational attention, cross-frame feature aggregation, and gated fusion, are closely related to existing techniques. While their integration is reasonable, the paper does not sufficiently establish its methodological advantages over prior object-centric 4D reconstruction approaches. Furthermore, all quantitative evaluations are restricted to traffic scenarios, predominantly involving vehicles, whereas several compared baselines address more diverse scenes and object categories. Consequently, the experimental evidence supports a relatively narrow vehicle-centric reconstruction setting rather than the broader scope suggested by the title and introduction.

2. Insufficient specification of the frozen decoders and inadequate validation of the core 4D geometry reconstruction.

The Appearance/Geometry Decoder and Layout Decoder are essential components of the proposed pipeline, yet their architectures, pretrained model sources, training procedures, and checkpoint provenance are not clearly specified or attributed to existing models. In particular, the output representation of the Appearance/Geometry Decoder (e.g., meshes, point clouds, or 3D Gaussians) and its generation process remain unclear, making the framework difficult to reproduce and its methodological contributions difficult to assess. Moreover, both decoders are frozen, while Eq. (15) only supervises translation, rotation, and scale. The quantitative evaluations primarily focus on object states and foreground appearance, without directly measuring canonical geometry accuracy, completeness, or cross-frame geometric consistency. These omissions make it difficult to determine whether the proposed relational–temporal modeling substantially improves 4D reconstruction beyond object-state estimation.

Questions for authors

1. What is the intended scope and methodological novelty of CoMO4D relative to existing 4D reconstruction approaches?

The proposed framework appears closer to observation-conditioned 4D reconstruction than general 4D generation. Could the authors clarify its intended scope and specific methodological advantages over related object-centric approaches such as Robust4DGen and Lift4D? Given that all quantitative evaluations focus on traffic scenarios and rigid vehicles, to what extent can the proposed relational–temporal modeling generalize to other scene types or dynamic object categories?

2. What are the origins and implementations of the frozen decoders, and how is the quality of the resulting 4D geometry validated?

Could the authors specify the architectures, pretrained model sources, training data, and checkpoint provenance of the frozen Appearance/Geometry and Layout Decoders, including appropriate references if they are adopted from existing methods? In particular, what is the exact output representation of the Geometry Decoder, and how is the shared canonical geometry generated and rendered across frames? Furthermore, given that CoScene4D provides object-level geometric ground truth, could the authors report direct geometry reconstruction metrics and temporal consistency evaluations to demonstrate the contributions of the proposed modules beyond pose estimation?

Reviewer_A7EmOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims at multi-object 4D reconstruction of rigid vehicles such as cars from 8 frame monocular clips. Preprocessing provides depth, instance masks, tracks and camera parameters. The method predicts a canonical rigid shape plus 6 DoF pose and scale per frame. Two branches process per-object tokens: a relation module and a temporal module. The two streams are then fused and processed, frozen pretrained decoders. The method is trained on a CARLA dataset and nuScenes.

Critical strengths

Using temporal information to reconstruct rigid objects from a video is sensible. The proposed pipeline is a reasonable approach to extend a non-temporal model with this capability.

The evaluation tests generalization across datasets and the results show consistent improvements.

Critical weaknesses

The paper seems to be compiled in camera-ready mode, which might have an effect on the page limit.

It is unclear what the frozen base model is. The visual encoder, point encoder, fusion, feature backbone, geometry decoder, appearance decoder and layout decoder are all frozen, but it is never stated what the pretrained model is. This is very important to show a comparison to the base model and to allow for reproducibility and fair comparisons to prior work. From the experiments, an uncertain guess is that CoMO4D is SAM 3D Objects plus two trained attention modules, but it is highly unclear.

The results in the ablation table do not match any other results in the paper. It is unclear what dataset was used and if and how the setting is different from Tables 1 and 2 because no results match across tables. This means the ablation might not fully support the claims.

The title and introduction make a point out of "canonical geometry" and "4D reconstruction", but those are not evaluated in the experiments. I would expect so geometric evaluation of the object reconstruction (eg Chamfer distance, 3D IoU, etc) to validate this claim; otherwise it is diffuclt to know if the method actually succeds in this task.

It is unclear how the inputs to the method are derived. Instance masks, tracks, camera parameters: are they GT inputs, or are they estimated by off-the-shelf methods?

Missing comparisons. A highly related task is monocular 3D object detection and tracking. Methods such as

  • Hu et al., Joint Monocular 3D Vehicle Detection and Tracking, ICCV 2019
  • Wang et al., FCOS3D, NeurIPS 2020 NuScenes 3D detection challenge winner
  • Brazil et al., Omni3D, CVPR 2023

The idea of canonical object plus per frame pose also appears in papers such as

  • Yan et al. Street Gaussians, ECCV 2024
  • Ost et al. Neural Scene Graphs, CVPR 2021
Questions for authors

There are many open questions in the above weaknesses:

  • What is the base model? And what is the raw performance of that model?
  • How are the inputs derived?
  • Why are the results in the ablation study different?
  • Results are "averaged over 3 runs" but no standard deviations are shown. Why?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.