acceptodds
Withdrawn from ICLR 2027

Mono4D: Lifting Single-Image 3D Gaussian Reconstruction to Dynamic Decomposed 4D Scenes from Monocular Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Monocular 4D scene reconstruction is a challenging yet important task for immersive 3D applications. Existing optimization-based methods require multi-view data and costly per-scene optimization, while camera-controlled video generation methods lack explicit 3D representations and demand repeated inference for new camera trajectories; yet recent feedforward 3D reconstruction models perform well on static scenes but suffer from severe geometric and inter-frame inconsistencies when handling dynamic scenes. To address these challenges, we introduce Mono4D, a framework built upon a geometry-aligned static–dynamic decoupled reconstruction paradigm that lifts single-image feedforward 3DGS models to monocular videos of dynamic scenes without additional 4D training or per-scene optimization, enabling efficient nearby novel view rendering. Given a monocular video, Mono4D first estimates camera poses and dynamic masks, then aligns frame-wise 3DGS predictions to a shared world coordinate system. Reliable cross-frame background Gaussians are fused into a globally consistent static representation, while dynamic objects are modeled as temporally indexed Gaussian sequences. To enhance static-dynamic separation, we introduce a boundary-aware mask refinement and Gaussian assignment strategy to mitigate static-dynamic leakage and edge artifacts near fine structures. We further propose a spatiotemporal Gaussian compression scheme leveraging spatial sparsity and temporal redundancy through Gaussian pruning, cluster-level motion encoding, and residual coding for a compact 4D scene representation. Extensive experiments on novel view synthesis benchmarks demonstrate that Mono4D achieves superior performance on monocular 4D scene reconstruction compared with existing approaches, in terms of both visual fidelity and inference efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
feedforward gaussian splatting, novel view synthesis, dynamic 4d reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MDqiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper attempted to solve an existing challenging task - 4D reconstruction- but not the first one attempting to address such problems. It is more likely to be a technical work rather than a research paper because it combines a lot of existing work. For example, it uses VGGT-Omega for camera parameter estimation and GMOS for motion segmentation. Why not consider Monst3r? The title is very confusing. In the title, the authors say single-image, but it is a monocular video. I have more concerns about the experiments and baselines, and I discuss these in the following sections.

Critical strengths
  1. The discussion in the paper is clear.
  2. The figure is very clear.
  3. The analysis, motivations, and objectives in the introduction make sense to me.
Critical weaknesses
  1. Table 1 is very confusing. It is misleading to compare the proposed method with other method categories. The proposed method is not the first to attempt to solve these 3 tasks. The authors should compare their proposed method with specific methods instead of general categories.
  2. This paper is a technical project instead of a research paper. The authors integrated a lot of methods into their framework. Therefore, I did not see enough contribution. The authors should rethink the motivation and the objective.
  3. This whole framework combines too many works. I am curious about the latency. In Table 7, I don't see any 4D reconstruction baselines. Besides, I do believe the motivations of several other baselines are not aligned with the authors' motivations. For example, can the proposed methods achieve the same goal with Neoverse? So, I would like to see more latency comparisons if the authors believe the latency is not a problem compared with other 4D reconstruction methods.
  4. The 3 testing datasets are definitely not enough. Each datasets only cover several cases, which cannot support the authors' claims. Davis dataset is a very good evaluation set for concrete demonstration.
  5. The baselines are not enough. The authors should compare their method with more related baselines, instead of doing so as I mentioned in No. 3.
  6. Given that the authors integrated a lot of existing works, it is better to do ablation studies to verify the contribution of each.
Questions for authors

Please see the critical weaknesses.

Reviewer_L9rcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Mono4D assembles a training-free monocular-video-to-4D-Gaussian pipeline from pretrained single-image 3DGS prediction, camera estimation, motion segmentation, video matting, feature matching, and voxel fusion. It creates one fused static background and a temporally indexed dynamic Gaussian sequence, then compresses the latter using opacity/scale pruning, canonical spatial clusters, rigid motion transforms, and residual coding. Experiments on Neural-3D-Video, NVIDIA-Dynamic-Scene, and StereoDataset report strong perceptual reconstruction metrics and a reduction in stored scene size. The method is practically motivated, but several core transformations and the attribution of its gains are insufficiently specified.

Critical strengths
  1. A relevant and usable systems objective. Producing an explicit dynamic representation from a single monocular video without per-scene gradient-based optimization is useful for repeated viewpoint queries. Separating a persistent background from frame-dependent foreground geometry is a reasonable way to address static flicker and unnecessary temporal storage.

  2. Encouraging results across several datasets. Table 2 reports Neural-3D-Video PSNR of 21.30 and LPIPS of 0.143; the NVIDIA evaluation obtains LPIPS of 0.068, although its PSNR is below Instant4D. Table 3 adds another synthetic test setting. The component ablation in Table 6 indicates that decomposition, boundary processing, and cross-frame completion all matter empirically.

  3. Compression is measured against an uncompressed version. Table 5 reduces storage from 767 MB to 152 MB while changing LPIPS from 0.139 to 0.143. This is a useful storage–quality trade-off, distinct from the claim of better reconstruction accuracy.

Critical weaknesses
  1. The coordinate-alignment and assignment pipeline is ambiguous. Section 3.2 describes the Sharp Gaussians as camera-local, but Equation (3) projects their centers using estimated extrinsics before the world-alignment stage. Equation (5) describes a solved Sim(3) registration but then fixes rotation to identity and translation to zero and estimates only a depth-ratio scale. It is unclear whether scale is estimated per frame or for the whole sequence, how corresponding 3D points from the two predictors are obtained, and how errors or degeneracies are handled. These details are necessary to verify that static fusion is geometrically valid.

  2. The proposed mask-refinement mechanism is not closed algorithmically. Equation (10) computes a Gaussian-footprint-integrated soft weight w_i, but the actual static/dynamic assignment in Equation (4) uses a hard threshold at the projected Gaussian center; the text never states how w_i affects assignment, blending, or rendering. Appendix E further uses uniform voxel averages, whereas Section 3.2 describes confidence-weighted attribute fusion. These are not merely omissions of hyperparameters: they make it uncertain which reconstruction procedure produced the reported results.

  3. Contribution attribution and fair comparisons remain uncertain. The pipeline heavily relies on Sharp, VGGT-Ω, GMOS, VideoMaMa, and LoMa, yet the ablations do not isolate the impact of estimated versus oracle camera poses, depth, and motion masks. Several feed-forward and optimization baselines were extended by the authors to dynamic playback, but the details needed to assess implementation parity are limited. The paper's broad capability claims are also too categorical for prior monocular dynamic reconstruction methods. A controlled comparison using the same input poses/masks and view ranges is needed to establish what is novel about this particular composition.

  4. Efficiency and compression claims require more precise accounting. Table 7 reports 603 seconds for a 49-frame video, compared with 495 seconds for MoVieS; this supports relatively fast preprocessing/reconstruction compared with some methods, not real-time 4D construction. Real-time rendering is asserted without a clear FPS/memory benchmark at the requested output resolution. Storage comparisons across methods and datasets should account for all residuals, quantized attributes, cluster motions, metadata, frame counts, and comparable rendered quality.

Questions for authors
  1. Are the covariance transform in Equation (6) and the Kabsch rotation in Appendix F implemented as written? Please give the exact tensor conventions, corrected formulas if needed, and tests verifying that rigid transformations and Gaussian rendering behave correctly under a known scale/rotation.
  2. Please specify, in execution order, how camera-local Sharp Gaussians are projected onto a frame's mask, how their scale is aligned to VGGT-Ω, and whether the scale is frame-dependent. What happens when camera or depth estimates disagree across frames?
  3. Where is the continuous footprint weight w_i from Equation (10) used in the actual implementation? Which of the conflicting uniform versus confidence-weighted voxel-fusion rules produced the tables?
  4. Can you add a controlled component study with ground-truth versus estimated camera poses and motion masks, a parameter-/input-matched alternative fusion strategy, and a breakdown of the 603-second inference time and rendering FPS?
Reviewer_nz3tOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a training-free framework that extends single-image feedforward 3D GS reconstruction to dynamic scenes from monocular videos. Specifically, the method separates static backgrounds and dynamic objects, performs cross-frame geometric alignment and background fusion, refines dynamic masks, and introduces spatialtemporal GS compression. The experimental results show improvements in NVS quality and efficiency over existing methods.

Critical strengths
  • The paper addresses a challenging problem for dynamic 4D reconstruction, without additional training or per-scene optimization.
  • The static-dynamic decomposition improves temporal consistency, while the compression strategy reduces storage and improves efficiency.
Critical weaknesses
  • The proposed framework appears to be engineering improvements, with limited technical contribution. It largely combines existing models, including motion segmentation, mask refinement, and compression. The paper should give clear clarification about their contributions.
  • Limited 4D modeling of dynamic geometry. Dynamic objects are represented through frame-wise Gaussian predictions, with cluster-level rigid transformations and residual encoding mainly introduced for compression. There is no explicit mechanism modelling temporally consistent dynamic geometry or continuous motion, raising concerns about the quality of the reconstructed 4D representation.
  • The performance heavily depends on the pretrained modules, relying on external camera estimation, depth estimation, segmentation, and feature matching. The error propagation among these multiple modules is not fully investigated, which requires ablation studies of their individual contribution and robustness.
  • Insufficient evaluation of geometric consistency and generalization. The experiments primarily focus on nearby novel view synthesis using image-based metrics, without directly evaluating 3D geometry, motion accuracy, or temporal geometric consistency. Moreover, the claimed compatibility with different feedforward 3DGS backbones is not experimentally demonstrated.
  • Efficiency and scalability concerns. Despite avoiding optimization, the complete pipeline requires 603 seconds (Table 7). Its computational overhead and scalability to longer sequences deserve further investigation.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.