acceptodds
Under review as a conference paper at ICLR 2027

Scene Flow Consistent and Depth-Aware Temporal Regularization for 4D Gaussian Splatting

Abstract

3DGS represents scenes using Gaussian primitives, making it well-suited for a wide range of static scenarios. However, achieving high-quality dynamic video synthesis and physically realistic rendering requires accurate 3D motion modeling and temporal consistency to prevent physically implausible output across time. To address these challenges, we introduce PhysScene4DGS, a physics-based algorithm for 4D dynamic scene generation and rendering. By jointly optimizing 3D scene flow consistency and temporal coherence, PhysScene4DGS enables the physically realistic rendering of dynamic objects. PhysScene4DGS begins by computing 3D scene flow using depth and optical flow. Then, we introduce 4D temporal consistency regularization, which penalizes velocity discrepancies through inverse normalization between velocity and depth, promoting smooth and continuous motion. Furthermore, the scene flow consistency regularization aligns the predicted 3D motion with observed motion derived from back-projecting depth and optical flow, reinforcing physical accuracy. Experimental results demonstrate that PhysScene4DGS ensures both physical plausibility and visual realism in the real-time rendering of complex dynamic scenes, while maintaining comparable training time and FPS.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel View Synthesis, Dynamic Scene, Gaussian Splatting, 4D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XAEVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PhysScene4DGS, a dynamic 3D Gaussian splash framework that incorporates scene flow consistency and depth-aware temporal regularization. The former utilizes depth and optical flow constraints to predict 3D motion, while the latter enhances the temporal smoothness of motion. The authors conducted experiments on the Plenoptic Video, D-NeRF, and HyperNeRF datasets, reporting improved rendering quality and computational efficiency, and claim that the method enhances the physical consistency of motion.

Critical strengths

This paper investigates the issue of motion consistency in dynamic Gaussian splatter, a problem of practical research value in the field of dynamic scene reconstruction. Combining depth information with optical flow to supervise the 3D motion of objects is a reasonable and well-motivated design approach.

The improvements in rendering quality achieved by this method on three benchmark datasets are encouraging, especially considering the efficiency advantages in training time and inference speed emphasized by the authors. If these experimental results can be validated under a unified and consistent evaluation protocol, it will provide more compelling evidence of the method’s effectiveness.

Critical weaknesses

Equation (12) achieves regularization by constraining changes in the ratio of velocity to depth, but this does not necessarily guarantee that three-dimensional motion conforms to physical laws. Even if an object maintains a constant velocity, this loss function may still impose a penalty as long as the depth changes. The paper lacks rigorous theoretical derivations and does not compare the method with traditional velocity- or acceleration-based regularization approaches; therefore, its core conclusion regarding physical consistency lacks sufficient support.

The experiments focus primarily on rendering quality and lack a direct assessment of the accuracy of real-world motion. Key implementation details regarding depth and optical flow supervision are unclear, and the ablation experiments lack comparisons with simpler methods. Furthermore, the results for RealTime4DGS in Table 1 and Table 2 are identical, and the statement regarding training time in Table 1 does not match the actual data. These issues undermine the credibility of the experimental conclusions.

Questions for authors

Q1. What theoretical foundation underpins Equation (12)? Why should a constant velocity‑to‑depth ratio correspond to physically‑valid 3D motion? Please elaborate on the underlying assumptions. Under matched experimental conditions, please compare the proposed loss against conventional velocity‑ and acceleration‑based regularizers, and supply quantitative motion error metrics.

Q2. How are depth maps, optical flow, and foreground masks generated? How do the authors link pixel‑wise observations to individual Gaussian primitives, and how are camera transforms and occlusion cases dealt with? Please lay out the full supervision pipeline, and clarify whether competing baseline approaches are given access to identical auxiliary inputs.

Q3. The paper makes claims of physically consistent motion, yet evaluations mostly rely on image‑quality metrics. Where ground‑truth references exist, could the authors report 3D scene‑flow error, trajectory consistency metrics or temporal motion error? Challenging test scenarios including fast‑moving objects, varying depth and occlusions should be covered. Without these measurements, it is hard to tell whether the reported gains stem from genuine physical‑motion improvements or merely improved image‑space fitting.

Q4. What core distinctions set this method apart from existing dynamic Gaussian approaches equipped with motion priors and temporal regularizers, especially ST‑4DGS and ReMatching? Please spell out the new algorithmic contributions, and offer controlled comparisons if possible. Head‑to‑head tests against standard motion regularizers would help justify the necessity of the proposed formulation.

Q5. RealTime4DGS shares identical numerical results across Table 1 and Table 2 even though the two tables correspond to different datasets. In addition, Table 1 states that the proposed method achieves the shortest training time, while several baseline methods actually run faster. Per‑scene results, baseline code sources, full evaluation configurations, and the computational cost of each proposed module should be provided. Given the large PSNR improvements reported, these details are critical for result verification.

Q6. The submitted manuscript seems to lack the AI Use Statement mandated by the ICLR 2027 author instructions. Could the authors confirm whether this required statement has been included somewhere within their submission package?

2 comments
AuthorsOpenReview ↗

Theoretical Foundations, Experimental Validation, and Implementation Clarifications

1. Theoretical Motivation and Adaptive Weighting

Equation (12) characterizes the relative variations between velocity and depth. As clarified in Lines 304–305 of the manuscript, this formulation introduces a novel inverse-depth weighting strategy that enables faster convergence toward the optimum compared with naive grid search. Under constant-velocity motion, the 4D Temporal Consistency Regularization remains constant. When abnormal motion fluctuations occur, the weighting coefficient of this regularization term should be increased to suppress such variations. Accordingly, as shown in Equation (13), we introduce a tunable hyperparameter to adaptively control the contribution of this loss term.

2. Rigorous and Comprehensive Experiments

All our experimental results are reliable and valid, as supported by the findings reported in 4D Rotor-GS. As shown in Table 2, PhysScene4DGS achieves substantially higher computational efficiency than Deformable3DGS. Therefore, when evaluated under an equivalent computational budget, our approach outperforms all Gaussian-based baselines in terms of PSNR. We not only present quantitative ablation results in Table 4 to demonstrate the impact of individual components of PhysScene4DGS on the D-NeRF dataset, but also provide qualitative ablation comparisons on the Plenoptic Video dataset in Figure 5. Furthermore, Figure 6 provides additional evidence of our method's effectiveness by illustrating improvements in optical flow consistency.

3. Direct evaluation of temporal and geometric behavior

Our primary goal is real-time dynamic novel-view synthesis. Reconstruction metrics evaluate agreement with observed images, including the appearance of moving surfaces, textures, illumination, and reflections. Together with the qualitative examples, these results support the dynamic rendering benefits of our regularization framework. Optical flow is obtained by estimating pixel-wise motion between frames, while depth maps are directly generated through the Gaussian rendering pipeline to capture the underlying 3D scene geometry. We have carefully reviewed the manuscript and corrected all numerical issues.

Duan, Yuanxing, et al. "4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes." ACM SIGGRAPH 2024 conference papers. 2024.

4. ReMatching is not a dynamic rendering method.

5. We will include the AI Use Statement in the camera-ready version of the manuscript.

Thank you for the authors' response and clarifications. However, several of my main concerns remain insufficiently addressed. First, the response does not provide a convincing theoretical justification for Eq. (12), particularly why the proposed velocity-to-depth regularization should ensure physically plausible 3D motion. Second, the claimed improvements in physical motion consistency are still not supported by direct quantitative evaluations or comparisons with conventional motion regularizers. Furthermore, the methodological distinctions from related work remain unclear, and important details of the scene-flow supervision pipeline are still missing. Although the authors state that the numerical issues have been corrected, no verifiable revised results or detailed explanations are provided in the response. Therefore, I believe the current evidence remains insufficient to support the paper's central claims, and I maintain my original rating of 2 (Weak Rejection).

Reviewer_ShT9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PhysScene4DGS, a 4D Gaussian Splatting framework for dynamic scene rendering. It introduces two main technical components: (1) 4D scene flow consistency regularization, which estimates per-Gaussian motion by fusing depth and optical flow to enforce alignment between predicted and observed motion; and (2) 4D temporal consistency regularization, which normalizes velocity by depth to suppress motion discontinuities across time. Experiments are reported on standard dynamic scene benchmarks. However, the credibility of the reported results is in serious doubt: in Figs. 2 and 3, the visual outputs labeled "Ours" appear to be identical to the ground truth images, which raises significant concerns about research integrity.

Critical strengths

None

Critical weaknesses
  1. The most recent baseline, Deformable4DGS, is from 2024, suggesting the comparison may be outdated relative to the current state of the art.
  2. In Fig. 2 and Fig. 3, the visual results shown under "Ours" appear to be identical to the ground truth images. If this is not a rendering or formatting artifact, it raises serious concerns about research integrity and renders the reported results unverifiable.
Questions for authors

Please clarify whether the "Ours" results in Fig. 2 and Fig. 3 are indeed identical to the ground truth images, and if so, provide an explanation. This is critical to the credibility of the paper's experimental evaluation.

1 comment
AuthorsOpenReview ↗

PhysMotion4DGS: Experimental Validation and Research Integrity Clarification

1. Comprehensive Comparisons with Diverse Dynamic Rendering Methods

Existing experiments compare PhysMotion4DGS with 15 distinct dynamic rendering methods across the Plenoptic Video and D-NeRF datasets. These comparisons cover time-conditioned neural radiance fields, deformation-based reconstruction, explicit voxel representations, factorized and hash-based spatiotemporal representations, streaming reconstruction, and dynamic Gaussian splatting. Notably, Deformable4DGS uses a deformation field, and RealTime4DGS uses native 4D Gaussian primitives. The selection therefore provides broad coverage of established approaches to modeling scene dynamics and balancing rendering quality with computational efficiency.

2. Clarification of Visual Results and Research Integrity

We take the reviewer's concerns regarding research integrity very seriously. However, we respectfully disagree that the apparent visual similarity between our rendered results and the ground-truth images constitutes evidence of data fabrication or result manipulation.

After verification, confirm that all images labeled Ours in Figures 2 and 3 were generated independently by the proposed rendering pipeline, without substituting ground-truth images.

High visual similarity is an expected objective of high-quality novel-view synthesis. Importantly, visual resemblance alone is insufficient to establish that two images are identical, let alone to substantiate an allegation of research misconduct.

We respectfully request that the reviewer and Area Chair evaluate this concern based on objective, reproducible evidence rather than visual impressions alone. Given the seriousness of research-integrity allegations, we believe that any such concerns should be supported by verifiable evidence and assessed through the conference's established procedures.

We remain fully committed to research integrity, experimental transparency, and reproducibility.

Reviewer_RvzqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PhysScene4DGS, a physics-based algorithm for 4D dynamic scene generation and rendering. By jointly optimizing 3D scene flow consistency and temporal coherence, PhysScene4DGS enables the physically realistic rendering of dynamic objects. That said, this paper is poorly structured and contains mistakes. It seems that there is a repetition of paragraphs in the introduction section. The contributions are not well-structured. The related work section includes redundant and unrelated papers, and it also lacks relevant work. The method is quite simple and lacks novelty. The experimental results seem to have limited quality and fidelity.

Critical strengths
  • The method is simple to implement and integrate. Fundamentally, the approach attaches two loss terms to 4DGS. It incurs no extra overhead during inference.
  • The motivation is understandable. The paper supervises the 3D scene flow using optical flow and depth, and further introduces a depth-weighted smoothness term to suppress temporal jitter.
Critical weaknesses
  • The paper suffers from serious writing issues. In the introduction, it only cites works from the static 3DGS family, many of which are not SOTA. And I suspect two paragraphs are redundant: the sections on combining depth and optical flow features and consistency regularization.
  • The contributions fail to provide a structured summary of the paper’s novelties. In Related Work, the authors list numerous redundant references with inconsistent formatting. Also, they are far more relevant to this work.
  • The proposed method lacks sufficient novelty. The scene flow and temporal consistency described in the paper are essentially two loss functions, with no architectural innovations whatsoever.
  • I also doubt the results listed in the experiments section. Deformable4DGS is actually "4D Gaussian splatting for real-time dynamic scene rendering", which is called 4D-GS in the original paper. DyNeRF mentioned that 1.3K GPU hours with 8 V100 GPUs, that is to say, about a week. But the paper said using a single NVIDIA RTX 3090 GPU. Furthermore, I didn't see any visual differences between the GT and ours. They are too similar, almost the same.
Questions for authors
  • How is vi(t) explicitly defined? How are the optical flow and depth obtained?
  • Since you treat "robustness against noise" as one of the challenges, could you demonstrate how your method degrades when the flow or depth contains noise or suffers from misalignment?
  • What exactly is "physical" within these two terms? The scene flow term uses supervised alignment, while the temporal term is an inverse-depth-weighted smoothness prior.
1 comment
AuthorsOpenReview ↗

PhysScene4DGS: Methodological Innovations and Experimental Reliability

1. Key Technical Contributions of PhysScene4DGS

We respectfully disagree that PhysScene4DGS merely introduces two additional functions. Our framework presents four interconnected technical contributions to physically plausible dynamic scene rendering:

(1) 4D Physics-Aware Scene Flow Consistency Regularization. We introduce a unified motion-consistency formulation that jointly incorporates 3D velocity, depth information, and optical flow to constrain Gaussian dynamics in four-dimensional space. By explicitly aligning 3D motion with observed inter-frame motion, this formulation enhances the physical plausibility and consistency of dynamic scene representations.

(2) Physics-Motivated 4D Temporal Consistency Regularization. We develop a depth-aware temporal regularization strategy that incorporates inverse-depth weighting into velocity-consistency constraints over continuous time intervals. This formulation aims to suppress temporal motion discontinuities and improve the stability and coherence of reconstructed dynamic scenes.

(3) Depth–Optical Flow Fusion for 3D Scene Flow Modeling. We explicitly incorporate scene flow as a motion representation by integrating depth with optical flow. This enables the recovery of three-dimensional motion cues from two-dimensional observations, providing more informative supervision for modeling complex object dynamics and maintaining consistent rendering across time.

(4) Efficient Dynamic Optimization of 4D Gaussian Properties. We propose an optimization strategy that progressively refines the spatiotemporal properties of Gaussian primitives to capture dynamically evolving scene structures. Importantly, this refinement improves the consistency and physical plausibility of dynamic representations without increasing the number of Gaussian primitives, thereby preserving computational efficiency.

These contributions establish a unified framework that integrates explicit 3D motion modeling, physics-motivated spatial and temporal regularization, multimodal motion supervision, and efficient Gaussian optimization for dynamic scene rendering.

2. Experimental Reliability

The credibility of our experimental findings is further corroborated by the results presented in 4D Rotor-GS. Achieving close visual agreement with reference images is a fundamental goal of high-fidelity novel-view synthesis. Nevertheless, perceptual similarity should not be conflated with pixel-level equivalence, nor does it provide a sufficient basis for questioning the authenticity of the reported results or suggesting potential research misconduct.

Duan, Yuanxing, et al. "4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes." ACM SIGGRAPH 2024 conference papers. 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.