acceptodds
Under review as a conference paper at ICLR 2027

Physically Plausible and Temporally Coherent Motion in 4D Gaussian Splatting

Abstract

Synthesizing novel views of dynamic scenes is challenging in computer vision. Without explicit constraints between Gaussian distributions and motion, 3DGS struggles to produce physically realistic renderings, a limitation amplified in 4DGS. The temporal dimension increases complexity and leads to geometric degradation. Although higher point cloud density can improve accuracy, it notably increases computational cost. To achieve physically consistent and temporally coherent rendering of dynamic scenes, we introduce PhysMotion4DGS, a physics-aware 4D Gaussian Splatting framework. It introduces 4D physical realism consistency regularization, which constrains the motion velocity of Gaussian distributions using cues from both image space and 3D physical space. Furthermore, PhysMotion4DGS introduces temporal consistency regularization to ensure smooth, continuous motion over time, preventing unnatural dynamics. To capture both global structure and fine details, we adopt a progressive multi-scale training strategy that optimizes Gaussian distributions in stages, achieving high-quality rendering without increasing point cloud size. Experimental results show that PhysMotion4DGS surpasses previous methods in the stability and realism of motion representation in dynamic scenes with comparable rendering speeds and training time.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel View Synthesis, Dynamic Scene, Gaussian Splatting, 4D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dMuROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PhysMotion4DGS for dynamic scene reconstruction and novel-view synthesis. It constrains Gaussian motion by matching projected 3D velocities to observed optical flow and matching 3D velocities to estimates recovered using flow and depth. Temporal regularization encourages smooth motion, while progressive coarse-to-fine training aims to improve reconstruction and efficiency. Experiments on Plenoptic Video and D-NeRF report 34.00 and 37.00 dB PSNR, with rendering speeds of 45 and 120 FPS, respectively. Component ablations are provided on D-NeRF.

Critical strengths
  1. Combining image-space motion matching, depth-assisted 3D constraints, and temporal regularization is a reasonable way to reduce motion ambiguity and improve stability. This gives the method a clear practical motivation.
  2. Table 3 shows that each regularizer improves D-NeRF reconstruction over the stated base model. These results support further investigation of the combined supervision, although the comparative evidence requires clarification.
Critical weaknesses
  1. The comparative results do not yet establish the claimed performance advantage. Several Plenoptic Video baseline values are substantially below published results: Table 1 reports 28.42 dB for Wu's 4D-GS, 29.95 for RealTime4DGS, and 30.43 for STG, whereas the corresponding papers report 31.15, 32.01, and 32.05 dB (Wu et al., Table 3, Yang et al., Table 1, Li et al., Table 1). Different protocols could explain some discrepancies, but their origin is not documented. There are also internal contradictions: Deformable3DGS exceeds the proposed method in D-NeRF PSNR (39.31 versus 37.00), and RealTime4DGS exceeds its Plenoptic rendering speed (72.80 versus 45.00 FPS), despite broader superiority claims. Moreover, ablations are limited to D-NeRF, leaving the large Plenoptic gain unexplained. These issues directly affect the paper's principal evidence for significance.

  2. The additional contribution over related motion-supervision methods is insufficiently established. GaussianFlow (2024) already connects Gaussian dynamics to image-space optical flow and evaluates this supervision on Plenoptic Video, but is neither discussed nor compared. Later relevant baselines, such as E-D3DGS and Grid4D, are also absent. The paper needs to distinguish the contribution of its depth-assisted velocity constraint and training strategy from existing flow supervision.

Questions for authors
  1. Evaluation clarification: Where do the Table 1 baseline results come from, and why do they differ substantially from the published values above? Please specify the scene list, frame range, held-out camera, resolution, training configuration, and metric implementation, and reconcile the superiority claims with Tables 1 and 2.

  2. Contribution clarification: What is the substantive methodological advance over GaussianFlow? A comparison under matched settings, including a flow-supervised version of the same backbone, would materially affect my assessment of novelty and significance.

  3. Assessment-relevant evidence: Can the 34.00 dB Plenoptic result be supported by per-scene results and same-backbone ablations of the 2D constraint, depth-assisted 3D constraint, temporal regularizer, and progressive training? Is there quantitative temporal or motion evidence demonstrating the claimed improvement beyond per-frame reconstruction quality?

2 comments
AuthorsOpenReview ↗

PhysMotion4DGS: Methodological Innovation and Comprehensive Experimental Validation

1. Rigorous and Comprehensive Experiments

All our experimental results are reliable and valid, as supported by the findings reported in 4D Rotor-GS. As shown in Table 2, PhysMotion4DGS achieves substantially higher computational efficiency than Deformable3DGS. Therefore, when evaluated under an equivalent computational budget, our approach outperforms all Gaussian-based baselines in terms of PSNR. The ablation experiments on the D-NeRF dataset sufficiently demonstrate the effectiveness of each component of PhysMotion4DGS. Furthermore, Figure 3 provides qualitative visualizations of different ablation configurations of PhysMotion4DGS on the Plenoptic Video dataset.

Duan, Yuanxing, et al. "4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes." ACM SIGGRAPH 2024 conference papers. 2024.

2. Novelty and Technical Contributions of PhysMotion4DGS

We respectfully disagree that the shared use of optical flow makes PhysMotion4DGS equivalent to GaussianFlow or establishes a lack of novelty. The relevant distinction is the motion formulation and the regularizations imposed on it, rather than whether optical flow is used.

(1) Different motion formulations and supervision targets. GaussianFlow constructs a dense image-space flow from the evolution of projected Gaussian distributions. PhysMotion4DGS instead introduces explicit 3D velocities of Gaussian primitives and constrains them through both image-space projection and depth-supported 3D motion supervision. The distinction is therefore not merely “velocity versus displacement,” but the novel motion representation for four-dimensional space.

(2) Depth-supported 3D velocity supervision is not equivalent to matching a 2D flow field. Our PA term aligns the projected velocity with optical flow. Our PM term additionally constructs a 3D motion target by back-projecting the flow endpoint using the next-frame depth and penalizes its discrepancy from the predicted 3D velocity.

These objectives constrain different quantities. The perspective projection Jacobian maps a 3D velocity to two image-space components and has an unobservable direction along the viewing ray: in camera coordinates. Consequently, image-space flow matching alone cannot distinguish certain different 3D velocities. The depth-supported PM term can constrain this otherwise unobserved component.

GaussianFlow’s covariance-aware 2D transport does not, by itself, provide this explicit depth-lifted 3D velocity target. Thus, PM is not a renaming of its Gaussian-flow loss.

(3) Explicit temporal velocity regularization provides another distinct constraint. PhysMotion4DGS also directly penalizes changes in predicted velocities across temporal intervals, together with a term involving changes in flow-derived velocities. This introduces an explicit temporal prior on velocity evolution. GaussianFlow’s reported formulation supervises Gaussian-induced image-space displacement between frame pairs.

Our contribution lies in the formulation combining explicit 3D velocity modeling, projected 2D motion consistency, depth-lifted 3D velocity consistency, and temporal velocity regularization, integrated into a progressive optimization framework.

Regarding the two additional papers mentioned by the reviewer, we could not locate an accessible record of E-D3DGS, while Grid4D does not report the corresponding evaluation metrics on the Plenoptic Video dataset, especially LPIPS.

I understand the explanation for the lower PSNR values, but my main concerns remain the ablations and compared methods.

Ablations on Plenoptic Video are necessary: D-NeRF is simple, with methods such as Grid4D already exceeding 40+ psnr on most scenes, while 34 psnr on Plenoptic Video is still a strong result. This is worth exploring through ablation studies.

My concern about the baselines is not whether PhysMotion4DGS is equivalent to GaussianFlow or lacks novelty, but that the paper overlooks almost all the works introduced after late 2023. I also could not understand that the absence of LPIPS in Grid4D is a reason to exclude it from comparison. I have provided links to the mentioned papers, although E-D3DGS does not need to be included in the comparison.

Reviewer_xBAcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PhysMotion4DGS, a 4D Gaussian Splatting framework that augments each Gaussian primitive with an explicit 3D velocity and regularizes that velocity with three terms. (i) A 4D physically-aware consistency term (Eq. 1--2) differentiates the pinhole projection of a Gaussian center to obtain a 2D displacement and matches it to an observed optical-flow field. (ii) A 4D physics-motion consistency term (Eq. 3--6) projects the Gaussian at time tt, adds the optical-flow displacement, back-projects the result with the depth at t+δtt+\delta t, and uses the resulting finite-difference velocity vflowv^{\mathrm{flow}} as a 3D supervision target. (iii) A temporal-consistency term (Eq. 7) penalizes the first-order difference of velocity, plus a difference of vflowv^{\mathrm{flow}}. The three terms are combined with weights λ1,λ2,λ3\lambda_1,\lambda_2,\lambda_3 and added to the photometric loss. A coarse-to-fine schedule applies the 2D term first and the 3D term later. Experiments are on the Plenoptic Video and D-NeRF datasets, reporting PSNR/SSIM/LPIPS, training time, and FPS.

Critical strengths

Training takes 10 min and rendering runs at 120 FPS on D-NeRF, and 60 min / 45 FPS on Plenoptic; the supporting engineering (buffer preallocation, stream overlap, footprint culling, SH-order adaptation) is described concretely enough to reproduce.

Critical weaknesses

All three terms are prior art. Eq. 1--2 is GaussianFlow (Gao et al., 2024), uncited, and weaker: it matches one Gaussian's projected displacement to a per-pixel flow field, whereas GaussianFlow splats the full Gaussian dynamics precisely because a pixel's flow is a blend over the many Gaussians covering it. Eq. 4--6 is standard depth-based 2D-to-3D flow lifting; Eq. 7 is a smoothness prior with no physical content, so the "physically plausible" framing is unsupported.

The claims are also contradicted by the paper's own table. Table 2 gives Deformable3DGS 39.31 dB against 37.00 dB here, yet Section 4.2 states the method surpasses all Gaussian-based baselines in PSNR, and Deformable3DGS is never mentioned. The abstract mixes datasets for its three headline numbers: +5.58+5.58 dB from Plenoptic, but −80%-80\% LPIPS and −23.08%-23.08\% time from D-NeRF. The ablation base reaches only 31.97 dB, below every Gaussian baseline in Table 2, so the gains are measured against a weakened implementation.

Finally, no equation relates position to velocity, and no temporal metric is reported -- the central claim is neither defined nor measured.

Questions for authors

How is position obtained from velocity? Please give the equation, state whether v\mathbf{v} is constant over the sequence, and define δv\delta\mathbf{v} from Figure 1.

Where do the optical flow and the depth in Eq. 4 come from, and are their costs included in the reported runtimes?

Table 2 shows Deformable3DGS 2.31 dB above your result. How does that reconcile with the claim in Section 4.2?

The abstract's +5.58 dB, -80% LPIPS and -23.08% time are each measured against a different baseline on a different dataset. Which single controlled comparison supports the headline claim?

1 comment
AuthorsOpenReview ↗

PhysMotion4DGS's Methodological Novelty and Computational Efficiency

1. Novelty and Technical Contributions of PhysMotion4DGS

We respectfully disagree that the shared use of optical flow makes PhysMotion4DGS equivalent to GaussianFlow or establishes a lack of novelty. The relevant distinction is the motion formulation and the regularizations imposed on it, rather than whether optical flow is used.

(1) Different motion formulations and supervision targets. GaussianFlow constructs a dense image-space flow from the evolution of projected Gaussian distributions. PhysMotion4DGS instead introduces explicit 3D velocities of Gaussian primitives and constrains them through both image-space projection and depth-supported 3D motion supervision. The distinction is therefore not merely “velocity versus displacement,” but the novel motion representation for four-dimensional space.

(2) Depth-supported 3D velocity supervision is not equivalent to matching a 2D flow field. Our PA term aligns the projected velocity with optical flow. Our PM term additionally constructs a 3D motion target by back-projecting the flow endpoint using the next-frame depth and penalizes its discrepancy from the predicted 3D velocity.

These objectives constrain different quantities. The perspective projection Jacobian maps a 3D velocity to two image-space components and has an unobservable direction along the viewing ray: in camera coordinates. Consequently, image-space flow matching alone cannot distinguish certain different 3D velocities. The depth-supported PM term can constrain this otherwise unobserved component.

GaussianFlow’s covariance-aware 2D transport does not, by itself, provide this explicit depth-lifted 3D velocity target. Thus, PM is not a renaming of its Gaussian-flow loss.

(3) Explicit temporal velocity regularization provides another distinct constraint PhysMotion4DGS also directly penalizes changes in predicted velocities across temporal intervals, together with a term involving changes in flow-derived velocities. This introduces an explicit temporal prior on velocity evolution. GaussianFlow’s reported formulation supervises Gaussian-induced image-space displacement between frame pairs.

Our contribution lies in the formulation combining explicit 3D velocity modeling, projected 2D motion consistency, depth-lifted 3D velocity consistency, and temporal velocity regularization, integrated into a progressive optimization framework.

Furthermore, the variables associated with each Gaussian primitive have been clearly illustrated in Figure 1.

2. Implementation Details and Efficiency

Optical flow computes inter-frame displacement. Rendered depth maps capture 3D structure in occluded regions and, together with optical flow, complement 2D–3D motion linking. Regularization weights were tuned via a hyperparameter search in [0.001, 0.1]. Moreover, thanks to our hardware-level optimizations, the computational efficiency of our method improves consistently across different scenes, as demonstrated in Tables 1 and 2.

3. Computational Efficiency and Comparison

As shown in Table 2, PhysMotion4DGS achieves substantially higher computational efficiency than Deformable3DGS. Therefore, when evaluated under an equivalent computational budget, our approach outperforms all Gaussian-based baselines in terms of PSNR. We have further clarified this point in the revised manuscript. And, the complete experimental setup and evaluation protocol have already been detailed in the Experiments section.

Reviewer_j6NzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PhysMotion4DGS for dynamic scene reconstruction and novel-view synthesis. It combines optical-flow supervision, depth-assisted 3D velocity consistency, temporal smoothness, and progressive training to improve rendering quality and efficiency.

Critical strengths
  1. Joint supervision in image and 3D space provides a practical way to constrain Gaussian motion and reduce reconstruction ambiguity.
  2. The reported results show a useful balance of rendering quality and speed, while component ablations support the effectiveness of the motion regularizers.
Critical weaknesses
  1. The technical novelty appears limited and incremental: the method combines familiar motion-alignment and smoothness constraints, while comparisons do not clearly establish an advantage over stronger alternatives.
  2. The evaluation mainly measures image quality and does not directly validate the claimed physical plausibility or temporal coherence.
Questions for authors
  1. Could you clarify the novelty relative to existing motion-supervised methods and strengthen comparisons under matched settings?
  2. Could you directly evaluate motion accuracy and temporal consistency to support the central claims?
1 comment
AuthorsOpenReview ↗

Novel Contributions and Extensive Experimental Validation

1. Clarification of Technical Novelty and Experimental Comprehensiveness

PhysMotion4DGS advances dynamic rendering toward physically interpretable motion by jointly constraining Gaussian velocities through image-space observations, depth-assisted 3D motion estimates, and temporal continuity. Its central contribution is a unified framework that makes motion consistency an explicit objective of dynamic scene reconstruction.

  • Unified 4D physical realism consistency regularization. The framework couples image-space motion alignment, 3D velocity consistency, and temporal smoothness to discourage inconsistent Gaussian trajectories and promote coherent motion over time.
  • Explicit alignment between Gaussian velocities and observed optical flow. The 4D physically-aware consistency term projects predicted 3D velocities into image space and aligns the resulting displacements with optical flow, grounding Gaussian motion in observed scene movement.
  • Depth-assisted 3D motion supervision. The 4D physics motion consistency term combines optical flow with depth to estimate 3D velocities and aligns the model’s predicted velocities with these estimates. This provides additional constraints on motion in depth and helps reduce reconstruction ambiguity.
  • Progressive optimization of motion and geometry. A coarse-to-fine training strategy first establishes global motion alignment, then refines local structure and 3D motion consistency. It aims to improve rendering quality and efficiency while controlling Gaussian representation complexity.

Our innovations provide a practical foundation for physically realistic dynamic rendering. The reported rendering improvements support the framework’s effectiveness.

Existing experiments compare PhysMotion4DGS with 15 distinct dynamic rendering methods across the Plenoptic Video and D-NeRF datasets. These comparisons cover time-conditioned neural radiance fields, deformation-based reconstruction, explicit voxel representations, factorized and hash-based spatiotemporal representations, streaming reconstruction, and dynamic Gaussian splatting. The selection therefore provides broad coverage of established approaches to modeling scene dynamics and balancing rendering quality with computational efficiency.

Furthermore, we provide additional experimental evaluations on 17 scenes in the supplementary material, including long-sequence datasets, to further demonstrate the effectiveness and generalization capability of our proposed method across diverse dynamic scenarios.

2. Direct Evaluation of Temporal and Geometric Behavior

Our primary goal is real-time dynamic novel-view synthesis. Reconstruction metrics evaluate agreement with observed images, including the appearance of moving surfaces, textures, illumination, and reflections. Together with the qualitative examples, these results support the dynamic rendering benefits of our regularization framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.