- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies dynamic scene reconstruction from monocular video with Gaussian Splatting. The core idea is to consider long-range 2D tracks as measurements of latent 3D motion rather than as the motion states themselves. The proposed RayCo4D has three parts. First, each track gets a 3D query constrained to its camera ray, and this query is softly linked to several shared latent motion nodes. Second, each node holds SE(3) poses and velocities at sparse times. Third, node motion drives the Gaussians via dual-quaternion blending, with a per-Gaussian local residual to capture fine non-rigid deformation. On DyCheck and NVIDIA benchmarks, the method reports the best results on all metrics against six baselines on 4D video reconstruction. Ablations show that removing each component degrades performance, and additional experiments cover motion correspondence (PCK-T) and intermediate-time reconstruction.
Critical strengths
-
The problem framing is sound, and the method design is coherent. Decoupling track identity from motion state targets an issue of track-driven methods, and the continuous-time formulation is well grounded in existing state-estimation theory.
-
The experimental results are convincing. The method performs best on all reported metrics on both benchmarks, and the extra tests probe the claimed benefits.
Critical weaknesses
The paper claims that it improves consistency under rapid motion and incomplete tracks, but it lacks controlled experiments specifically for these problems, such as stratifying by motion speed, dropping or corrupting tracks, or measuring temporal consistency.
The full system also adds many extra modules (Eq. 11) that are not individually ablated, and it is unclear whether baselines receive the same inputs or are reproduced under the same setup.
Questions for authors
What are the number of Gaussians, memory use, and rendering speed? How is the efficiency of the proposed method compared to the baselines?
4 comments
Initial Response to Reviewer 4Nhi (Part 2/2)
W2. Ablations and Baseline Comparisons
Table 1 evaluates four core components. Removing MSD, RCQ, the WNOA factor, and ARM reduces DyCheck mPSNR and NVIDIA PSNR by 1.35/1.33, 1.19/1.22, 1.03/1.17, and 0.18/0.17 dB, respectively. The w/o ARM variant disables both the local residual motion and its associated regularization, while the ARAP term follows the standard motion-scaffold regularization used in MoSca [1].
We also ablate two late-stage auxiliary components in Eq. (11):
- w/o Calibration: Disables camera-pose and depth-scale refinement, together with the associated calibration regularization.
- w/o Appearance Warp: Disables adjacent-frame appearance transport and its corresponding coverage constraint.
All variants retain the core motion modeling components and use the same training budget and evaluation protocol as the full model.
Table R2. Ablation of auxiliary components on DyCheck and NVIDIA.
| Setting | DyCheck mPSNR ↑ | mSSIM ↑ | mLPIPS ↓ | NVIDIA PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|---|---|
| w/o Calibration | 20.83 | 0.740 | 0.245 | 27.83 | 0.870 | 0.048 |
| w/o Appearance Warp | 20.93 | 0.741 | 0.244 | 27.88 | 0.869 | 0.048 |
| RayCo4D (Full) | 20.97 | 0.743 | 0.241 | 27.94 | 0.872 | 0.045 |
Removing Calibration reduces DyCheck mPSNR and NVIDIA PSNR by 0.14/0.11 dB, while removing Appearance Warp reduces them by 0.04/0.06 dB (Table R2). The other metrics also degrade slightly. By comparison, removing MSD, RCQ, or the WNOA factor lowers both PSNR measures by more than 1 dB (Table 1). The two auxiliary components therefore have smaller effects than these core motion modeling components.
For the comparisons with baseline methods in Table 1, we use the official implementations of the baselines with their recommended native preprocessing configurations, preserving their original visual priors and optimization pipelines. SE(3) B-splineGS [2] also uses publicly available implementations or author-provided models with recommended parameter settings for its baseline comparisons.
All methods in these comparisons are evaluated using consistent dataset splits, evaluation resolutions, and metrics (Sec. 4.1 and Appendix D.9). Since the native visual priors are not necessarily identical across methods, these comparisons reflect the reconstruction performance of the respective complete systems.
All RayCo4D core ablations in Table 1 use the same DA3, BootsTAPIR, and RAFT inputs, initialization, and training budget as the full model. Removing MSD, RCQ, and the WNOA factor reduces DyCheck mPSNR to 19.62, 19.78, and 19.94 dB, respectively. These comparisons isolate the contributions of the motion modeling components under fixed input priors.
References
[1] Lei et al. MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds. CVPR 2025.
[2] Zhang et al. Learning Explicit Continuous Motion Representation for Dynamic Gaussian Splatting from Monocular Videos. CVPR 2026.
Q1. Model Size and Efficiency
In Table R3, we report the average number of Gaussians and end-to-end rendering speed (FPS) on DyCheck (2× resolution), measured on an NVIDIA RTX PRO 6000 GPU. RayCo4D's peak training GPU memory usage is 9.12 GiB.
Table R3. Comparison of average number of Gaussians and rendering speed on DyCheck (2× resolution).
| Metric | MoSca | Shape of Motion | OriGS | RiGS | SE(3) B-splineGS | USPLAT4D | RayCo4D |
|---|---|---|---|---|---|---|---|
| Gaussians | 473,311 | 575,724 | 520,699 | 416,258 | 321,680 | 451,459 | 316,452 |
| FPS | 40.7 | 139.3 | 38.7 | 150.9 | 50.2 | 68.4 | 70.3 |
RayCo4D uses 316,452 Gaussians on average, fewer than all six baselines (321,680–575,724), and achieves an end-to-end rendering speed of 70.3 FPS. It is faster than MoSca, OriGS, SE(3) B-splineGS, and USPLAT4D, but slower than Shape of Motion and RiGS (Table R3).
On 200-frame DyCheck sequences, RayCo4D's total runtime is 86 minutes (36 minutes for preprocessing and 50 for training), compared with 88–107 minutes for Shape of Motion, OriGS, RiGS, and SE(3) B-splineGS, whose training stages take 56–77 minutes. MoSca requires 60 minutes, while USPLAT4D requires 80 minutes, including 60 minutes of preprocessing (covering MoSca prior generation) and 20 minutes of training (Appendix D.10, Table 8).
Thanks for the rebuttal
Thank the authors for the detailed rebuttal. I also read the revised paper. My concerns are largely addressed by the added experiments on challenging tracking, ablation, and efficiency. The proposed method is useful considering the number of Gaussians and FPS. I therefore keep my original score of 3: weak accept.
Thank You for Your Feedback
Thank you for reviewing our revised paper and for your thoughtful follow-up. We appreciate your comments on RayCo4D's coherent design and practical efficiency. Your suggestions have also helped us better explain and validate how measurement–state decoupling, WNOA-based continuous-time modeling, and reference-anchored local residuals work together.
Initial Response to Reviewer 4Nhi (Part 1/2)
We thank the reviewer for recognizing our coherent method design, theoretically grounded formulation, and strong experimental results across both benchmarks. We address the concerns and questions below.
W1. Robustness to Incomplete Tracks and Challenging Motion
We evaluate robustness to incomplete tracks on the same seven DyCheck scenes as in
Table 1, retaining RGB supervision in every training frame (Table R1). The original tracks serve as the 0% additional-masking baseline. For each track, we independently select a starting point and progressively extend a contiguous masked segment from that point to obtain nested masks at 20%, 40%, and 60%. RayCo4D, MoSca, and SE(3) B-splineGS use the masked tracks from the start of motion initialization. UnlikeTable 3, which withholds supervision at the same timestamps across all tracks, this experiment introduces track-specific observation gaps.Table R1. Robustness to contiguous track-observation masking on DyCheck.
At 60% masking, RayCo4D's mPSNR and PCK-T decrease by 1.16 dB and 0.095, respectively, compared with 1.81 dB and 0.189 for MoSca, and 1.63 dB and 0.169 for SE(3) B-splineGS. RayCo4D also shows smaller degradation in mSSIM and mLPIPS. These results support robustness to incomplete tracks and are consistent with MSD's use of observations from different visibility intervals to constrain shared motion nodes, together with WNOA's temporal regularization across observation gaps. The results are included in
Sec. 4.3(Table 4) of the revised manuscript, with the experimental protocol detailed inAppendix E.3.For rapid motion and non-rigid deformation,
Fig. 3andAppendix F(Figs. 8–13) provide qualitative comparisons on challenging dynamic scenes, including Breakdance-flare, Parkour, Wheel, and Skating. Compared with the baselines, RayCo4D better preserves structural integrity and reconstruction clarity around fast-moving human silhouettes, rotating object boundaries, and fine-scale details.The qualitative ablations in
Fig. 4illustrate the complementary roles of WNOA and ARM: WNOA encourages temporal continuity in shared motion, while ARM captures fine-scale non-rigid deformations beyond the shared motion representation. On held-out intermediate timestamps (Table 3andAppendix Fig. 5), RayCo4D improves PSNR by 1.11 dB and reduces LPIPS by 25.9% compared with SE(3) B-splineGS.