acceptodds
Under review as a conference paper at ICLR 2027

RayCo4D:Ray-Conditioned Continuous Motion Model for Monocular 4D Dynamic Reconstruction

Abstract

Reconstructing dynamic 3D scenes from monocular video requires inferring motion from noisy and intermittently visible observations. Long-range tracks provide valuable geometric constraints, but integrating their fragmented temporal coverage into shared, continuous motion remains challenging. We introduce RayCo4D, a ray-conditioned continuous motion model for monocular 4D reconstruction that explicitly separates visual measurements from latent 3D motion states. Each track is associated with a reference-time 3D query constrained to its camera ray, and its image and depth observations constrain shared latent motion nodes through local many-to-many associations. Each node maintains an pose and body-frame velocity at sparse support times. A white-noise-on-acceleration process prior couples neighboring states and defines continuous-time interpolation, allowing observations across visibility intervals to jointly constrain the same latent state sequence. The inferred shared motion is transferred to Gaussian primitives through dual-quaternion blending, while reference-anchored local residuals capture fine-scale non-rigid deformation beyond the shared scaffold. Experiments on diverse monocular dynamic scenes show that RayCo4D achieves competitive novel-view synthesis quality, improves structural and temporal consistency under rapid motion and incomplete tracks, and supports continuous-time rendering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Monocular 4D Reconstruction, Dynamic Gaussian Splatting, Ray-Conditioned Motion Estimation, Measurement–State Decoupling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_4NhiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies dynamic scene reconstruction from monocular video with Gaussian Splatting. The core idea is to consider long-range 2D tracks as measurements of latent 3D motion rather than as the motion states themselves. The proposed RayCo4D has three parts. First, each track gets a 3D query constrained to its camera ray, and this query is softly linked to several shared latent motion nodes. Second, each node holds SE(3) poses and velocities at sparse times. Third, node motion drives the Gaussians via dual-quaternion blending, with a per-Gaussian local residual to capture fine non-rigid deformation. On DyCheck and NVIDIA benchmarks, the method reports the best results on all metrics against six baselines on 4D video reconstruction. Ablations show that removing each component degrades performance, and additional experiments cover motion correspondence (PCK-T) and intermediate-time reconstruction.

Critical strengths
  1. The problem framing is sound, and the method design is coherent. Decoupling track identity from motion state targets an issue of track-driven methods, and the continuous-time formulation is well grounded in existing state-estimation theory.

  2. The experimental results are convincing. The method performs best on all reported metrics on both benchmarks, and the extra tests probe the claimed benefits.

Critical weaknesses

The paper claims that it improves consistency under rapid motion and incomplete tracks, but it lacks controlled experiments specifically for these problems, such as stratifying by motion speed, dropping or corrupting tracks, or measuring temporal consistency.

The full system also adds many extra modules (Eq. 11) that are not individually ablated, and it is unclear whether baselines receive the same inputs or are reproduced under the same setup.

Questions for authors

What are the number of Gaussians, memory use, and rendering speed? How is the efficiency of the proposed method compared to the baselines?

4 comments
AuthorsOpenReview ↗

Initial Response to Reviewer 4Nhi (Part 1/2)

We thank the reviewer for recognizing our coherent method design, theoretically grounded formulation, and strong experimental results across both benchmarks. We address the concerns and questions below.

W1. Robustness to Incomplete Tracks and Challenging Motion

We evaluate robustness to incomplete tracks on the same seven DyCheck scenes as in Table 1, retaining RGB supervision in every training frame (Table R1). The original tracks serve as the 0% additional-masking baseline. For each track, we independently select a starting point and progressively extend a contiguous masked segment from that point to obtain nested masks at 20%, 40%, and 60%. RayCo4D, MoSca, and SE(3) B-splineGS use the masked tracks from the start of motion initialization. Unlike Table 3, which withholds supervision at the same timestamps across all tracks, this experiment introduces track-specific observation gaps.

Table R1. Robustness to contiguous track-observation masking on DyCheck.

MethodTrack MaskingmPSNR ↑mSSIM ↑mLPIPS ↓PCK-T ↑
MoSca0%19.860.7020.2980.822
20%19.610.6990.3040.798
40%19.050.6920.3180.749
60%18.050.6790.3430.633
SE(3) B-splineGS0%20.530.7130.2760.832
20%20.300.7100.2810.811
40%19.870.7020.2920.768
60%18.900.6910.3230.663
RayCo4D (Full)0%20.970.7430.2410.840
20%20.790.7410.2430.825
40%20.320.7360.2490.798
60%19.810.7270.2590.745

At 60% masking, RayCo4D's mPSNR and PCK-T decrease by 1.16 dB and 0.095, respectively, compared with 1.81 dB and 0.189 for MoSca, and 1.63 dB and 0.169 for SE(3) B-splineGS. RayCo4D also shows smaller degradation in mSSIM and mLPIPS. These results support robustness to incomplete tracks and are consistent with MSD's use of observations from different visibility intervals to constrain shared motion nodes, together with WNOA's temporal regularization across observation gaps. The results are included in Sec. 4.3 (Table 4) of the revised manuscript, with the experimental protocol detailed in Appendix E.3.

For rapid motion and non-rigid deformation, Fig. 3 and Appendix F (Figs. 8–13) provide qualitative comparisons on challenging dynamic scenes, including Breakdance-flare, Parkour, Wheel, and Skating. Compared with the baselines, RayCo4D better preserves structural integrity and reconstruction clarity around fast-moving human silhouettes, rotating object boundaries, and fine-scale details.

The qualitative ablations in Fig. 4 illustrate the complementary roles of WNOA and ARM: WNOA encourages temporal continuity in shared motion, while ARM captures fine-scale non-rigid deformations beyond the shared motion representation. On held-out intermediate timestamps (Table 3 and Appendix Fig. 5), RayCo4D improves PSNR by 1.11 dB and reduces LPIPS by 25.9% compared with SE(3) B-splineGS.

AuthorsOpenReview ↗

Initial Response to Reviewer 4Nhi (Part 2/2)

W2. Ablations and Baseline Comparisons

Table 1 evaluates four core components. Removing MSD, RCQ, the WNOA factor, and ARM reduces DyCheck mPSNR and NVIDIA PSNR by 1.35/1.33, 1.19/1.22, 1.03/1.17, and 0.18/0.17 dB, respectively. The w/o ARM variant disables both the local residual motion and its associated regularization, while the ARAP term follows the standard motion-scaffold regularization used in MoSca [1].

We also ablate two late-stage auxiliary components in Eq. (11):

  • w/o Calibration: Disables camera-pose and depth-scale refinement, together with the associated calibration regularization.
  • w/o Appearance Warp: Disables adjacent-frame appearance transport and its corresponding coverage constraint.

All variants retain the core motion modeling components and use the same training budget and evaluation protocol as the full model.

Table R2. Ablation of auxiliary components on DyCheck and NVIDIA.

SettingDyCheck mPSNR ↑mSSIM ↑mLPIPS ↓NVIDIA PSNR ↑SSIM ↑LPIPS ↓
w/o Calibration20.830.7400.24527.830.8700.048
w/o Appearance Warp20.930.7410.24427.880.8690.048
RayCo4D (Full)20.970.7430.24127.940.8720.045

Removing Calibration reduces DyCheck mPSNR and NVIDIA PSNR by 0.14/0.11 dB, while removing Appearance Warp reduces them by 0.04/0.06 dB (Table R2). The other metrics also degrade slightly. By comparison, removing MSD, RCQ, or the WNOA factor lowers both PSNR measures by more than 1 dB (Table 1). The two auxiliary components therefore have smaller effects than these core motion modeling components.

For the comparisons with baseline methods in Table 1, we use the official implementations of the baselines with their recommended native preprocessing configurations, preserving their original visual priors and optimization pipelines. SE(3) B-splineGS [2] also uses publicly available implementations or author-provided models with recommended parameter settings for its baseline comparisons.

All methods in these comparisons are evaluated using consistent dataset splits, evaluation resolutions, and metrics (Sec. 4.1 and Appendix D.9). Since the native visual priors are not necessarily identical across methods, these comparisons reflect the reconstruction performance of the respective complete systems.

All RayCo4D core ablations in Table 1 use the same DA3, BootsTAPIR, and RAFT inputs, initialization, and training budget as the full model. Removing MSD, RCQ, and the WNOA factor reduces DyCheck mPSNR to 19.62, 19.78, and 19.94 dB, respectively. These comparisons isolate the contributions of the motion modeling components under fixed input priors.

References

[1] Lei et al. MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds. CVPR 2025.

[2] Zhang et al. Learning Explicit Continuous Motion Representation for Dynamic Gaussian Splatting from Monocular Videos. CVPR 2026.

Q1. Model Size and Efficiency

In Table R3, we report the average number of Gaussians and end-to-end rendering speed (FPS) on DyCheck (2× resolution), measured on an NVIDIA RTX PRO 6000 GPU. RayCo4D's peak training GPU memory usage is 9.12 GiB.

Table R3. Comparison of average number of Gaussians and rendering speed on DyCheck (2× resolution).

MetricMoScaShape of MotionOriGSRiGSSE(3) B-splineGSUSPLAT4DRayCo4D
Gaussians473,311575,724520,699416,258321,680451,459316,452
FPS40.7139.338.7150.950.268.470.3

RayCo4D uses 316,452 Gaussians on average, fewer than all six baselines (321,680–575,724), and achieves an end-to-end rendering speed of 70.3 FPS. It is faster than MoSca, OriGS, SE(3) B-splineGS, and USPLAT4D, but slower than Shape of Motion and RiGS (Table R3).

On 200-frame DyCheck sequences, RayCo4D's total runtime is 86 minutes (36 minutes for preprocessing and 50 for training), compared with 88–107 minutes for Shape of Motion, OriGS, RiGS, and SE(3) B-splineGS, whose training stages take 56–77 minutes. MoSca requires 60 minutes, while USPLAT4D requires 80 minutes, including 60 minutes of preprocessing (covering MoSca prior generation) and 20 minutes of training (Appendix D.10, Table 8).

Reviewer_4NhiOpenReview ↗

Thanks for the rebuttal

Thank the authors for the detailed rebuttal. I also read the revised paper. My concerns are largely addressed by the added experiments on challenging tracking, ablation, and efficiency. The proposed method is useful considering the number of Gaussians and FPS. I therefore keep my original score of 3: weak accept.

AuthorsOpenReview ↗

Thank You for Your Feedback

Thank you for reviewing our revised paper and for your thoughtful follow-up. We appreciate your comments on RayCo4D's coherent design and practical efficiency. Your suggestions have also helped us better explain and validate how measurement–state decoupling, WNOA-based continuous-time modeling, and reference-anchored local residuals work together.

Reviewer_DvKjOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RayCo4D, a monocular dynamic reconstruction method that treats 2D tracks as measurements of an underlying 3D motion rather than as the motion itself. The authors argue that prior track-driven methods such as MoSca and SE(3) B-splineGS tie motion states closely to lifted tracks, which weakens motion estimation when tracks are fragmented by occlusion. To address this, each track is represented as a single 3D query constrained to its camera ray in a reference frame, and is linked to several nearby shared motion nodes. Observations from other frames then constrain these shared nodes through reprojection and depth losses. As a result, tracks visible at different times can jointly inform the same latent motion. Over time, each node keeps a sparse sequence of SE(3) pose and velocity states, which are coupled by a white-noise-on-acceleration prior. The same prior is used to interpolate between states, so unobserved intervals stay connected and the scene can be rendered at arbitrary times within the sequence. Gaussians follow a blend of their neighboring nodes’ motion, and a per-Gaussian local residual, anchored to be zero at its reference time, captures fine non-rigid deformation that the shared motion cannot represent.

Critical strengths
  1. Consistent separation of measurements and motion states.

The paper consistently carries the view that 2D tracks are measurements of motion, not the motion itself, through its entire model design. Each track is represented as a 3D query whose depth is optimized along its camera ray in a reference frame, while time-varying motion is handled entirely by shared nodes. This allows tracks observed at different times to jointly constrain the same nodes. Shared motion representations are not new in themselves (e.g., Shape of Motion, MoSca), but explicitly designing this separation at the track level directly targets a core difficulty of monocular reconstruction: integrating fragmented observations. The ablation also shows the largest performance drop when this component is removed.

  1. Principled continuous-time modeling.

Instead of choosing a spline form by hand, the paper adopts a WNOA prior from state estimation, so that a single pose–velocity model defines both the coupling between neighboring states and the interpolation at intermediate times. The derivation is clear, and the method shows a clear improvement over SE(3) B-splineGS when reconstructing intermediate frames held out from training, which supports the practical value of this design.

Critical weaknesses
  1. Limited direct evidence for robustness to incomplete tracks.

The paper’s central motivation is integrating fragmented tracks into shared motion, but the current evidence mainly comes from overall image quality and qualitative comparisons. Table 3 is a helpful step, but it removes all supervision at the same frames for every track, so it mainly tests temporal interpolation rather than the targeted scenario where individual tracks are visible over different intervals. A more targeted analysis would strengthen the paper, e.g., randomly dropping segments of individual tracks while keeping image supervision, and comparing degradation against the baselines as the dropped portion or gap length increases. A metric reflecting the claimed temporal consistency would also help. As this is the central claim, I consider it the most important point for my assessment.

Questions for authors
  1. Direct validation under incomplete tracks

Could the authors provide a controlled experiment in which segments of individual tracks are randomly dropped while image supervision is kept, and report how performance degrades relative to the baselines as the dropped portion or gap length increases? Table 3 withholds all supervision at the same frames for every track, so it seems closer to a test of temporal interpolation. I believe such an experiment would most directly support the paper’s core motivation.

  1. Selection of the reference frame r_i

The paper states that r_i is selected based on observation reliability. What are the specific criteria? Since the depth can be corrected along the ray through η_i but the ray direction itself is fixed, how sensitive are the results when the camera pose or track location at the reference frame is inaccurate?

4 comments
AuthorsOpenReview ↗

Initial Response to Reviewer DvKj (Part 1/2)

We thank the reviewer for the positive assessment of our measurement–state decoupling, principled continuous-time modeling, and supporting experiments, as well as the constructive comments and questions. We address each point below.

W1 & Q1. Robustness to Incomplete Tracks

We evaluate robustness to incomplete tracks on the same seven DyCheck scenes as in Table 1, retaining RGB supervision in every training frame (Table R1). The original tracks serve as the 0% additional-masking baseline. For each track, we independently select a starting point and progressively extend a contiguous masked segment from that point to obtain nested masks at 20%, 40%, and 60%. RayCo4D, MoSca, and SE(3) B-splineGS use the masked tracks from the start of motion initialization. Unlike Table 3, which withholds supervision at the same timestamps across all tracks, this experiment introduces track-specific observation gaps.

Table R1. Robustness to contiguous track-observation masking on DyCheck.

MethodTrack MaskingmPSNR ↑mSSIM ↑mLPIPS ↓PCK-T ↑
MoSca0%19.860.7020.2980.822
20%19.610.6990.3040.798
40%19.050.6920.3180.749
60%18.050.6790.3430.633
SE(3) B-splineGS0%20.530.7130.2760.832
20%20.300.7100.2810.811
40%19.870.7020.2920.768
60%18.900.6910.3230.663
RayCo4D (Full)0%20.970.7430.2410.840
20%20.790.7410.2430.825
40%20.320.7360.2490.798
60%19.810.7270.2590.745

At 60% masking, RayCo4D's mPSNR and PCK-T decrease by 1.16 dB and 0.095, respectively, compared with 1.81 dB and 0.189 for MoSca, and 1.63 dB and 0.169 for SE(3) B-splineGS. RayCo4D also shows smaller degradation in mSSIM and mLPIPS. These results support robustness to incomplete tracks and are consistent with MSD's use of observations from different visibility intervals to constrain shared motion nodes, together with WNOA's temporal regularization across observation gaps. The results are included in Sec. 4.3 (Table 4) of the revised manuscript, with the experimental protocol detailed in Appendix E.3.

AuthorsOpenReview ↗

Initial Response to Reviewer DvKj (Part 2/2)

Q2. Reference-Frame Selection and Sensitivity

We clarify the selection rule and evaluate the effects of reference-pixel and camera-pose errors through controlled perturbation experiments. For each track ii, we prioritize visible frames with valid depth and select the reference frame rir_i based on tracking confidence and depth reliability (See Appendix B.1, Eqs. (13) and (14)). For track ii, ties are resolved by selecting the earliest candidate frame. The initial depth di0d_i^0 is taken from the selected frame; if no valid depth is available, it is initialized from a front-facing motion node whose projection is close to the reference pixel (Sec. 3.1).

Although the reference-frame selection prioritizes reliable observations, errors in track localization and camera poses may still affect the reference ray. We therefore analyze the following two error sources:

  • Track localization errors: RCQ optimizes only the depth scalar ηi\eta_i along the reference ray (Eq. (1)). At the reference time, Eq. (3) gives x^i(ri)=qi\hat{\mathbf{x}}_i(r_i)=\mathbf{q}_i, so the reprojection residual is zero when ray construction and reprojection use the same observation and camera parameters. An inaccurate reference pixel therefore introduces a ray-direction error that optimizing ηi\eta_i alone cannot remove. MSD associates each query with K=16K=16 neighboring nodes, so observations from other tracks and timestamps also constrain the shared motion; together with the confidence-weighted Smooth-ℓ1\ell_1 loss in Eq. (4), this helps mitigate the influence of individual track errors on shared motion estimation.

  • Camera-pose errors: Although the reference ray is fixed in camera coordinates, its world-space origin and direction can change with the camera extrinsics. When camera refinement is enabled, we jointly optimize camera pose corrections using a pose prior, temporal smoothness, and reprojection errors of fixed static landmarks (Appendix D.5, Eqs. (61)–(62) and (64)).

Then, we evaluate their impact through controlled perturbation experiments by fixing the initially selected reference frames rir_i on DyCheck, applying two types of perturbations, and reoptimizing the reconstruction (Table R2):

  1. Reference-pixel perturbation. We shift each dynamic track's reference-frame pixel by 2 or 4 px along a random direction, consistently updating the reference ray and its corresponding 2D supervision while resampling depth and reliability. Observations at other timestamps remain unchanged.

  2. Camera-pose perturbation. After DA3 estimation, we independently perturb the rotations of training cameras by 0.5∘0.5^\circ or 1.0∘1.0^\circ, keeping the first-frame pose and camera centers fixed. Original depth, tracks, and optical flow remain unchanged. Test cameras are not perturbed, and all variants follow the same evaluation protocol.

Table R2. Sensitivity to reference pixel and camera pose perturbations on DyCheck.

Perturbation SettingmPSNR ↑mSSIM ↑mLPIPS ↓
Unperturbed (Full)20.970.7430.241
Ref. pixel jitter (2 px)20.940.7410.242
Ref. pixel jitter (4 px)20.880.7380.243
Camera pose jitter (0.5∘0.5^\circ)20.890.7350.245
Camera pose jitter (1.0∘1.0^\circ)20.790.7330.249

Analysis. Reference-pixel perturbations of 2–4 px reduce mPSNR by 0.03–0.09 dB, and camera rotation perturbations of 0.5∘0.5^\circ–1.0∘1.0^\circ reduce mPSNR by 0.08–0.18 dB and increase mLPIPS by 0.004–0.008. Both cause limited degradation within the tested ranges, with larger changes under camera rotation errors.

The reference-frame selection rule is summarized in Sec. 3.1 and detailed in Appendix B.1 (Eqs. (13)–(14)) of the revised manuscript.

Reviewer_DvKjOpenReview ↗

Response to Rebuttal

Thank you for the detailed rebuttal and the revised manuscript.

Regarding W1, which I considered the most important point, the added experiment masks segments of individual tracks while keeping image supervision. As the masking ratio increases, RayCo4D degrades less than the baselines, and the same trend appears in PCK-T. I believe this directly supports the paper’s central claim of robustness to incomplete tracks.

Regarding Q2, the reference-frame selection criterion is now clearly stated, and the perturbation experiments on the reference pixel and camera pose show that the performance drop remains limited within the tested ranges.

My concerns have been addressed, and I will maintain my original rating.

AuthorsOpenReview ↗

Thank You for Your Feedback

Thank you for taking the time to revisit our revised manuscript and for your thoughtful feedback. We especially appreciate your comments on separating track observations from shared motion states and our WNOA-based continuous-time formulation. Your suggestions helped us further validate these designs through the additional track-masking and perturbation experiments.

Reviewer_37mCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This method focuses on performing 4D Gaussian Splatting using depth, long term tracking priors and flow as additional priors to further constrain moving sequences. By applying a many-to-many tracking system ray constrained continuous motion can be cleanly derived. This is then applied to hierarchical reference anchored Gaussians allowing for superior dynamic reconstruction with faster performance than competing methods.

Critical strengths
  1. The primary results are quite good and consistent across scenes even with complex motion. This is further motivated by the rigorous derivation backing up the method. On the experimental side the fact these gains seemingly save compute speed despite the added subsystems is quite impressive.

  2. The ablation explores the added benefits of each component quite well showing where particular gains comes from, showing the reason behind what can appear like an over constrained formulation.

Critical weaknesses
  1. Novelty: the main weakness is that while the exact formulation of the continuous motion modelling is new, the rest of the system does seem to borrow a lot from existing works like MoSca, which also performs multi node blending. This make the primary novelty seem more incremental, however the improved results do show it to be a useful improvement.

  2. Failure/extreme cases: the initial phrasing of the method seems to suggest this is well suited for very long sequences or ones with more extreme motion (although limitations seems to contradict this?). Yet these comparisons especially to the baselines is missing. Are there particularly challenging sequences or longer ones you can compare against to maintain these claims?

Questions for authors
  1. How does your method handle objects that move out of frame and back in, is unobserved geometry preserved and if so for how long?

  2. You highlight better speed for the efficiency claims, how about memory consumption from the additional components?

  3. As your method uses long-term tracking, how far can this be pushed? Is there a quality fall off, or improvement with longer vs shorter sequences?

3 comments
AuthorsOpenReview ↗

Initial Response to Reviewer 37mC (Part 1/2)

We thank the reviewer for the positive assessment of our reconstruction performance, methodological formulation, computational efficiency, and ablation studies, as well as the constructive comments and questions. We address each point below.

W1. Novelty over MoSca

The novelty of RayCo4D does not lie in introducing shared nodes, but in explicitly decoupling visual track measurements from latent 3D motion states through RCQ (Ray-Constrained Queries) and MSD (Measurement–State Decoupling) (Fig. 1). Both MoSca and RayCo4D use lifted tracks to initialize their motion scaffolds. The distinction lies in subsequent optimization: RayCo4D represents each track using a ray-constrained reference query (RCQ, Eq. (1)), whose observations constrain shared latent motion nodes through local many-to-many associations (MSD, Eqs. (2)–(4)), without maintaining a dedicated motion state for each track. Consequently, fragmented tracks from different visibility intervals can jointly constrain shared motion states, reducing dependence on uninterrupted track observations. Building on this formulation, WNOA couples sparse SE(3) pose–velocity states through process constraints to model continuous-time motion (Eqs. (6)–(7)), while reference-anchored ARM compensates for fine-scale non-rigid deformations beyond the shared motion representation (Eqs. (9)–(10)).

Recent methods also build on shared motion representations: USPLAT4D [1] utilizes uncertainty to guide motion propagation, and SE(3) B-splineGS [2] models continuous motion with cumulative B-splines. RayCo4D instead focuses on ray-conditioned measurement–state decoupling and continuous pose–velocity dynamics. Removing MSD causes the largest drops among the four core ablations: 1.35 dB in DyCheck mPSNR and 1.33 dB in NVIDIA PSNR (Table 1). RayCo4D also achieves higher PCK-T than MoSca on DyCheck (0.840 vs. 0.822; Table 2).

References

[1] Guo et al. Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction. ICLR 2026.

[2] Zhang et al. Learning Explicit Continuous Motion Representation for Dynamic Gaussian Splatting from Monocular Videos. CVPR 2026.

W2. Challenging Motion, Long Sequences, and Limitations

Scope and Limitations: RayCo4D aims to integrate fragmented track observations across different visibility intervals, rather than recover arbitrary motion during prolonged periods without observations. When individual tracks are interrupted by occlusion or leaving the frame, other tracks with overlapping node support can still jointly constrain shared motion states through MSD, while WNOA provides continuous-time motion constraints. However, when a moving region lacks reliable observations for an extended period and undergoes abrupt motion changes, the smooth WNOA prior may fail to accurately recover the underlying motion (Appendix E.4).

Challenging Motion: For challenging motion, Table 1 provides quantitative comparisons against six baselines on DyCheck and NVIDIA, where RayCo4D achieves the best reconstruction metrics under both evaluation protocols. Fig. 3 and Appendix F (Figs. 8–13) show qualitative comparisons on dynamic scenes such as Breakdance-flare, Parkour, Wheel, and Paper-windmill. Moreover, on held-out intermediate frames (Table 3), RayCo4D improves PSNR by 1.11 dB and reduces LPIPS by 25.9% compared with SE(3) B-splineGS.

Long-Sequence Reconstruction: Shared nodes aggregate track observations across visibility intervals. Appendix D.10 (Table 8) reports runtime on 200-frame DyCheck sequences. We also evaluate 100-, 200-, and 400-frame inputs from the DyCheck spin scene. Reconstruction quality improves from 100 to 200 frames and declines only slightly at 400 frames, remaining better than the 100-frame results (Q3, Table R2).

AuthorsOpenReview ↗

Initial Response to Reviewer 37mC (Part 2/2)

Q1. Object Exit and Re-entry

RayCo4D does not employ a geometry expiration mechanism based on consecutive invisible frames. Each dynamic Gaussian stores its reference-time geometry and temporally shared scale, opacity, and spherical-harmonic coefficients (Appendix D.4). Its continuous-time pose is computed through shared motion nodes and reference-anchored local residuals (Sec. 3.3, Eqs. (8)–(10)). Consequently, the reconstructed dynamic representation can be queried and rendered at any real-valued timestamp within the entire reconstructed interval [0,F−1][0,F-1], without requiring continuous object visibility.

Additionally, RayCo4D jointly optimizes the full video offline, including observations after re-entry. Through MSD, observations before exit and after re-entry can constrain shared motion nodes with overlapping support, while WNOA provides continuous-time motion priors and interpolation across unobserved intervals (Secs. 3.1–3.2). Recovery accuracy primarily depends on track association quality and motion complexity.

We evaluate object exit and re-entry on the DAVIS drone (91 frames) and walking (72 frames) sequences, containing consecutive target-invisible intervals of 52 frames (18–69) and 17 frames (35–51), respectively. Qualitative training-view comparisons with MoSca using the same original RGB video inputs show that RayCo4D better preserves object structural integrity and fine details after re-entry. These comparisons are shown in Appendix E.2 (Fig. 6) of the revised manuscript.

Q2. Memory Consumption of Additional Components

Across seven DyCheck scenes (200 frames, 2× resolution, RTX PRO 6000), the states and associated tensors of WNOA, RCQ, and ARM occupy 118 MiB (~0.12 GiB) on average, or ~0.42 GiB including gradients and Adam optimizer states, corresponding to 2%–5% of the full-model peak training memory (Table R1).

Table R1. Average memory footprint of additional components on DyCheck.

WNOARCQARMTotal Storagew/ Grad & Optimizer
86 MiB14 MiB18 MiB118 MiB~0.42 GiB

This compact footprint follows from shared motion modeling: the WNOA-based motion scaffold maintains node pose–velocity states only at sparse temporal supports, RCQ parameterizes each track's reference query with a single learnable depth scalar, and MSD associates these queries with shared motion nodes, avoiding independent motion-state sequences for individual tracks.

Q3. Long-Sequence Reconstruction

We evaluate 100-, 200-, and 400-frame inputs from the DyCheck spin scene at 2× resolution.

Table R2. Sequence-length ablation on DyCheck spin.

Sequence Length (Frames)mPSNR ↑mSSIM ↑mLPIPS ↓
10020.640.7460.174
20020.830.7630.167
40020.740.7580.170

mPSNR improves by 0.19 dB from 100 to 200 frames and decreases by 0.09 dB from 200 to 400 frames. All three metrics at 400 frames remain better than at 100 frames. On spin, RayCo4D maintains relatively stable reconstruction quality up to 400 frames. Scaling to thousands of frames may require increased temporal capacity or chunk-based processing, given hardware, preprocessing, and optimization costs.

AuthorsOpenReview ↗

W2 & Q3. Additional Long-Sequence Baseline Comparison

To further address the reviewer's concerns regarding long-sequence baseline comparisons, we extend the previously reported Table R2 by adding results for MoSca and SE(3) B-splineGS on the DyCheck spin scene with 100-, 200-, and 400-frame inputs at 2× resolution, while retaining the original RayCo4D results (see Table R3).

Table R3. Sequence-length comparison with baselines on DyCheck spin.

MethodSequence Length (Frames)mPSNR ↑mSSIM ↑mLPIPS ↓
MoSca10019.930.7120.230
20019.810.7030.235
40019.620.6900.243
SE(3) B-splineGS10020.250.7290.211
20020.290.7320.209
40020.180.7210.215
RayCo4D10020.640.7460.174
20020.830.7630.167
40020.740.7580.170

RayCo4D achieves the best results across all three metrics at every evaluated sequence length. From 100 to 400 frames, MoSca and SE(3) B-splineGS exhibit mPSNR decreases of 0.31 dB and 0.07 dB, respectively, whereas RayCo4D improves by 0.10 dB. At 400 frames, RayCo4D outperforms MoSca and SE(3) B-splineGS by 1.12 dB and 0.56 dB in mPSNR, respectively, demonstrating stable reconstruction quality on spin up to 400 frames.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Area Chair and Reviewers,

We thank the reviewers for their careful evaluation and constructive feedback. We have uploaded a revised PDF and provide point-by-point responses to the comments and questions in the individual reviewer threads. All additions and modifications in the revised manuscript are highlighted in blue. References to manuscript sections, equations, tables, and figures follow the numbering in the revised PDF. Additional tables labeled R1–R3 are provided separately in the individual reviewer responses, with numbering specific to each response.

Main Changes in the Revised Manuscript

1. Reference-Frame Selection (Sec. 3.1, Appendix B.1, Eqs. (13)–(14))

We clarified the reference-frame selection criterion in Sec. 3.1 and added its explicit mathematical formulation, including the depth-reliability weights, in the new Appendix B.1 (Eqs. (13)–(14)).

2. Object Exit and Re-entry (Appendix E.2, Fig. 6)

We added a qualitative evaluation of objects leaving the field of view and subsequently re-entering. Figure 6 compares RayCo4D with MoSca on the DAVIS Walking and Drone sequences, involving consecutive target-invisible intervals of 17 and 52 frames, respectively.

3. Robustness to Incomplete Tracks (Sec. 4.3, Table 4)

We added controlled track-masking experiments across all seven DyCheck scenes in Sec. 4.3. RGB supervision is retained in every training frame. For each track, we independently select a starting point and progressively extend a contiguous masked segment to obtain nested masks at 20%, 40%, and 60%. Table 4 compares RayCo4D with MoSca and SE(3) B-splineGS in terms of cross-time motion correspondence (PCK-T) and novel-view synthesis quality (mPSNR, mSSIM, and mLPIPS). The detailed experimental protocol is provided in Appendix E.3.

All the above changes are incorporated into the revised PDF. In addition, our individual reviewer responses provide supplementary experiments and analyses on sequence-length scaling, component memory consumption, sensitivity to reference-pixel and camera-pose perturbations, auxiliary-component ablations, Gaussian counts, rendering speed (FPS), and computational efficiency.

Please refer to the individual reviewer threads for detailed responses and the corresponding experimental results.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.