acceptodds
Withdrawn from ICLR 2027

SD4R: Decoupling Motion and Correspondence for 4D Gaussian Scene Reconstruction and Editing from Unposed Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unposed dynamic reconstruction from casually captured videos is fundamentally underconstrained, as camera motion, non-rigid dynamics, and temporal correspondence must be inferred jointly from image observations. Beyond the well-known ambiguity between camera and scene motion, we identify correspondence instability as another source of optimization failure: a representation may explain each frame well while gradually losing the identity of the physical elements it is intended to track. % We present , a unified framework that treats unposed dynamic reconstruction as the joint estimation of geometry, motion, and persistent correspondence. The central idea is to decouple motion representation from identity preservation, allowing scene dynamics to remain flexible while correspondence provides a stable temporal reference. We further introduce a compact sparse motion representation that captures coherent non-rigid dynamics through shared continuous structure. % Importantly, the resulting persistent representation naturally extends beyond reconstruction. By linking image-space semantics to temporally persistent scene primitives, enables controllable object-level 4D editing within the same representation. Experiments demonstrate that this formulation improves dynamic reconstruction consistency while supporting localized scene editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
unposed video, Sparse Control Fields, Dynamic 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3hjhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SD4R, a framework for dynamic 4D Gaussian scene reconstruction and editing based on monocular video without pose information. The method first estimates the camera pose during a camera warm-up phase, then initializes the dynamic Gaussian using the estimated depth and long-range 2D trajectories. Subsequently, it models continuous non-rigid motion using a sparse control field and provides continuous supervision through fixed trajectory identities during the joint optimization process. Furthermore, semantic masks are associated with Gaussian primitives using the same correspondence, enabling object-level 4D editing. The paper achieves competitive new-view synthesis results on seven sequences from the NVIDIA Dynamic Scenes dataset and provides supplementary validation through qualitative reconstruction results and editing examples on the DAVIS dataset. Ablation experiments analyze the roles of the motion modeling and persistent correspondence modules.

Critical strengths

This paper investigates the problem of dynamic scene reconstruction based on monocular video without pose information. This task requires the joint estimation of camera motion and non-rigid deformation, making it highly challenging. The core design proposed in this paper is the decoupling of motion optimization from persistent correspondences, driven by a clear and intuitive research motivation. Maintaining fixed trajectory associations throughout the optimization process provides a promising approach for enhancing temporal supervision, while further integrating the scene reconstruction results with object-level four-dimensional editing.

The method achieved competitive reconstruction results on the NVIDIA Dynamic Scenes benchmark dataset. Across seven test sequences, the average PSNR reached 27.49 dB and the LPIPS reached 0.050, both representing the best average performance among the compared methods in Table 1. Furthermore, ablation experiments preliminarily demonstrated that both the sparse motion control mechanism and persistent correspondence supervision can effectively improve overall reconstruction quality.

Critical weaknesses

Several technical modules of SD4R overlap with existing methods. SC-GS and MoSca have already explored sparse motion control, while MOSAIC-GS also utilizes depth, optical flow, segmentation information, and point trajectories for dynamic scene reconstruction. Although the persistent correspondence mechanism may have some innovative value, the technical distinctions between it and existing trajectory-supervised methods and persistent dynamic representations remain unclear. The authors need to further clarify the innovativeness of this mechanism and its independent technical contributions.

Table 2 primarily evaluates correspondence supervision using rendering metrics such as PSNR, SSIM, and LPIPS; it cannot directly demonstrate a reduction in correspondence drift or an improvement in tracking accuracy. We recommend validating the core mechanism through controlled comparisons with iterative rematching schemes and direct trajectory consistency metrics. Furthermore, Figure 4 shows only a few examples of independent edits and lacks sufficient evidence of temporal and cross-view consistency; relevant experiments should be added to support the claimed advantages of 4D editing.

Questions for authors

Q1. Sparse‑motion‑control designs have been investigated in SC‑GS and MoSca, and Shape of Motion together with MOSAIC‑GS leverage trajectory cues for dynamic reconstruction. What are the core technical distinctions between these prior works and SD4R? An ablation study that disentangles the effect of persistent correspondence from sparse motion field and camera initialization is expected. Head‑to‑head comparisons against closely related approaches would further clarify the paper’s technical novelty.

Q2. Table 2 assesses correspondence supervision mostly via rendering‑quality metrics. Could the authors conduct direct comparisons between fixed track‑identity scheme and iterative correspondence reassignment, and report tracking error or correspondence consistency metrics? Merely counting reduced identity switches is not convincing given the fixed‑by‑design track identities. It also remains unclear how the framework copes with erroneous initial track assignments.

Q3. The editing samples in Figure 4 are isolated cases and cannot fully demonstrate temporal consistency. Results showing the same edited object across multiple frames and novel viewpoints should be supplemented. Ablations disabling the persistent‑trajectory‑to‑Gaussian association or Gaussian‑wise parameter gating are needed to validate their individual contributions. Quantitative measurements for temporal consistency and background preservation are also desirable.

Q4. Section 3.3 requires that trajectories remain visible throughout the entire video sequence in the main experiment, which may result in trajectories affected by occlusion being excluded. Under this setup, how many valid trajectories are actually retained? We recommend evaluating the impact of different visibility requirements, initial trajectory noise, and depth estimation errors on the method’s performance. Additionally, the authors should clarify how the camera’s intrinsic parameters are obtained and specify the loss function used during the camera warm-up phase.

Q5. Table 1 covers seven evaluation sequences, whereas ablation results in Table 2 are computed over only five selected scenes. Which exact scenes are chosen, and what motivates this selection? Per‑scene ablation numbers should be given; ideally, key ablations should run across all seven sequences. Since efficiency is highlighted in the paper, training runtime, total parameter count and peak GPU memory consumption ought to be reported, together with clear breakdowns of preprocessing overhead.

Reviewer_8yPyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SD4R reconstructs and edits dynamic Gaussian scenes from unposed monocular videos. To reduce ambiguity between camera motion, scene deformation, and the introduced correspondence assignment, it combines camera warm-up, sparse motion controls, and persistent track identities. Experiments report improved reconstruction quality on NVIDIA Dynamic Scenes subset and DAVIS dataset.

Critical strengths
  1. The method achieves better average reconstruction quality than the competitors on the NVIDIA subset, improving PSNR by 0.89 dB and LPIPS by 0.021 compared with MoSca.
Critical weaknesses
  1. The evaluation does not sufficiently support the claims.
    • The track-supervision ablations do not directly demonstrate reduced correspondence drift (see Q2 for more detail).
    • Camera warm-up is not ablated, and camera accuracy is not reported.
    • I highly recommend including video comparisons for a more comprehensive evaluation.
    • Editing is evaluated only limited qualitatively without quantitative comparisons.
  2. The difference from the cited prior work is not clearly stated.
    • Sparse motion fields have already been explored in SC-GS and MoSca, but the paper does not explain the differences.
    • SplineGS already uses camera-only warm-up followed by joint camera–scene optimization, yet its relationship to the proposed strategy is not discussed.
Questions for authors
  1. How does the proposed persistent track loss perform compared to existing track losses (e.g., MoSca or more recent methods)?
    • While Table 2 demonstrates its effectiveness, comparisons with existing approaches are missing.
  2. Can the authors provide quantitative results that the proposed method reduces correspondence drift?
  3. How robust is the method to incorrect initial correspondences, and what are its potential failure cases?
Reviewer_mVtbOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SD4R, a method for reconstructing dynamic 3D Gaussian scenes from unposed monocular video and then editing them. Beyond the usual ambiguity between camera motion and scene motion, the authors argue that unposed reconstruction suffers from “correspondence instability”, a Gaussian’s supervising observation can switch during optimization even when the rendering loss stays low. Their proposed remedy is to decouple how a Gaussian moves from which observation supervises it. Their 3 primary claims are, first that this correspondence instability is a major issue in performing dynamic 3DGS reconstruction. Second, a correspondence persistent representation that decouple motion from object-level identity, and finally, a learned model to perform object level editing in a 4D scene.

Critical strengths
  1. Overall the formulation of the methodology and its use to address these very present issues is quite sound. This results in quite good benchmark numbers in the end when compared against many other 4DGS methods.

  2. The ability to recover this level of reconstruction without pose priors is quite impressive and could arguably be suitable as a core contribution of the work to strengthen its standing.

Critical weaknesses
  1. The primary issue lies in the novelty and validation of the primary claims. The core methodology seems quite similar to existing works (see point 2), and does not present clear metric for evaluating the issue of correspondence instability, quantitative or qualitative. While this is indeed a present issue in such methods, do you present any means of evaluating it to see if this issue is being addressed. For the final claim of the novelty of the network masking for the object level editing, this also does not appear to be evaluated or compared. There are other works attempting or performing similar tasks, but again no experiments exploring these.

  2. Additionally, the Related Works sections is underdeveloped and seems to be missing some key comparisons. The issue of correspondence instability in moving 3DGS scenes is very present in the field, but is not as under explored as the related works would suggest. Works such as Mosiac-GS (Morkva, CVPR 2026) address this instance consistency with explicit object instancing from the network provided masks. This seems quite similar to the core novelty claims with the instancing networking and to a lesser extent the canonical vs dynamic Gaussian separation, while the 4DGS aspects seem to closely match the methodology of MoSCA (Lei, CVPR 2025 not Arkiv) but an explanation of how these methods is absent from the Related Works.

Questions for authors

Clarifications

  • How are you evaluating correspondence integrity? While on a qualitative and high level it is an observed issue, and your supervision formulation provides the math to try and enforce it there does not appear to be an evaluation of this metric, especially with other baselines.
  • Are there any limitations?
  • How does your method handle objects that move in and out of view either out of frame or behind occlusion?

Future Work

  • Is there any ablation or work into how different network masking approaches impact the editing claim? Do a certain type of features, or temporally guided networks out perform others?
  • Minor point regarding presentation for future submissions, Figure 3 appears to be missing the images in the close up views? Are the red and blue boxes supposed to be white? On downloaded versions the images appear to be rendering on top of each other…

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.