acceptodds
Under review as a conference paper at ICLR 2027

GaussTrace: Persistent 3D Trajectories for Dynamic Gaussian Reconstruction from Monocular Video

18 views

Abstract

Reconstructing dynamic 3D scenes from casually captured monocular video requires temporally coherent motion under sparse, ambiguous observations, yet existing 4D Gaussian Splatting (4DGS) pipelines rely on fragmented preprocessing or expensive scene-specific optimization. We present **GaussTrace**, a framework built around persistent 3D trajectories that converts a single feed-forward 4D prediction into unified priors for dynamic Gaussian reconstruction. *Trajectory-Basis Motion Modeling* (TBMM) groups similar trajectory bases and learns shared translation corrections with local residuals, while *Motion-Aware Routing and Temporal Coreset* (MRTC) combines tri-state motion routing with motion-aware time selection to allocate motion freedom and focus refinement on representative observations. *Progressive Gaussian Optimization* (PGO) separates scene stabilization from motion refinement before jointly optimizing Gaussian attributes and motion corrections. Experiments on NVIDIA Dynamic Scenes, iPhone DyCheck, and DAVIS show strong reconstruction quality, sharper dynamic boundaries, and substantially more efficient optimization than representative monocular 4DGS systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Monocular dynamic reconstruction, 4D Gaussian Splatting, dynamic novel-view synthesis, 3D point trajectories

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PpKgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose GaussTrace, a monocular 4DGS method combing the feedforward model with the per-scene optimization. Their method have 3 main stages: TBMM to initialize Gaussian and simplify the groping representation, MRTC to differentiate the dynamic region, static region, and ambiguous dynamic boundaries, and PGO to stabilize the scene and refine motion separately. The experiments demonstrate the fast optimization and best metrics compared with baselines.

Critical strengths
  1. This paper propose a very close desgin based on the priors obtained from OmniX and achieve much faster per-scene optimization than the other baselines. The typical time cost is 2-4 min compared with 19min-6hours. Also, instead of relying on multiple standalone foundation model for post estimation, depth estimation, maksing, and trackings, the close couping make the initialization workflow to be neat and simple.
  2. The ablation studies, limitation, budget discussion are sufficient. The logics of the paper is smooth and the figures are intuitive.
Critical weaknesses
  1. The paper directly inherits the limitation of the priors and does not solve it. The foundation model such as OmniX perform well on SEEN regions but is limited under the occlusion. Although the NVS near the input views is qualitatively and quantitatively better than baselines, the performance obviously drops when camera has a large shift. The proposed method can neither fill the NEVER UNSEEN regions nor recover the temporally occluded dynamic region from other available frames. Although the paper partially admit some limitations (where we thank for the honesty of the authors), these unsolved issues restrict the novelty and contributions.
  2. The paper does not fully discuss the influence of foundation model. The baselines such as SoM and MoSca utilizes much earlier visual/geometry foundational models while GaussTrace utilize OmniX (2026). Since the main contribution which this paper claimed is the three stages for per-scene optimization, the paper needs to validate the effectiveness of these three stages with a fair comparison. Although the author replaced the foundation model with others with the same GaussTrace pipeline, the performance becomes worse than baselines which is not persuasive to validate the effectives of the GaussTrace pipeline and, on the contrary, shows the quality seems to come from OmniX. Also, even if GaussTrace utilizes the SOTA foundation model, GaussTrace gains a marginal improvement and even LPIPS is worse than the best baselines in Tables 1 and 2.
Questions for authors

Besides the weekness above, there are some additional questions:

  1. In the demo video, each case is very short compared with the raw clip. It is necessary to show the long-sequence results since the current GaussTrace is claimed to support longer video with Exact-Stream engineering. Could the authors update the long-sequence resutls?
  2. There are clear Gaussian artifacts on the Gaussians' scene boundaries which should be handled individually. Is it possible for the method to solve this issue?
Reviewer_vUyTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents GaussTrace for dynamic Gaussian reconstruction from monocular video, initialized with geometry, camera poses, and persistent 3D trajectories from frozen OmniX predictions. It refines Gaussian attributes and motion through shared and local residual corrections, motion routing, and staged optimization with temporal frame selection. Experiments report competitive novel-view synthesis on NVIDIA Dynamic Scenes and iPhone DyCheck, improved tracking accuracy, and 2–4 minutes of scene-specific optimization.

Critical strengths
  1. The method provides a practical way to convert persistent trajectory priors into renderable dynamic Gaussian scenes. It reports competitive rendering quality with 2–4 minutes of scene-specific optimization.

  2. The refinement improves geometric tracking as well as image reconstruction. The ablations also show benefits from both shared and local motion corrections.

Critical weaknesses
  1. The method seems mainly to adapt OmniX to Gaussian reconstruction through grouped residual corrections, motion routing, and staged optimization. I find the Gaussian-specific design less developed: the integration largely focuses on center trajectories, while trajectory association is fixed and Gaussian covariance remains shared across time. For rotating or deforming surfaces, accurate center tracking alone may not provide a suitable evolving Gaussian representation. This makes the methodological contribution beyond adapting the existing trajectory prior less clear to me.

  2. My main concern with the trajectory-to-Gaussian association is how it handles errors in the initial prediction. Canonical-space nearest neighbors may connect points on nearby surfaces with different motions, particularly around dynamic boundaries. These associations and motion assignments are subsequently fixed, so a mistakenly static Gaussian cannot acquire time-varying motion. Table 9 shows encouraging average tracking improvements; a breakdown by initial tracking error or motion misclassification would help establish which errors the refinement can actually recover from.

  3. I would appreciate clarification of the evaluation protocols. Appendix A.1 describes fixed camera 0 training and camera 1 evaluation for both NVIDIA settings, which differs from the protocols described by RoDynRF (Liu et al., 2023) and Gaussian Marbles (Stearns et al., 2024). It would help to specify whether this reflects camera renumbering or a different split. For DyCheck, the statement about foreground masks also leaves unclear whether the official co-visibility masks are used for evaluation.

  4. The runtime and memory results would benefit from a more consistent accounting. For example, Gaussian Flow takes 27 minutes for optimization in Table 2 but 23 minutes including preprocessing in Table 20. The long-sequence resource measurements also cover different workloads: Table 10 uses 12-frame sequences, while Table 19 profiles 200 frames with only three trajectory queries. Reporting the complete initialization and optimization costs for one representative long sequence would make the practical efficiency easier to assess.

Questions for authors
  1. Could the authors elaborate on the Gaussian-specific modeling choices beyond the OmniX trajectory prior, particularly the use of time-shared covariance for rotating or deforming surfaces?

  2. Could the authors provide tracking errors before and after refinement, grouped by initial error, and separately examine dynamic points initially assigned to the static state?

  3. Could the authors specify the training and evaluation camera mappings for both NVIDIA settings and explain their relationship to the cited protocols? Please also clarify whether the official DyCheck co-visibility masks are used when computing the reported metrics.

  4. Could the authors clarify the Gaussian Flow timing discrepancy between Tables 2 and 20? For one representative long sequence, could they also report prior-extraction and Gaussian-optimization times, together with peak memory, using the complete initialization query set?

2 comments

Dear Reviewer vUyT, if you have any further comments regarding my paper or my responses, please feel free to share them. This will allow me to provide additional clarifications before the deadline. Thank you.

Dear Reviewer vUyT,

Thank you for your earlier review and for identifying the problems in my work. The deadline is now only about two days away, and I fully understand how busy your schedule must be. Should you have any remaining concerns, I would be extremely grateful if you could let me know as soon as convenient, so that I would still have a chance to respond with clarifications, revisions, or additional material. Thank you very much for your help.

Reviewer_bLkCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents GaussTrace, a monocular dynamic Gaussian reconstruction framework that combines a feed-forward 4D model with short per-scene optimization. A single frozen forward pass of OmniX provides cameras, depth–ray geometry, persistent dense 3D point trajectories, dynamic scores, and 516 trajectory bases, which initialize the Gaussians and serve as a fixed motion scaffold. Three components build on this prior. Trajectory-Basis Motion Modeling (TBMM) groups the bases into 32 groups and learns group-shared translation corrections together with per-Gaussian residuals. Motion-Aware Routing and Temporal Coreset (MRTC) assigns each Gaussian a static, uncertain, or dynamic state from the OmniX dynamic score, and for long videos it concentrates updates on representative timestamps. Progressive Gaussian Optimization (PGO) first stabilizes appearance, shape, cameras, and topology (Stage A), then jointly refines the Gaussian attributes, cameras, and motion corrections with the topology fixed (Stage B). On NVIDIA Dynamic Scenes and iPhone DyCheck, GaussTrace reports the highest PSNR among the compared methods with 2–4 minutes of per-scene optimization, and DAVIS is used for qualitative comparison.

Critical strengths
  1. Valid problem setting and a large gain at a moderate optimization cost. The limitations of prior work identified by the authors are all valid and convincing: per-scene optimization is slow, single-pass feed-forward prediction yields lower quality, and existing monocular pipelines must assemble priors from several separately trained estimators. I do not consider adding post-optimization to compensate for limited feed-forward quality to be the future direction. I believe the goal should ultimately be a very short post-optimization or the feed-forward output itself. Nevertheless, the post-optimization time here is not long, though I would not call it short: 2–4 minutes per scene, and 8 minutes end-to-end on iPhone DyCheck including OmniX prior extraction. The performance gain is also sufficiently large. GaussTrace far exceeds the feed-forward StreamSplat. It also achieves the highest PSNR among the per-scene methods with much less optimization time.

  2. An effective way to exploit the OmniX prior. The overall scheme itself is common (coarse-to-fine, two-stage optimization). Continuous static–dynamic blending also has precedents, such as the blending weights in NSFF (Li et al., CVPR 2021) and Gao et al. (ICCV 2021), as well as OmniX’s own continuous gate, and the authors themselves do not claim continuous blending as new. Still, I agree that this is one of the effective ways to exploit a prior that is robust to dynamic objects. The method takes the OmniX outputs from a single forward pass (cameras, geometry, persistent 3D trajectories, dynamic scores, and trajectory bases), treats them as a fixed scaffold, and learns only small corrections on top.

Critical weaknesses
  1. Insufficient description of the initialization and Stage A.

Several details needed to understand the method are missing.

(i) The paper does not state whether the initial Gaussians are created only from the points predicted at the first frame or from points of multiple frames. If only the first frame is used, it is unclear how regions not visible in that frame are covered (e.g., disoccluded background or objects that enter later).

(ii) How the initial rotation, scale, opacity, and color of each Gaussian are set is not described.

(iii) It is unclear whether static Gaussians in Stage A also inherit the OmniX trajectories and move over time, or stay at a fixed position. The paper only states that the static offset and trajectory corrections are fixed at zero in Stage A, and the static/uncertain/dynamic gates are constructed after Stage A.

(iv) Splitting (densification) and pruning are said to occur only in Stage A, but their criteria are not given.

(v) Stage A optimizes all Gaussian attributes except the centers, which are given by the OmniX prior. There is no explanation of how this converges while dynamic objects keep moving across views. Where the OmniX trajectories are inaccurate, a dynamic object seen from one view should leave ghosts of the same object observed from other views. The paper neither explains how these ghosts are resolved (removed in Stage B, or absorbed into opacity and color and left in the scene) nor analyzes the Stage A result. The limitations section itself notes that imperfect motion support near boundaries can be compensated by adjusting the coverage, opacity, or color of nearby Gaussians, which leaves blurred contours.

  1. Only the Gaussian centers vary over time, and the fixed-shape approximation is not justified.

Even in Stage B, only the Gaussian centers change over time, while color, opacity, rotation, and scale are shared across all frames. This is a strong assumption that each Gaussian only translates while keeping its shape and orientation. The authors describe it only as “a fixed-shape approximation that favors a stable shared shape model.” They do not explain why it is sufficient or compare it with a variant whose rotation and scale vary over time. For a rotating object, for example, the Gaussian centers can follow circular paths, but their orientations stay fixed, so elongated Gaussians cannot rotate with the object. The authors state that the rotating Umbrella sequence evaluates this approximation, but no controlled comparison is given, and GaussTrace is below SplineGS on this sequence (24.87 vs. 25.73 PSNR, 0.076 vs. 0.061 LPIPS). Objects that stretch or bend (e.g., cloth or arms) would similarly lose expressiveness because neither scale nor orientation can change. This part needs further explanation from the authors.

Questions for authors
  1. Are the initial Gaussians created only from the points predicted at the first frame, or from points of multiple frames? How are their rotation, scale, opacity, and color initialized, and how are regions not visible in the first frame covered?

  2. In Stage A, do static Gaussians stay at a fixed position or follow their OmniX trajectories? What are the splitting and pruning criteria?

  3. How does Stage A converge when dynamic objects move across views and the OmniX trajectories are inaccurate? Do ghosts of dynamic objects remain after Stage A, and are they removed in Stage B or absorbed into opacity and color? Showing Stage A renderings would help.

  4. Why is the fixed-shape approximation sufficient? Please compare it with a variant that also rotates each Gaussian with the per-frame OmniX basis rotations, or one that learns time-varying rotation and scale. The comparison should include rotating objects such as Umbrella and stretching or bending objects.

My rating is close to borderline, and the answers to these questions will largely determine my final assessment.

6 comments
AuthorsOpenReview ↗

W1. Gaussian initialization and regions unseen in the first frame

The canonical reference time c=0c=0 does not mean that Gaussians are initialized only from the first frame.

(1) Multi-source Gaussian initialization and trajectory merging.

For the 12-frame evaluation, we initialize from the middle frame, following a setup similar to Shape of Motion. For long sequences, a heuristic inspired by the motion-aware temporal coreset (MATC) component of Motion-Aware Routing and Temporal Coreset (MRTC) balances motion importance and spatiotemporal coverage: the first and last frames are fixed, and one anchor is selected from each of three interior intervals by trajectory confidence, motion strength, and cross-frame projection coverage:

\qquad \mathcal A=\{0,s_1,s_2,s_3,T-1\},$$ $$F(\mathcal A)= \sum_{\gamma\in\bigcup_{s\in\mathcal A}\mathcal G_s} \max_{\substack{i:\,s_i\in\mathcal A,t:\,\gamma_{i,t}=\gamma}} c_{i,t}(1+\hat r_i).$$ Here, $\mathcal G_s$ is the set of coverage cells observed from source frame $s$; $\gamma$ is a spatiotemporal cell, $c_{i,t}$ is trajectory confidence, and $\hat r_i$ is the normalized motion score. Each cell contributes only its maximum weight, limiting redundant observations. We query pixels in each selected source frame and use OmniX to predict their persistent 3D trajectories over the full sequence. Because trajectories share a coordinate system, **we concatenate them along the point dimension without cross-frame registration or point matching**: $$\mathcal P= \bigsqcup_{s\in\mathcal A^\star} \{(s,p)\mid p\in\mathcal Q_s\}.$$ Each query $i=(s_i,p_i)$ retains its source identity; its canonical center and color are initialized as: $$\boxed{ \mu_i^{(0)}=\mu^o_{0,i}, \qquad \mathbf c_i^{(0)}=I_{s_i}(p_i). }$$ Source-support filtering keeps reference-frame static queries and multi-source dynamic candidates to limit redundant static geometry. On 192-frame Balloon1, five anchors yield **44,100 registered trajectory queries** and **19,365 initial Gaussians**. **(2) Gaussian attribute initialization.** As in **3D Gaussian Splatting (3DGS)**, we initialize from a point cloud, using local PCA for anisotropic scale and rotation: - **Center:** OmniX's 3D coordinate at the canonical time. - **Color:** source-frame RGB, parameterized as logits. - **Scale and rotation:** **16-NN local PCA** estimates directions and scales and initializes the quaternion and anisotropic scale; $a=0.20$, $\rho=0.6$. - **Opacity:** $\alpha=0.9$, parameterized as a logit. Unlike standard 3DGS's neighbor-distance scale and identity rotation, PCA estimates local orientation here. **All listed Gaussian attributes remain learnable.** **(3) Regions unseen in the first frame.** If a dynamic surface is absent in the first frame but visible in a later selected source frame, its query trajectory supplies a canonical-space location, allowing multi-source initialization to add geometry missed by a single reference frame. Coverage remains observation-limited: source-support filtering may omit later-revealed static background, and no source anchor can recover a never-observed surface. **(4) Controlled ablation of initialization strategies.** We compare initialization strategies under the **seven-scene NVIDIA fixed-camera complete-sequence cross-view evaluation (camera 0→1)**, holding OmniX, TBMM/MRTC/PGO, budget, and evaluation fixed. **Table W1: Gaussian initialization ablation on NVIDIA (7-scene complete-sequence cross-view evaluation)** | Initialization strategy | ΔPSNR ↑ (dB) | |---|---:| | First-frame only | -1.65 | | SoM-style single canonical frame | -1.65 | | Adaptive 3 anchors | -0.61 | | Uniform 5 anchors | -0.32 | | **Adaptive 5 anchors (ours)** | **0.00** | | Adaptive 7 anchors | +0.02 | | MoSca-style multi-time fusion | +0.10 | *ΔPSNR is relative to Adaptive 5 anchors. SoM-style and MoSca-style replace only the initialization strategy; they are not comparisons of the complete methods.* The results show: - **Multi-source:** Five anchors improve PSNR by **1.65 dB** over both single-frame variants. - **Adaptive selection:** At five anchors, adaptive selection gains **0.32 dB** over uniform selection. - **Anchor count:** Five anchors gain **0.61 dB** over three; seven add **0.02 dB**. MoSca-style fusion is **0.10 dB** higher. Additional anchors increasingly overlap in geometric support, so coverage gains saturate. **Five adaptive anchors approach denser-initialization quality with a limited query budget.** Thus, **adaptive multi-source queries, a shared canonical space, and persistent 3D trajectories** expand initial coverage without cross-source point-cloud registration.
AuthorsOpenReview ↗

W2. Motion and topology of static Gaussians during Stage A

(1) Do static Gaussians move during Stage A?

Stage A freezes learnable motion corrections, not the base trajectories predicted by OmniX. Tri-state motion routing is not introduced until Stage B. During Stage A, Gaussian centers follow the frozen, time-varying OmniX trajectories:

=A^o_{t,i}\mu^o_{0,i}+b^o_{t,i}.$$ Thus, a Gaussian later assigned to the Static route may still move along its OmniX-predicted trajectory during Stage A. Stage A primarily optimizes color, opacity, rotation, scale, and cameras, while static offsets and trajectory corrections are fixed to zero. **Only in Stage B does the method construct tri-state routing from dynamic scores, constrain Static Gaussians to time-invariant centers, and apply the corresponding motion corrections to other Gaussians.** **(2) Splitting and pruning criteria.** In Stage A, we use 3DGS-style gradient-driven densification and opacity pruning: - **Pruning:** remove a Gaussian if its opacity is $\alpha_i<0.005$. - **Splitting:** split an unpruned Gaussian if its mean screen-space position-gradient magnitude is $h_i>10^{-5}$, its maximum scale is $s_i^{\max}>0.01$, and its mean trajectory confidence is $\bar c_i\ge0.5$. - **Duplication:** duplicate an unpruned Gaussian if $h_i>10^{-5}$, $s_i^{\max}\le0.01$, and $\bar c_i\ge0.5$. New Gaussians inherit their parent's motion support. After Stage A, we associate the current Gaussians with source persistent trajectories by nearest-neighbor search in canonical space and transfer the trajectory, confidence, dynamic score, and sparse basis coefficients. **Stage B keeps the Gaussian topology fixed and further optimizes motion and Gaussian attributes; it performs no splitting or pruning.**
AuthorsOpenReview ↗

W3. Stage A optimization of dynamic scenes and ghosting from trajectory errors

(1) Stage A optimization.

Stage A does not fit dynamic objects at static positions; OmniX's persistent 3D trajectories provide Gaussian centers at each timestamp. Thus, even with additional motion corrections frozen, cross-frame RGB supervision can optimize Gaussian appearance, shape, opacity, and cameras.

When initial trajectories are inaccurate, Stage A may partially compensate for projection errors by adjusting Gaussian shape, opacity, and color, leaving ghosting or blurred boundaries. With topology fixed, Stage B further corrects motion through TBMM's group-shared translation correction ∑gρi,gΔτt,g\sum_g\rho_{i,g}\Delta\tau_{t,g} and point-wise residual Δxt,i\Delta x_{t,i}, improving cross-frame projection consistency. Residual artifacts may remain when geometry is missing or motion associations are incorrect, consistent with the revised manuscript's Boundary Failure and Limitations discussions.

(2) Experimental evidence.

The existing ablations establish the value of staged optimization and motion correction:

AblationVariant PSNR ↑Full-method PSNR ↑
w/o PGO (NVIDIA complete-sequence)21.3923.91
w/o group-shared correction (NVIDIA 12-frame)24.5127.64
w/o point-wise residual (NVIDIA 12-frame)19.5627.64

Staged optimization improves PSNR by 2.52 dB over one-stage joint optimization. The group-shared-correction and point-wise-residual ablations further show that both motion-correction components are important for reconstruction quality.

We also provide Stage A and Stage B rendering comparisons in the supplementary video Rebuttal-bLkC.mp4, directly showing the reconstruction and boundary-artifact changes before and after motion refinement.

In summary, Stage A stabilizes the Gaussian representation using frozen persistent trajectories, and Stage B further corrects motion errors. The two-stage optimization improves reconstruction quality but does not guarantee removal of artifacts caused by incorrect geometry or motion associations.

Under a first-order projection approximation, a 3D center error δx\delta\mathbf x induces an image-space displacement of approximately

δp≃JπR δx,\delta\mathbf p\simeq J_\pi R\,\delta\mathbf x,

and at a high-contrast boundary its effect on color reconstruction is approximately related to ∣∇I⊤δp∣|\nabla I^\top\delta\mathbf p|. Trajectory errors therefore often first appear as contour ghosting. Correcting the 3D centers in Stage B can reduce this term, but incorrect visibility and missing geometry cannot be uniquely resolved through color or opacity optimization alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.