acceptodds
Under review as a conference paper at ICLR 2027

TracksTo4DGS: Feed-Forward 4D Gaussian Splatting from Dense Tracks

Abstract

Rendering dynamic scenes from monocular video has attracted growing interest, driven by applications in AR/VR, egocentric perception, and embodied AI. The task is challenging, as it requires jointly recovering accurate geometry, motion, and appearance from a single moving viewpoint. Recently, pretrained foundation models for tracking and reconstruction have made remarkable progress on the first two, providing robust 3D geometry and dense correspondences across diverse environments. However, their outputs are point clouds, which lack the representation needed to render high-quality video. We present TRACKSTO4DGS, an architecture that closes this gap by decoupling scene geometry and motion from appearance. Rather than learning temporal dynamics from scratch through an ambiguous rendering loss, TRACKSTO4DGS leverages frozen foundation models to explicitly define the 3D trajectory of each Gaussian, while a dedicated Attribute Network learns only the remaining properties required for high-fidelity rendering. To make dense tracking data computationally tractable, a sparse bottleneck reasons over a compact set of representative trajectories before decoding attributes for the full dense set. By inheriting the robust geometry of these foundation models, TRACKSTO4DGS achieves strong zero-shot generalization. Trained exclusively with a rendering loss on fixed-length egocentric video, a single model improves average PSNR by 2.0 dB over the strongest baseline across six diverse benchmarks, spanning egocentric, handheld, and object-centric captures, and seamlessly generalizes to unseen video lengths without per-dataset tuning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4DGS, Gaussian splats, point tracks, depth maps, 4D, 3DGS, dense tracking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GiWCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TracksTo4DGS, a feed-forward framework for 4D reconstruction. It leverages frozen geometry and tracking pre-trained foundation models to obtain dense 3D trajectories, while a sparse-bottleneck network predicts Gaussian attributes for rendering. Detailed experiments demonstrating its effectiveness.

Critical strengths
  1. Clear and well-motivated design. The decoupling of geometry and motion from appearance prediction provides a straightforward approach to feed-forward 4D reconstruction. The proposed method effectively utilizes off-the-shelf pre-trained foundation models and added simple yet effective components to achieve more resource-friendly and higher-quality 4D reconstruction.

  2. Extensive experimental evaluation. Results across 6 benchmarks demonstrate competitive reconstruction quality and cross-dataset generalization, supported by ablation studies of the main components. The authors also provided detailed failure cases for discussion.

Critical weaknesses
  1. Potential evaluation bias. On some benchmarks, held-out target images are jointly processed with input images by VGGT-Ω for camera prediction. Since VGGT-Ω performs joint geometric estimation, these target images may also introduce information leakage. An input-only evaluation would make the conclusion more convincing.

  2. Dependence on upstream predictions. The location of Gaussian primitives are directly determined by frozen pre-trained geometry and tracking models, preventing the network from correcting inaccurate trajectories. Furthermore, the experiments primarily rely on a single combination of foundation models (VGGT-Ω and AllTracker). Evaluating alternative models (e.g. CoTracker3) would help assess the robustness and generalizability of the proposed framework.

  3. Insufficient architectural ablations. While the paper provides detailed ablation study, the individual contributions of the global Transformer and motion graph are not sufficiently illustrated. Additional comparisons with simpler architectures or other variations would help to determine the necessity of these components.

Questions for authors
  1. The paper claims that TracksTo4DGS can generalized to unseen video lengths, while it is trained on 51-frame sequences and evaluated on shorter sequences. Could the authors clarify whether the framework can directly process substantially longer videos, and how temporal consistency is maintained across different reconstruction windows?

  2. Given the feed-forward nature of the proposed framework, do the authors see potential for extending it to streaming/online reconstruction? What would be the main challenges in adapting the current pipeline to an online setting?

  3. The proposed model is trained exclusively on egocentric Aria. Could the authors elaborate on the motivation behind this choice? Would the cross-domain generalization be maintained if the model were trained on a different dataset? Furthermore, could the framework benefit from scaling up the training data?

6 comments
AuthorsOpenReview ↗

Response 1/4 to Reviewer GiWC

Potential evaluation bias (W1)

We re-ran all six benchmarks with VGGT-Ω seeing only the input frames, so no held-out image reaches any part of the reconstruction, and we remain more than 2 dB ahead of the best baseline. Because our method predicts its own cameras, each held-out camera must still be placed in our coordinate frame before it can be rendered; this alignment is needed only for evaluation and does not change the reconstruction.

In the paper, we placed the held-out cameras in our predicted frame by running VGGT-Ω jointly on the input and held-out frames, which is how the held-out images could reach our geometry. We now place each held-out camera in one of two ways, neither of which changes the reconstruction:

  • (A) Camera interpolation: each held-out frame along the input camera's path is rendered from a camera interpolated between our predicted cameras for the two neighbouring input frames. This uses no ground truth and no target pixels. Held-out views from a different physical camera (DyCheck's test cameras, NVIDIA's camera rig) cannot be interpolated, so their ground-truth pose is mapped into our frame by a similarity transform fitted on the input cameras, as in the paper's NVIDIA and Pets protocol.
  • (B) BARF camera alignment: the standard evaluation protocol for methods that predict their own cameras (BARF [1], NeRF-- [2], CF-3DGS [3]). Each held-out ground-truth pose is mapped into our frame by a similarity transform fitted on the input cameras. Since PSNR is very sensitive to small misalignment, the 6 pose parameters of each held-out camera are then refined against its image, with the reconstructed scene frozen.
PSNR ↑ADTAriaTUMDyCheckNVIDIAPetsAvg
MoVieS-64, best baseline (GT cameras)27.6622.8019.5815.2220.8723.3421.58
Ours, paper (joint VGGT-Ω)31.5627.4222.2517.2319.5423.7023.62
Ours, input-only, (A) camera interpolation33.1227.2521.4516.2219.5426.1723.96
Ours, input-only, (B) BARF camera alignment31.6326.4522.5317.3621.8928.1124.66

On ADT, Aria, TUM and DyCheck, input-only geometry lowers PSNR by about 0.1 dB on average; the rise in the average comes from NVIDIA and Pets, explained below. Without using any target pixels, (A) reaches 23.96, 2.4 dB above MoVieS-64. With (B), we reach 24.66 and lead MoVieS-64 on all six datasets.

On ADT, Aria, TUM and DyCheck, the paper's joint VGGT-Ω run could have passed information from the held-out images to our reconstruction. To test this, we kept the cameras of the joint run and replaced only its depth with input-only depth. PSNR was unchanged on ADT, Aria and TUM (26.86 with either depth, 12 windows) and changed by 0.2 dB or less on DyCheck (6 windows), so the held-out images had a negligible effect on our depth. The per-dataset differences in the table therefore come from the cameras rather than the depth. In particular, the paper estimated each held-out camera from its own image, while (A) places it without any target pixels, which costs 0.2–1.0 dB on Aria, TUM and DyCheck and gains 1.6 dB on ADT. With (B), all four datasets are within 1 dB of the paper, and above it on ADT, TUM and DyCheck.

On NVIDIA and Pets, the paper already ran VGGT-Ω on the input frames only, so the reconstruction is the same in all three of our rows and only the held-out cameras differ. The paper mapped their ground-truth poses into our frame by a similarity transform, which leaves a small registration error between two independently estimated camera trajectories. BARF alignment corrects it by a median of only 0.16° on NVIDIA and 0.33° on Pets, yet raises PSNR from 19.54 to 21.89 and from 23.70 to 28.11. On Pets, interpolating our own cameras avoids this error without any ground truth and reaches 26.17. NVIDIA's held-out views come from other cameras of its rig, so (A) uses the paper's mapping and the result is unchanged. The baselines receive ground-truth cameras and never pay this registration cost.

In the revised paper: Tables 1 and 2 now use input-only geometry with BARF camera alignment (Section 4, Protocol). Appendix D.2 and Table 5 compare the three ways of placing the held-out cameras and report the depth-only control.

AuthorsOpenReview ↗

Response 2/4 to Reviewer GiWC

Dependence on upstream predictions (W2)

Our method depends on the frozen models, but not on one particular combination: it runs with CoTracker3 instead of AllTracker without retraining, losing only 0.35 dB PSNR, and with Depth Anything 3 instead of VGGT-Ω. Letting the network or per-window optimization correct the trajectories did not improve the final quality. The main cost of fixed centres comes from tracks lost during occlusion, which better trackers would reduce.

Correcting inaccurate trajectories. We tested whether correcting the trajectories improves the results. Early experiments showed that letting the network predict corrections to the tracks did not improve over fixed tracks. We also optimized the Gaussians of each window directly, starting from our tracks, with and without moving the centres (see response to Reviewer 4CNU, W4/Q3). At convergence, moving the centres gives 23.92 PSNR, versus 24.28 when they stay fixed. To see where the fixed centres cost most, we compared rendering quality across image regions. The main error comes from tracks lost during occlusion, where centres are interpolated rather than observed: about 14% of pixels, about 5 dB below regions with observed tracks. Better completion of lost tracks is therefore the most promising direction for future work.

Other geometry model. Table 3 (Table 4 in the revision) swaps VGGT-Ω for Depth Anything 3. This lowers the average PSNR (21.98 vs. 23.32) but improves Pets (25.61 vs. 24.00), so the method also benefits when a geometry model is better suited to a domain.

Other tracker. We replaced AllTracker with CoTracker3 at inference only. The network never saw CoTracker3 tracks in training. Everything else is as in Table 1 (cameras, depth, checkpoint, scoring, all 790 windows). CoTracker3 tracks the same grid of points as AllTracker (every 8 pixels on each keyframe), but it has no upsampling to full resolution, so we upsample its tracks bilinearly. As a control, we also run AllTracker with the same bilinear upsampling.

PSNR ↑ADTAriaTUMDyCheckNVIDIAPetsAvg
AllTracker (paper's Table 1)31.5627.4222.2517.2319.5423.7023.62
AllTracker, bilinear upsampling31.0027.1221.8017.1219.4123.6223.34
CoTracker3, bilinear upsampling30.8126.9521.9516.9319.1623.8023.27

CoTracker3 stays 1.7 dB above the best baseline (MoVieS, 21.58). With the same upsampling, the two trackers differ by less than 0.1 dB, and CoTracker3 is better on TUM and Pets. Since the Gaussian centres come directly from the tracker, better trackers can improve our method without retraining.

In the revised paper: tracker swap in Section 4.2 (Further ablations) and Appendix E.2 with Table 8 (c); trajectory correction by optimization in Appendix E.3; error analysis in Appendix E.4, referenced from the limitations (Section 4.3).

Insufficient architectural ablations, for the Global Transformer and the Motion Graph. (W3)

We trained each ablation from scratch under the Table 3 protocol (Table 4 in the revision), changing one component at a time. The global transformer (attention) matters most: replacing its attention with self-only attention costs 1.0 dB PSNR. The motion graph serves efficiency rather than final quality: it speeds up training at 3.5% of the parameters. We clarify its role in the revision.

Global transformer (attention). We replaced its attention with self-only attention, so tracks no longer exchange information, and kept the parameter count the same. PSNR drops by 1.0 dB on average and by 0.4 to 1.5 dB on every dataset. SSIM and LPIPS also get worse.

Table 4 protocol, 100K updatesPSNR ↑SSIM ↑LPIPS ↓
Full model23.320.7300.161
No global attention22.320.7110.198

Motion graph. The motion graph speeds up convergence, especially on dynamic scenes. It gives the network explicitly what global attention would otherwise have to learn: which seeds lie on the same surface and move together. At 10K updates, the model without it is 0.22 dB behind, with most of the gap on DyCheck, Pets and NVIDIA, the most dynamic datasets. With longer training, the model without it catches up. The graph costs only 1.6M parameters (3.5% of the network).

In the revised paper: Section 4.2, Further ablations, and Appendix E.2 with Table 8 (a).

AuthorsOpenReview ↗

Response 3/4 to Reviewer GiWC

Longer videos and temporal consistency across windows (Q1)

Without retraining, the model directly processes windows of up to 101 input frames, twice its training length, with only a gradual quality drop, and longer videos are processed as consecutive windows. Within a window, each Gaussian keeps one identity over all frames. Across windows, our method currently has no mechanism that relates the reconstructions.

Longer videos. In Table 1, windows on the 4DGT benchmarks have 51 input frames, the training length, and windows on NVIDIA and Pets have 12. A longer video is split into consecutive windows that are reconstructed independently, as in our evaluation protocol (Section 4). To test longer windows, we ran the model on ADT, Aria, TUM and DyCheck with up to 101 input frames per window, rescaling the frame and keyframe embeddings to their training range, and scored the same targets at every length. Values differ from Table 1 because DyCheck's novel views are excluded, as are NVIDIA and Pets, whose clips are too short for these window lengths.

Input frames per window51 (training)6475101
PSNR ↑26.1825.6425.1724.52
Time per window×1.0×1.6×2.1×3.4
Peak GPU memory×1.0×1.4×1.7×2.6

Quality drops by 0.5 dB at 64 input frames and 1.7 dB at 101, yet even at 101 input frames our method remains above every baseline on the same targets. Training on longer windows could reduce the quality drop.

Temporal consistency. Within a window, VGGT-Ω predicts all frames jointly, in one coordinate frame. Tracks start every 10 frames but run both forward and backward over the whole window, so each Gaussian persists across the full window with fixed attributes, and there are no seams at keyframes. Each window, however, is reconstructed independently. Linking windows is a natural extension that we leave to future work: Gaussians could be kept across windows in a shared coordinate frame, with new ones added only where the scene is not yet covered, as coverage selection already does within a window.

In the revised paper: longer windows in Section 4.2 (Further ablations) and Appendix E.2 with Table 8 (d); reconstructing windows independently is discussed in the limitations (Section 4.3).

Extending it to streaming/online reconstruction (Q2)

We believe our method could be extended to streaming reconstruction, since it is feed-forward and produces Gaussians that render in real time. The main challenge is that it currently relies on non-causal geometry and tracking models, which could be replaced by causal ones, as our pipeline already works with other backbones.

The reconstruction itself already suits an online setting: there is no per-scene optimization, the model already works on short windows (12 frames on NVIDIA and Pets), and its Gaussians render at over 300 FPS. The challenge lies in the frozen models that provide geometry and tracks. VGGT-Ω estimates all frames of a window jointly, and AllTracker tracks each keyframe both forward and backward in time, so both rely on future frames. Streaming would require causal, low-latency models in their place: for example, CUT3R [5], which updates its reconstruction one frame at a time, for geometry, and the online mode of CoTracker3 [4] for tracking. Latency matters as well: a 12-frame window currently takes 2.4 s on one H200, more than half of it in the tracker. Our pipeline is not tied to particular backbones. It already runs with CoTracker3 instead of AllTracker without retraining (W2 above), and with Depth Anything 3 instead of VGGT-Ω (Table 3; Table 4 in the revision). Replacing them with causal, low-latency models is therefore a natural path to streaming reconstruction, and an interesting direction for future work.

Reviewer_W3HyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TracksTo4DGS, a feed-forward framework for dynamic novel-view synthesis from monocular videos. The key idea is to decouple geometry and motion from appearance prediction. The method leverages frozen pretrained models, VGGT-Ω and AllTracker, to construct dense 3D trajectories that directly define Gaussian centers. An Attribute Network with a sparse bottleneck then predicts Gaussian attributes for rendering. Experiments across six benchmarks demonstrate competitive performance and strong cross-dataset generalization.

Critical strengths

The paper addresses an important problem with a clear and reasonable motivation. Instead of learning geometry, motion, and appearance jointly, the proposed method leverages pretrained geometry and tracking models to simplify feed-forward 4D reconstruction.

Strong quantitative results across diverse benchmarks. The method achieves competitive performance on six datasets covering different camera configurations and dynamic scenes. The cross-dataset results are particularly encouraging, given that the model is trained only on egocentric videos.

The overall framework is simple and well motivated. The sparse bottleneck provides a practical solution for processing dense trajectories, while the decoupling of motion and appearance avoids learning complex dynamics solely through rendering supervision.

Critical weaknesses
  1. Potential information leakage from held-out target images in the evaluation protocol. On ADT, Aria, TUM, and DyCheck, the authors jointly feed input and held-out target images into VGGT-Ω for camera localization. Since VGGT-Ω performs joint multi-view inference, the held-out images may also influence the predicted depth and camera poses of the input frames, which directly determine the Gaussian trajectories. This raises concerns about the fairness of the evaluation, particularly on DyCheck, where held-out novel-view images may provide additional geometric information. Although the authors acknowledge this issue in Appendix D.1, I recommend reporting results under a strict input-only evaluation protocol, using source-only geometry prediction and target camera alignment based on ground-truth poses, as already implemented for NVIDIA and Pets.

  2. The computational efficiency of the complete pipeline remains a concern. Although the method is feed-forward and avoids per-video optimization, Table 6 shows substantially higher inference time than MoVieS (36.3 vs. 3.6 seconds on NVIDIA). Since the framework relies on multiple pretrained models, it would be useful to provide a runtime breakdown and evaluate scalability to longer input videos.

Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response 1/2 to Reviewer W3Hy

Potential information leakage from held-out target images in the evaluation protocol. (W1)

We re-ran all six benchmarks with VGGT-Ω seeing only the input frames, so no held-out image reaches any part of the reconstruction, and we remain more than 2 dB ahead of the best baseline. Because our method predicts its own cameras, each held-out camera must still be placed in our coordinate frame before it can be rendered; this alignment is needed only for evaluation and does not change the reconstruction.

In the paper, we placed the held-out cameras in our predicted frame by running VGGT-Ω jointly on the input and held-out frames, which is how the held-out images could reach our geometry. We now place each held-out camera in one of two ways, neither of which changes the reconstruction:

  • (A) Camera interpolation: each held-out frame along the input camera's path is rendered from a camera interpolated between our predicted cameras for the two neighbouring input frames. This uses no ground truth and no target pixels. Held-out views from a different physical camera (DyCheck's test cameras, NVIDIA's camera rig) cannot be interpolated, so their ground-truth pose is mapped into our frame by a similarity transform fitted on the input cameras, as in the paper's NVIDIA and Pets protocol.
  • (B) BARF camera alignment: the standard evaluation protocol for methods that predict their own cameras (BARF [1], NeRF-- [2], CF-3DGS [3]). Each held-out ground-truth pose is mapped into our frame by a similarity transform fitted on the input cameras. Since PSNR is very sensitive to small misalignment, the 6 pose parameters of each held-out camera are then refined against its image, with the reconstructed scene frozen.
PSNR ↑ADTAriaTUMDyCheckNVIDIAPetsAvg
MoVieS-64, best baseline (GT cameras)27.6622.8019.5815.2220.8723.3421.58
Ours, paper (joint VGGT-Ω)31.5627.4222.2517.2319.5423.7023.62
Ours, input-only, (A) camera interpolation33.1227.2521.4516.2219.5426.1723.96
Ours, input-only, (B) BARF camera alignment31.6326.4522.5317.3621.8928.1124.66

On ADT, Aria, TUM and DyCheck, input-only geometry lowers PSNR by about 0.1 dB on average; the rise in the average comes from NVIDIA and Pets, explained below. Without using any target pixels, (A) reaches 23.96, 2.4 dB above MoVieS-64. With (B), we reach 24.66 and lead MoVieS-64 on all six datasets.

On ADT, Aria, TUM and DyCheck, the paper's joint VGGT-Ω run could have passed information from the held-out images to our reconstruction. To test this, we kept the cameras of the joint run and replaced only its depth with input-only depth. PSNR was unchanged on ADT, Aria and TUM (26.86 with either depth, 12 windows) and changed by 0.2 dB or less on DyCheck (6 windows), so the held-out images had a negligible effect on our depth. The per-dataset differences in the table therefore come from the cameras rather than the depth. In particular, the paper estimated each held-out camera from its own image, while (A) places it without any target pixels, which costs 0.2–1.0 dB on Aria, TUM and DyCheck and gains 1.6 dB on ADT. With (B), all four datasets are within 1 dB of the paper, and above it on ADT, TUM and DyCheck.

On NVIDIA and Pets, the paper already ran VGGT-Ω on the input frames only, so the reconstruction is the same in all three of our rows and only the held-out cameras differ. The paper mapped their ground-truth poses into our frame by a similarity transform, which leaves a small registration error between two independently estimated camera trajectories. BARF alignment corrects it by a median of only 0.16° on NVIDIA and 0.33° on Pets, yet raises PSNR from 19.54 to 21.89 and from 23.70 to 28.11. On Pets, interpolating our own cameras avoids this error without any ground truth and reaches 26.17. NVIDIA's held-out views come from other cameras of its rig, so (A) uses the paper's mapping and the result is unchanged. The baselines receive ground-truth cameras and never pay this registration cost.

In the revised paper: Tables 1 and 2 now use input-only geometry with BARF camera alignment (Section 4, Protocol). Appendix D.2 and Table 5 compare the three ways of placing the held-out cameras and report the depth-only control.

AuthorsOpenReview ↗

Response 2/2 to Reviewer W3Hy

Computational efficiency of the complete pipeline remains a concern. (W2)

Our method reconstructs a 12-frame window in 2.4 s, within about 2× of MoVieS, the strongest baseline, while scoring 3.1 dB higher in average PSNR. On the longer DyCheck windows, it is almost 4× faster than MoVieS (6.2 s vs 23.1 s, on the same 51-frame windows). There, only 4DGT reconstructs faster, but with rendering it is slower (7.7 s vs 7.2 s), and it is 4.5 dB lower in average PSNR. The 34–36 s in Table 6 (replaced by Tables 11 and 12 in the revision) was mostly evaluation I/O and start-up cost, which each stage of our evaluation code paid separately; all methods are now timed without it. Of the 2.4 s, more than half is spent in the frozen tracker, and our network takes about 0.6 s.

On NVIDIA, 11.4 s of the 36.3 s in Table 6 was evaluation I/O (an image-shape probe, PNG writing and setup). Most of the rest was GPU start-up: tracking, depth/camera estimation and reconstruction each ran in a fresh process and paid it separately. MoVieS (3.6 s) runs as a single model in one process, so it paid this cost once. We now time all methods the same way, with models loaded, removing startup cost. All methods are timed on the same H200 GPU (Table 6 used an H100), reporting the median of 7 runs after 2 warm-up runs. Reconstruction time is the time to produce the 4D Gaussians from the input frames, including tracking and depth/camera estimation.

Ours, per window, one H20012 input frames (NVIDIA, Pets)51 input frames (ADT, Aria, TUM, DyCheck)
AllTracker (s)1.414.00
VGGT-Ω (s)0.221.29
Lifting (s)0.190.71
Our network (s)0.580.60
Reconstruction time (s)2.46.6
Rendering (FPS)440343
Peak GPU memory (GiB)15.319.8

On the same 12-frame windows, MoVieS takes 1.2 s, since it receives ground-truth cameras and runs no separate tracking or depth estimation. Rendering takes about the same time for both methods. Peak GPU memory is 15–20 GiB with all models loaded. The quality–time trade-off against per-scene optimization is in our response to Reviewer 4CNU (W1).

In the revised paper: Appendix E.5 and Tables 11 and 12, which replace the earlier runtime table, summarized in Section 4.1.

Scalability to longer input videos. (W2)

Long videos are processed as consecutive windows, as in our evaluation protocol (Section 4), so the total cost grows linearly with video length. A single window can also be longer than in training: without retraining, the model runs on up to 101 input frames, twice its training length, with only a gradual quality drop.

To test longer windows, we ran the model on ADT, Aria, TUM and DyCheck with up to 101 input frames per window, rescaling the frame and keyframe embeddings to their training range, and scored the same targets at every length. Values differ from Table 1 because DyCheck's novel views are excluded, as are NVIDIA and Pets, whose clips are too short for these window lengths.

Input frames per window51 (training)6475101
PSNR ↑26.1825.6425.1724.52
Time per window×1.0×1.6×2.1×3.4
Peak GPU memory×1.0×1.4×1.7×2.6

Quality drops by 0.5 dB at 64 input frames and 1.7 dB at 101, yet even at 101 input frames our method remains above every baseline on the same targets. Training on longer windows could reduce the quality drop.

In the revised paper: Section 4.2, Further ablations, and Appendix E.2 with Table 8 (d).

[1] Lin et al., "BARF: Bundle-Adjusting Neural Radiance Fields," ICCV 2021.

[2] Wang et al., "NeRF--: Neural Radiance Fields Without Known Camera Parameters," arXiv 2021.

[3] Fu et al., "COLMAP-Free 3D Gaussian Splatting," CVPR 2024.

Dear Reviewer W3Hy,

Thank you again for your review. We'd appreciate your thoughts on whether our rebuttal and the additional experiments address your concerns. For convenience, we summarize the main points:

  • Information leakage. We re-ran all benchmarks with VGGT-Ω seeing only the input frames, so no held-out image reaches the reconstruction. We remain more than 2 dB above the best baseline.
  • Running time. On short windows, our method takes 2.4 s and MoVieS 1.2 s; on long windows, our method takes 6.2 s and MoVieS 23.1 s. We also added a per-stage breakdown of our pipeline.
  • Longer videos. Without retraining, the model runs on windows twice its training length and still outperforms all baselines.

Please let us know if any points need further clarification. Thank you for your time.

Reviewer_4CNUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TracksTo4DGS presents a feed-forward framework for reconstructing dynamic Gaussian scenes from monocular video. Frozen geometry and tracking models generate dense 3D trajectories, which directly determine Gaussian centers. A trainable Attribute Network predicts covariance, opacity, and view-dependent color through a sparse trajectory bottleneck, low-rank motion conditioning, transformer processing, and graph refinement.

The paper reports improved average rendering metrics across six benchmarks. However, the evaluation primarily establishes competitiveness within a selected set of feed-forward and geometry-based pipelines. It does not adequately establish the method’s position relative to established methods addressing the underlying reconstruction and rendering tasks.

Critical strengths
  1. The decomposition of trajectory estimation and Gaussian attribute prediction is clear and practically motivated.

  2. The sparse bottleneck provides a reasonable mechanism for processing dense tracks efficiently.

  3. The method achieves strong average results against the evaluated baselines using one checkpoint without dataset-specific tuning.

  4. The manuscript discloses important implementation details and acknowledges dependence on frozen geometry and tracking models.

Critical weaknesses
  1. Insufficient comparisons with established methods for the claimed tasks. The evaluation does not systematically include dedicated reconstruction and rendering methods. Without these comparisons, the paper cannot establish whether its rendering quality is competitive or quantify the quality sacrificed for amortized reconstruction.

  2. Incomplete quality–efficiency analysis. Avoiding per-video optimization is a useful property, but it does not by itself establish a favorable practical trade-off. The complete pipeline takes approximately 34–36 seconds per window on NVIDIA and Pets, while MoVieS takes approximately 4 seconds. Comparisons should report reconstruction quality, reconstruction time, and rendering speed under clearly specified budgets.

  3. Held-out images influence geometry on four benchmarks. Joint geometry estimation from input and held-out images complicates the interpretation of the main results. Providing ground-truth cameras to some baselines does not make their information access equivalent to conditioning geometry on target images.

  4. The incremental contribution is insufficiently isolated. The system combines established geometry prediction, dense tracking, low-rank motion descriptors, and Gaussian rendering. More direct comparisons with track-guided reconstruction methods and stronger component ablations are needed to demonstrate what the proposed integration adds.

  5. The representation has substantial unaddressed restrictions. Gaussian centers cannot correct tracking or depth errors, while covariance and appearance remain constant within a window. The experiments do not sufficiently characterize the impact of these restrictions on deformation, rotation, occlusion, and appearance changes.

Questions for authors
  1. Why are dedicated Gaussian reconstruction and rendering methods absent from the main quantitative comparison, despite addressing the same reconstruction and novel-view tasks?

  2. How do the results change when VGGT-Ω receives only input images on all benchmarks?

  3. What improvement does the proposed Attribute Network provide over a strong track-initialized Gaussian reconstruction baseline using the same geometry, trajectories, and image access?

  4. Can you separately ablate motion-graph refinement, coverage selection, and global attention?

  5. What are the full pipeline’s peak memory, component-wise runtime, and rendering throughput?

5 comments
AuthorsOpenReview ↗

Response 1/4 to Reviewer 4CNU

Comparison to dedicated per-scene reconstruction methods (W1, Q1)

As requested, we now compare against Shape of Motion [1], a dedicated per-scene method that builds on 2D tracks and monocular depth. Our single forward pass comes close to its rendering quality while taking about 100 times less time, so the quality sacrificed for amortized reconstruction is small.

Table 1 compares methods that share our setting: a new video is reconstructed without per-scene optimization. Per-scene methods such as Shape of Motion [1] address the same task at a much higher cost. We ran Shape of Motion with the same input frames, held-out targets and scoring code as Table 1. It receives ground-truth cameras for the inputs and the targets, computes its own masks, depth and 2D tracks from the input frames only, and uses its default training schedule. Because of the rebuttal time limit, it ran on a subset of the Table 1 windows. We compare our method on exactly the same windows and targets, with the BARF camera alignment described in our answer to W3 below.

PSNR ↑ / LPIPS ↓Time (s), 12 / 51 framesAriaTUMDyCheckNVIDIAPetsAvg
Shape of Motion239 / 65129.50 / 0.15520.91 / 0.28017.25 / 0.39422.18 / 0.19329.27 / 0.21523.82 / 0.247
Ours2.4 / 6.629.55 / 0.11921.13 / 0.22616.98 / 0.37021.25 / 0.23728.03 / 0.21823.39 / 0.234

Times are per window on one H200. For Shape of Motion, we report its optimization alone and its preprocessing adds another 68 / 333 s. Our time includes tracking, depth and camera estimation.

On the 51-frame windows (Aria, TUM and DyCheck), the PSNR differences are 0.3 dB or less: we are ahead on Aria and TUM and 0.27 dB behind on DyCheck. We are also better in LPIPS on all three datasets (Aria 0.119 vs 0.155, TUM 0.226 vs 0.280, DyCheck 0.370 vs 0.394). On the 12-frame NVIDIA and Pets clips, Shape of Motion is ahead by 0.9 and 1.2 dB, which accounts for its 0.4 dB lead on the five-dataset average.

In the revised paper: Section 4.1, Comparison to per-scene optimization, and Table 3.

Attribute Network over a strong track-initialized baseline (W4, Q3)

We compare the Attribute Network with per-window optimization of Gaussians placed on the same tracks. Optimization needs more than ten times our time to exceed our PSNR, stays worse in SSIM and LPIPS at every budget, and never reaches our quality on views from new cameras.

The baseline uses the same geometry, trajectories, cameras and input frames as our network. Gaussians are placed on the tracks and optimized on the window's input frames with an L1 + SSIM loss, as in our training, in two variants:

  • Attributes: all the attributes our network predicts (colour, SH, opacity, scale and rotation), with the centres fixed on the tracks.
  • Attributes + centres: the same attributes plus a per-frame offset on every centre, so that optimization can also correct the trajectories.

We evaluate all the datasets windows with the paper's Table 1 protocol.

Per window, one H200Time (s) ↓, 12 / 51 input framesPSNR ↑SSIM ↑LPIPS ↓
Ours, one forward pass2.4 / 6.623.620.7430.212
Attributes, 100 steps9 / 13 (×4 / ×2)21.740.6990.353
Attributes, 300 steps22 / 28 (×9 / ×4)22.720.7160.308
Attributes, 1,000 steps69 / 80 (×29 / ×12)23.510.7220.270
Attributes, 3,000 steps200 / 226 (×83 / ×34)24.280.7230.243
Attributes + centres, 100 steps9 / 15 (×4 / ×2)22.280.7070.335
Attributes + centres, 300 steps23 / 32 (×10 / ×5)23.530.7180.294
Attributes + centres, 1,000 steps70 / 92 (×29 / ×14)23.880.7110.278
Attributes + centres, 3,000 steps201 / 261 (×84 / ×40)23.920.6990.267

Times are per window on one H200 and include the front end shared by all rows (tracking, VGGT-Ω and lifting). The bracketed factor is how many times slower than ours each run is. NVIDIA and Pets use 12-frame windows. ADT, Aria, TUM and DyCheck use 51-frame windows.

Optimizing the attributes alone approaches our PSNR after 1,000 steps (12 to 29 times our time) and exceeds it by 0.7 dB after 3,000 steps (34 to 83 times our time). Also optimizing the centres approaches our PSNR sooner, after 300 steps (5 to 10 times our time), and exceeds it after 1,000 steps, but gains little beyond that. At every budget, both variants remain worse than ours in SSIM and LPIPS. On NVIDIA and DyCheck, whose held-out views come from separate cameras placed away from the input camera's path, neither variant reaches our PSNR at any budget: at best 18.97 and 16.29, against our 19.54 and 17.23. Ablations of the network's components (W4) are in our answer to Q4.

In the revised paper: Appendix E.3 and Table 9.

AuthorsOpenReview ↗

Response 2/4 to Reviewer 4CNU

Ablation on motion-graph refinement, coverage selection, and global attention (Q4, W4)

We trained each ablation from scratch under the Table 3 protocol (Table 4 in the revision), changing one component at a time. Global attention matters most: removing it costs 1.0 dB PSNR. The other two components serve efficiency rather than final quality: the motion graph speeds up training, and coverage selection removes 16% of the Gaussians with no loss in PSNR. We clarify their roles in the revision.

Global attention. We replaced it with self-only attention, so tracks no longer exchange information, and kept the parameter count the same. PSNR drops by 1.0 dB on average and by 0.4 to 1.5 dB on every dataset. SSIM and LPIPS also get worse.

Table 4 protocol, 100K updatesPSNR ↑SSIM ↑LPIPS ↓
Full model23.320.7300.161
No global attention22.320.7110.198

Motion graph. Without it, the model is 0.22 dB behind at 10K updates, mostly on the dynamic datasets (DyCheck, Pets and NVIDIA). The gap closes with longer training. The motion graph adds 1.6M parameters (3.5% of the network).

Coverage selection. Tracks started from different keyframes often cover the same surface. Coverage selection removes these duplicates. At 0.25 px, it reduces the Gaussians per window from 581k to 486k without changing PSNR. Larger radii remove more Gaussians but lower PSNR.

Coverage radius (Table 4 protocol, 20K updates)PSNR ↑Gaussians per window
Keep every Gaussian22.61581k
0.25 px (ours)22.62486k
0.5 px21.98203k
1 px21.68171k
2 px21.37150k

In the revised paper: Section 4.2, Further ablations, and Appendix E.2 with Table 8 (a, b).

Runtime and Efficiency, memory (W2, Q5)

Our method reconstructs a 12-frame window in 2.4 s, within about 2× of MoVieS, the strongest baseline, while scoring 3.1 dB higher in average PSNR. On the longer DyCheck windows, it is almost 4× faster than MoVieS (6.2 s vs 23.1 s, on the same 51-frame windows). There, only 4DGT reconstructs faster, but with rendering it is slower (7.7 s vs 7.2 s), and it is 4.5 dB lower in average PSNR. The 34–36 s in Table 6 (replaced by Tables 11 and 12 in the revision) was mostly evaluation I/O and start-up cost, which each stage of our evaluation code paid separately; all methods are now timed without it. Of the 2.4 s, more than half is spent in the frozen tracker, and our network takes about 0.6 s.

On NVIDIA, 11.4 s of the 36.3 s in Table 6 was evaluation I/O (an image-shape probe, PNG writing and setup). Most of the rest was GPU start-up: tracking, depth/camera estimation and reconstruction each ran in a fresh process and paid it separately. MoVieS (3.6 s) runs as a single model in one process, so it paid this cost once. We now time all methods the same way, with models loaded, removing startup cost. All methods are timed on the same H200 GPU (Table 6 used an H100), reporting the median of 7 runs after 2 warm-up runs. Reconstruction time is the time to produce the 4D Gaussians from the input frames, including tracking and depth/camera estimation.

Ours, per window, one H20012 input frames (NVIDIA, Pets)51 input frames (ADT, Aria, TUM, DyCheck)
AllTracker (s)1.414.00
VGGT-Ω (s)0.221.29
Lifting (s)0.190.71
Our network (s)0.580.60
Reconstruction time (s)2.46.6
Rendering (FPS)440343
Peak GPU memory (GiB)15.319.8

On the same 12-frame windows, MoVieS takes 1.2 s, since it receives ground-truth cameras and runs no separate tracking or depth estimation. Rendering takes about the same time for both methods. Peak GPU memory is 15–20 GiB with all models loaded. The quality–time trade-off against per-scene optimization is in our answer to W1.

In the revised paper: Appendix E.5 and Tables 11 and 12, which replace the earlier runtime table, summarized in Section 4.1.

AuthorsOpenReview ↗

Response 3/4 to Reviewer 4CNU

Held-out images influencing geometry (W3, Q2)

We re-ran all six benchmarks with VGGT-Ω seeing only the input frames, so no held-out image reaches any part of the reconstruction, and we remain more than 2 dB ahead of the best baseline. Because our method predicts its own cameras, each held-out camera must still be placed in our coordinate frame before it can be rendered; this alignment is needed only for evaluation and does not change the reconstruction.

In the paper, we placed the held-out cameras in our predicted frame by running VGGT-Ω jointly on the input and held-out frames, which is how the held-out images could reach our geometry. We now place each held-out camera in one of two ways, neither of which changes the reconstruction:

  • (A) Camera interpolation: each held-out frame along the input camera's path is rendered from a camera interpolated between our predicted cameras for the two neighbouring input frames. This uses no ground truth and no target pixels. Held-out views from a different physical camera (DyCheck's test cameras, NVIDIA's camera rig) cannot be interpolated, so their ground-truth pose is mapped into our frame by a similarity transform fitted on the input cameras, as in the paper's NVIDIA and Pets protocol.
  • (B) BARF camera alignment: the standard evaluation protocol for methods that predict their own cameras (BARF [2], NeRF-- [3], CF-3DGS [4]). Each held-out ground-truth pose is mapped into our frame by a similarity transform fitted on the input cameras. Since PSNR is very sensitive to small misalignment, the 6 pose parameters of each held-out camera are then refined against its image, with the reconstructed scene frozen.
PSNR ↑ADTAriaTUMDyCheckNVIDIAPetsAvg
MoVieS-64, best baseline (GT cameras)27.6622.8019.5815.2220.8723.3421.58
Ours, paper (joint VGGT-Ω)31.5627.4222.2517.2319.5423.7023.62
Ours, input-only, (A) camera interpolation33.1227.2521.4516.2219.5426.1723.96
Ours, input-only, (B) BARF camera alignment31.6326.4522.5317.3621.8928.1124.66

On ADT, Aria, TUM and DyCheck, input-only geometry lowers PSNR by about 0.1 dB on average; the rise in the average comes from NVIDIA and Pets, explained below. Without using any target pixels, (A) reaches 23.96, 2.4 dB above MoVieS-64. With (B), we reach 24.66 and lead MoVieS-64 on all six datasets.

On ADT, Aria, TUM and DyCheck, the paper's joint VGGT-Ω run could have passed information from the held-out images to our reconstruction. To test this, we kept the cameras of the joint run and replaced only its depth with input-only depth. PSNR was unchanged on ADT, Aria and TUM (26.86 with either depth, 12 windows) and changed by 0.2 dB or less on DyCheck (6 windows), so the held-out images had a negligible effect on our depth. The per-dataset differences in the table therefore come from the cameras rather than the depth. In particular, the paper estimated each held-out camera from its own image, while (A) places it without any target pixels, which costs 0.2–1.0 dB on Aria, TUM and DyCheck and gains 1.6 dB on ADT. With (B), all four datasets are within 1 dB of the paper, and above it on ADT, TUM and DyCheck.

On NVIDIA and Pets, the paper already ran VGGT-Ω on the input frames only, so the reconstruction is the same in all three of our rows and only the held-out cameras differ. The paper mapped their ground-truth poses into our frame by a similarity transform, which leaves a small registration error between two independently estimated camera trajectories. BARF alignment corrects it by a median of only 0.16° on NVIDIA and 0.33° on Pets, yet raises PSNR from 19.54 to 21.89 and from 23.70 to 28.11. On Pets, interpolating our own cameras avoids this error without any ground truth and reaches 26.17. NVIDIA's held-out views come from other cameras of its rig, so (A) uses the paper's mapping and the result is unchanged. The baselines receive ground-truth cameras and never pay this registration cost.

In the revised paper: Tables 1 and 2 now use input-only geometry with BARF camera alignment (Section 4, Protocol). Appendix D.2 and Table 5 compare the three ways of placing the held-out cameras and report the depth-only control.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their comments. The reviewers agreed that separating geometry and motion from appearance is well motivated, and that a single model generalizes across the six benchmarks. They raised three main concerns: whether the held-out images helped our geometry, the running time, and how our method compares with per-scene optimization. We summarize our answers below. Details are in the individual responses.

Held-out images (all reviewers). Our method predicts its own cameras, so each held-out camera must be placed in our coordinate frame before it can be rendered and scored; this step is needed only for evaluation. In the paper, we did this by running VGGT-Ω jointly on the input and held-out frames. We re-ran all six benchmarks with VGGT-Ω seeing only the input frames, so no held-out image reaches the reconstruction. With the standard BARF camera alignment, our average PSNR is 24.66, 3.1 dB above the best baseline, MoVieS (21.58), which receives ground-truth cameras. With camera interpolation, which uses no target pixels, it is 23.96, still 2.4 dB ahead. The rise over the paper’s 23.62 comes from NVIDIA and Pets, where the reconstruction is unchanged and only the held-out cameras are placed differently; on the other four datasets, input-only geometry lowers PSNR by about 0.1 dB on average. A controlled test also shows that the joint setup of the paper had a negligible effect on depth: replacing only the depth with input-only depth left PSNR unchanged.

Running time (4CNU, W3Hy). On short 12-frame windows, our running time is within about 2× of MoVieS, the strongest baseline; on the longer 51-frame DyCheck windows, we are almost 4× faster than MoVieS. There, only 4DGT reconstructs faster, and it is 4.5 dB lower in average PSNR; including rendering, we are the fastest of all methods. The submission's runtime table timed a single cold run of our evaluation code, in which evaluation I/O and the start-up of each pipeline stage in its own process made up most of the reported time. With models loaded, our method reconstructs a 12-frame window in 2.4 s and MoVieS in 1.2 s, and our PSNR is 3.1 dB higher on average.

Per-scene optimization (4CNU). We compared our method with Shape of Motion, a per-scene method. On 51-frame windows, the two are within 0.3 dB PSNR, and our method has better LPIPS on all three datasets. Shape of Motion is ahead on the short 12-frame clips. In all settings, it takes about 100 times longer. We also optimized Gaussians placed on our own tracks. This needs more than 10 times our time to exceed our PSNR, and remains worse in SSIM and LPIPS. So our network gets in one pass what optimization needs much longer to reach.

Other points. Removing global attention costs 1.0 dB. The motion graph speeds up training but does not change final quality. Coverage selection removes 16% of the Gaussians without loss (4CNU, GiWC). Our method also works with CoTracker3 instead of AllTracker, without retraining. With the same upsampling of the tracks, the two trackers differ by less than 0.1 dB (GiWC). The model runs on windows twice its training length, with a gradual drop of 1.7 dB (W3Hy, GiWC). An analysis of the errors shows that the main cost of fixed centres comes from tracks lost during occlusion (4CNU).

Changes to the paper. We uploaded a revised version, with all changes marked in blue. The main changes are:

  • Held-out cameras: Tables 1 and 2 now use input-only geometry with the standard BARF camera alignment (Section 4, Protocol). Appendix D.2 and Table 5 compare the three ways of placing the held-out cameras, including camera interpolation, and report the depth-only control.
  • Per-scene optimization: the comparison with Shape of Motion is in Section 4.1 (Comparison to per-scene optimization, Table 3), and per-window optimization from our tracks is in Appendix E.3 (Table 9).
  • Running time: the corrected timings are in Section 4.1 and Appendix E.5 (Tables 11 and 12), which replace the earlier runtime table.
  • Further ablations: global attention, the motion graph, coverage selection, the tracker swap and longer windows are summarized in Section 4.2 (Further ablations) and detailed in Appendix E.2 (Table 8).
  • Error analysis: in Appendix E.4 (Table 10), referenced from the limitations (Section 4.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.