- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TracksTo4DGS, a feed-forward framework for 4D reconstruction. It leverages frozen geometry and tracking pre-trained foundation models to obtain dense 3D trajectories, while a sparse-bottleneck network predicts Gaussian attributes for rendering. Detailed experiments demonstrating its effectiveness.
Critical strengths
-
Clear and well-motivated design. The decoupling of geometry and motion from appearance prediction provides a straightforward approach to feed-forward 4D reconstruction. The proposed method effectively utilizes off-the-shelf pre-trained foundation models and added simple yet effective components to achieve more resource-friendly and higher-quality 4D reconstruction.
-
Extensive experimental evaluation. Results across 6 benchmarks demonstrate competitive reconstruction quality and cross-dataset generalization, supported by ablation studies of the main components. The authors also provided detailed failure cases for discussion.
Critical weaknesses
-
Potential evaluation bias. On some benchmarks, held-out target images are jointly processed with input images by VGGT-Ω for camera prediction. Since VGGT-Ω performs joint geometric estimation, these target images may also introduce information leakage. An input-only evaluation would make the conclusion more convincing.
-
Dependence on upstream predictions. The location of Gaussian primitives are directly determined by frozen pre-trained geometry and tracking models, preventing the network from correcting inaccurate trajectories. Furthermore, the experiments primarily rely on a single combination of foundation models (VGGT-Ω and AllTracker). Evaluating alternative models (e.g. CoTracker3) would help assess the robustness and generalizability of the proposed framework.
-
Insufficient architectural ablations. While the paper provides detailed ablation study, the individual contributions of the global Transformer and motion graph are not sufficiently illustrated. Additional comparisons with simpler architectures or other variations would help to determine the necessity of these components.
Questions for authors
-
The paper claims that TracksTo4DGS can generalized to unseen video lengths, while it is trained on 51-frame sequences and evaluated on shorter sequences. Could the authors clarify whether the framework can directly process substantially longer videos, and how temporal consistency is maintained across different reconstruction windows?
-
Given the feed-forward nature of the proposed framework, do the authors see potential for extending it to streaming/online reconstruction? What would be the main challenges in adapting the current pipeline to an online setting?
-
The proposed model is trained exclusively on egocentric Aria. Could the authors elaborate on the motivation behind this choice? Would the cross-domain generalization be maintained if the model were trained on a different dataset? Furthermore, could the framework benefit from scaling up the training data?
6 comments
Response 2/4 to Reviewer GiWC
Dependence on upstream predictions (W2)
Our method depends on the frozen models, but not on one particular combination: it runs with CoTracker3 instead of AllTracker without retraining, losing only 0.35 dB PSNR, and with Depth Anything 3 instead of VGGT-Ω. Letting the network or per-window optimization correct the trajectories did not improve the final quality. The main cost of fixed centres comes from tracks lost during occlusion, which better trackers would reduce.
Correcting inaccurate trajectories. We tested whether correcting the trajectories improves the results. Early experiments showed that letting the network predict corrections to the tracks did not improve over fixed tracks. We also optimized the Gaussians of each window directly, starting from our tracks, with and without moving the centres (see response to Reviewer 4CNU, W4/Q3). At convergence, moving the centres gives 23.92 PSNR, versus 24.28 when they stay fixed. To see where the fixed centres cost most, we compared rendering quality across image regions. The main error comes from tracks lost during occlusion, where centres are interpolated rather than observed: about 14% of pixels, about 5 dB below regions with observed tracks. Better completion of lost tracks is therefore the most promising direction for future work.
Other geometry model. Table 3 (Table 4 in the revision) swaps VGGT-Ω for Depth Anything 3. This lowers the average PSNR (21.98 vs. 23.32) but improves Pets (25.61 vs. 24.00), so the method also benefits when a geometry model is better suited to a domain.
Other tracker. We replaced AllTracker with CoTracker3 at inference only. The network never saw CoTracker3 tracks in training. Everything else is as in Table 1 (cameras, depth, checkpoint, scoring, all 790 windows). CoTracker3 tracks the same grid of points as AllTracker (every 8 pixels on each keyframe), but it has no upsampling to full resolution, so we upsample its tracks bilinearly. As a control, we also run AllTracker with the same bilinear upsampling.
| PSNR ↑ | ADT | Aria | TUM | DyCheck | NVIDIA | Pets | Avg |
|---|---|---|---|---|---|---|---|
| AllTracker (paper's Table 1) | 31.56 | 27.42 | 22.25 | 17.23 | 19.54 | 23.70 | 23.62 |
| AllTracker, bilinear upsampling | 31.00 | 27.12 | 21.80 | 17.12 | 19.41 | 23.62 | 23.34 |
| CoTracker3, bilinear upsampling | 30.81 | 26.95 | 21.95 | 16.93 | 19.16 | 23.80 | 23.27 |
CoTracker3 stays 1.7 dB above the best baseline (MoVieS, 21.58). With the same upsampling, the two trackers differ by less than 0.1 dB, and CoTracker3 is better on TUM and Pets. Since the Gaussian centres come directly from the tracker, better trackers can improve our method without retraining.
In the revised paper: tracker swap in Section 4.2 (Further ablations) and Appendix E.2 with Table 8 (c); trajectory correction by optimization in Appendix E.3; error analysis in Appendix E.4, referenced from the limitations (Section 4.3).
Insufficient architectural ablations, for the Global Transformer and the Motion Graph. (W3)
We trained each ablation from scratch under the Table 3 protocol (Table 4 in the revision), changing one component at a time. The global transformer (attention) matters most: replacing its attention with self-only attention costs 1.0 dB PSNR. The motion graph serves efficiency rather than final quality: it speeds up training at 3.5% of the parameters. We clarify its role in the revision.
Global transformer (attention). We replaced its attention with self-only attention, so tracks no longer exchange information, and kept the parameter count the same. PSNR drops by 1.0 dB on average and by 0.4 to 1.5 dB on every dataset. SSIM and LPIPS also get worse.
| Table 4 protocol, 100K updates | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|
| Full model | 23.32 | 0.730 | 0.161 |
| No global attention | 22.32 | 0.711 | 0.198 |
Motion graph. The motion graph speeds up convergence, especially on dynamic scenes. It gives the network explicitly what global attention would otherwise have to learn: which seeds lie on the same surface and move together. At 10K updates, the model without it is 0.22 dB behind, with most of the gap on DyCheck, Pets and NVIDIA, the most dynamic datasets. With longer training, the model without it catches up. The graph costs only 1.6M parameters (3.5% of the network).
In the revised paper: Section 4.2, Further ablations, and Appendix E.2 with Table 8 (a).
Response 3/4 to Reviewer GiWC
Longer videos and temporal consistency across windows (Q1)
Without retraining, the model directly processes windows of up to 101 input frames, twice its training length, with only a gradual quality drop, and longer videos are processed as consecutive windows. Within a window, each Gaussian keeps one identity over all frames. Across windows, our method currently has no mechanism that relates the reconstructions.
Longer videos. In Table 1, windows on the 4DGT benchmarks have 51 input frames, the training length, and windows on NVIDIA and Pets have 12. A longer video is split into consecutive windows that are reconstructed independently, as in our evaluation protocol (Section 4). To test longer windows, we ran the model on ADT, Aria, TUM and DyCheck with up to 101 input frames per window, rescaling the frame and keyframe embeddings to their training range, and scored the same targets at every length. Values differ from Table 1 because DyCheck's novel views are excluded, as are NVIDIA and Pets, whose clips are too short for these window lengths.
| Input frames per window | 51 (training) | 64 | 75 | 101 |
|---|---|---|---|---|
| PSNR ↑ | 26.18 | 25.64 | 25.17 | 24.52 |
| Time per window | ×1.0 | ×1.6 | ×2.1 | ×3.4 |
| Peak GPU memory | ×1.0 | ×1.4 | ×1.7 | ×2.6 |
Quality drops by 0.5 dB at 64 input frames and 1.7 dB at 101, yet even at 101 input frames our method remains above every baseline on the same targets. Training on longer windows could reduce the quality drop.
Temporal consistency. Within a window, VGGT-Ω predicts all frames jointly, in one coordinate frame. Tracks start every 10 frames but run both forward and backward over the whole window, so each Gaussian persists across the full window with fixed attributes, and there are no seams at keyframes. Each window, however, is reconstructed independently. Linking windows is a natural extension that we leave to future work: Gaussians could be kept across windows in a shared coordinate frame, with new ones added only where the scene is not yet covered, as coverage selection already does within a window.
In the revised paper: longer windows in Section 4.2 (Further ablations) and Appendix E.2 with Table 8 (d); reconstructing windows independently is discussed in the limitations (Section 4.3).
Extending it to streaming/online reconstruction (Q2)
We believe our method could be extended to streaming reconstruction, since it is feed-forward and produces Gaussians that render in real time. The main challenge is that it currently relies on non-causal geometry and tracking models, which could be replaced by causal ones, as our pipeline already works with other backbones.
The reconstruction itself already suits an online setting: there is no per-scene optimization, the model already works on short windows (12 frames on NVIDIA and Pets), and its Gaussians render at over 300 FPS. The challenge lies in the frozen models that provide geometry and tracks. VGGT-Ω estimates all frames of a window jointly, and AllTracker tracks each keyframe both forward and backward in time, so both rely on future frames. Streaming would require causal, low-latency models in their place: for example, CUT3R [5], which updates its reconstruction one frame at a time, for geometry, and the online mode of CoTracker3 [4] for tracking. Latency matters as well: a 12-frame window currently takes 2.4 s on one H200, more than half of it in the tracker. Our pipeline is not tied to particular backbones. It already runs with CoTracker3 instead of AllTracker without retraining (W2 above), and with Depth Anything 3 instead of VGGT-Ω (Table 3; Table 4 in the revision). Replacing them with causal, low-latency models is therefore a natural path to streaming reconstruction, and an interesting direction for future work.
Response 1/4 to Reviewer GiWC
Potential evaluation bias (W1)
We re-ran all six benchmarks with VGGT-Ω seeing only the input frames, so no held-out image reaches any part of the reconstruction, and we remain more than 2 dB ahead of the best baseline. Because our method predicts its own cameras, each held-out camera must still be placed in our coordinate frame before it can be rendered; this alignment is needed only for evaluation and does not change the reconstruction.
In the paper, we placed the held-out cameras in our predicted frame by running VGGT-Ω jointly on the input and held-out frames, which is how the held-out images could reach our geometry. We now place each held-out camera in one of two ways, neither of which changes the reconstruction:
On ADT, Aria, TUM and DyCheck, input-only geometry lowers PSNR by about 0.1 dB on average; the rise in the average comes from NVIDIA and Pets, explained below. Without using any target pixels, (A) reaches 23.96, 2.4 dB above MoVieS-64. With (B), we reach 24.66 and lead MoVieS-64 on all six datasets.
On ADT, Aria, TUM and DyCheck, the paper's joint VGGT-Ω run could have passed information from the held-out images to our reconstruction. To test this, we kept the cameras of the joint run and replaced only its depth with input-only depth. PSNR was unchanged on ADT, Aria and TUM (26.86 with either depth, 12 windows) and changed by 0.2 dB or less on DyCheck (6 windows), so the held-out images had a negligible effect on our depth. The per-dataset differences in the table therefore come from the cameras rather than the depth. In particular, the paper estimated each held-out camera from its own image, while (A) places it without any target pixels, which costs 0.2–1.0 dB on Aria, TUM and DyCheck and gains 1.6 dB on ADT. With (B), all four datasets are within 1 dB of the paper, and above it on ADT, TUM and DyCheck.
On NVIDIA and Pets, the paper already ran VGGT-Ω on the input frames only, so the reconstruction is the same in all three of our rows and only the held-out cameras differ. The paper mapped their ground-truth poses into our frame by a similarity transform, which leaves a small registration error between two independently estimated camera trajectories. BARF alignment corrects it by a median of only 0.16° on NVIDIA and 0.33° on Pets, yet raises PSNR from 19.54 to 21.89 and from 23.70 to 28.11. On Pets, interpolating our own cameras avoids this error without any ground truth and reaches 26.17. NVIDIA's held-out views come from other cameras of its rig, so (A) uses the paper's mapping and the result is unchanged. The baselines receive ground-truth cameras and never pay this registration cost.
In the revised paper: Tables 1 and 2 now use input-only geometry with BARF camera alignment (Section 4, Protocol). Appendix D.2 and Table 5 compare the three ways of placing the held-out cameras and report the depth-only control.