- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses feed-forward 4D reconstruction from monocular video. It argues that per-pixel Gaussian prediction in existing feed-forward methods leads to ghosting under large motion and over-reliance on temporally nearby views. To address this, the proposed T4GS uses a compact set of learnable queries that aggregate VGGT features across all frames and are decoded into 3D Gaussians at a target time, so that the same query yields a Gaussian trajectory over time. The model is trained with a static-to-dynamic curriculum and additional pseudo-supervised geometry and motion losses. Experiments on four dynamic benchmarks show that T4GS outperforms 4DGT, MoVieS and NeoVerse on most metrics with far fewer Gaussians. The representation is further applied to camera-controllable video generation and 4D feature lifting.
Critical strengths
-
The paper is well written and easy to follow. The motivation is clearly stated, the failure modes of per-pixel feed-forward 4DGS are well illustrated, and the appendix provides thorough implementation and evaluation details.
-
The experiments and analyses are extensive. The paper evaluates on four dynamic benchmarks and includes point tracking, diagnostic analyses of opacity persistence and tracking consistency, computational cost comparisons, and ablations on time embedding, losses, token count and data curation. The tracking results on ADT, together with the attention visualizations in Fig. 4 showing that each query attends to consistent regions across target times, provide evidence of better motion consistency than per-pixel methods. The paper also explores two downstream applications, video generation and 4D feature lifting.
Critical weaknesses
-
The reviewer considers the novelty of this work to be quite limited. The Gaussian decoder, training setup, feature lifting and attention analysis largely follow C3G [1], and the only architectural addition is a relative time embedding. Extending a static model to dynamic scenes by adding time conditioning is an intuitive and fairly standard design, and similar ideas have been explored in prior work, such as the time-conditioned dynamic tokens in TokenGS [2], which is not cited. The main difference lies in where the time condition is injected, yet the paper neither compares with these prior designs nor provides sufficient analysis of why its choice is advantageous.
-
The reviewer has concerns about the fairness of the experimental comparison. The evaluation protocols on ADT, TUM-Dynamics and NVIDIA differ from those used in the baseline papers and closely match the paper's own training configuration in frame stride, number of context frames and resolution. The baselines are run off-the-shelf rather than retrained under the same conditions, and their numbers fall well below those reported in their original papers, by up to about 10 dB on ADT. In addition, no controlled comparison is provided to isolate the effect of the query-based representation.
-
The reviewer finds that several key claims are not sufficiently supported. The central claim that the same query decodes the same Gaussian across time is not directly validated. The tracking evaluations measure an alpha-blended motion field rather than individual Gaussians, and appearance attributes are left unconstrained over time. By contrast, TokenGS [2] directly visualizes the trajectory of each dynamic Gaussian across time as emergent scene flow, whereas this paper provides no such per-Gaussian evidence. Moreover, the claimed robustness to large temporal gaps is contradicted by Tab. 2.
[1] H. An, J. Jung, M. Kim, S. Hong, C. Kim, K. Fukuda, M. Jeon, J. Han, T. Narihira, H. Ko, J. Kim, Y. Mitsufuji, and S. Kim. C3G: Learning Compact 3D Representations with 2K Gaussians. CVPR, 2026.
[2] J. Ren, M. J. Tyszkiewicz, J. Huang, and Z. Gojcic. TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens. CVPR, 2026.
Questions for authors
-
After reading the full paper, the reviewer finds that, compared with C3G [1], the proposed method only adds a time embedding module , and its training pipeline appears to be C3G's training further extended with additional dynamic datasets. Could the authors clarify the differences from C3G in network architecture, training pipeline and settings?
-
Are the authors aware of TokenGS [2]? The reviewer considers it the most closely related work to this paper, yet it is neither cited nor compared.
-
The training and evaluation protocols in the experiments differ from those used by the compared methods, particularly on ADT, TUM-Dynamics and NVIDIA. Why did the authors choose these protocols, and in what sense are they more reasonable than the protocols used in prior work? Were the baselines evaluated with their publicly released models, or retrained under settings equivalent to those of the proposed method?
-
Regarding the Gaussians decoded from the same token at different timestamps, the reviewer understands that they should represent the same physical point in the scene and move with it as the object moves. However, the paper does not provide intuitive results to verify this. Could the authors visualize the trajectories of individual Gaussians over time, as TokenGS [2] does with its emergent scene flow visualization (Fig. 9 in TokenGS)?
-
The dynamic datasets used in the experiments appear to have short temporal spans, with relatively small motion and simple deformation patterns. Could the authors evaluate the method on datasets more commonly used for 4D reconstruction, such as N3DV [3], Panoptic Sports [4] and Technicolor [5]?
-
For more practical real-world scenarios such as sports games that may last tens of minutes, how could the proposed framework be extended? Given that it uses a fixed set of tokens and processes all context frames jointly, what kind of solution would the authors consider for such long sequences?
[1] H. An, J. Jung, M. Kim, S. Hong, C. Kim, K. Fukuda, M. Jeon, J. Han, T. Narihira, H. Ko, J. Kim, Y. Mitsufuji, and S. Kim. C3G: Learning Compact 3D Representations with 2K Gaussians. CVPR, 2026.
[2] J. Ren, M. J. Tyszkiewicz, J. Huang, and Z. Gojcic. TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens. CVPR, 2026.
[3] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, and Z. Lv. Neural 3D Video Synthesis from Multi-View Video. CVPR, 2022.
[4] J. Luiten, G. Kopanas, B. Leibe, and D. Ramanan. Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis. 3DV, 2024.
[5] N. Sabater, G. Boisson, B. Vandame, P. Kerbiriou, F. Babon, M. Hog, R. Gendrot, T. Langlois, O. Bureller, A. Schubert, and V. Allié. Dataset and Pipeline for Multi-View Light-Field Video. CVPR Workshops, 2017.
7 comments
Response to the Reviewer vUYm (2/4)
[W1/Q2] Are the authors aware of TokenGS [2]? The reviewer considers it the most closely related work to this paper, yet it is neither cited nor compared.
We thank the reviewer for pointing out TokenGS. We have cited TokenGS and included a comprehensive discussion and experimental comparison in our revised manuscript (§2, §4.2, Tab 1,2,3,4).
Differences in Motivation and Design (TokenGS vs. T4GS)
We would like to first clarify that TokenGS and T4GS focuses on different primary objectives, which leads to different architectural designs and training recipes. While both TokenGS and T4GS depart from pixel-aligned Gaussian prediction via learnable query tokens, they target fundamentally different objectives:
- Primary Objective: TokenGS focuses on novel-view synthesis, showing that query-based coordinate regression can replace pixel-aligned prediction in both static and dynamic scenes. In contrast, T4GS focuses on motion understanding: it formulates query-based decoding for dynamic scenes to overcome the failure modes of feed-forward 4D methods, namely ghosting and view-dependent opacity decay (Fig. 1). This motion-faithful representation enables three applications: novel-view synthesis, 4D feature lifting, and camera-controllable video generation.
- Motion Modeling & Training Recipe: A key advantage of query-based decoding is the ability to model scene motion continuously by decoding the same query across timestamps, bypassing brittle per-pixel 3D tracking. However, we observe that naively training a query-based network with photometric novel-view synthesis (NVS) loss alone fails to unlock this property: tokens still drift or alter opacity/color over time rather than tracking a physical surface (Fig. 7), also causing static regions to jitter. To resolve this, we introduce several recipes including relative feature-side time shifting (), a static-to-dynamic multi-view coverage curriculum, and explicit motion supervision (), whereas TokenGS simply adopts NVS loss as their primary supervision signal. Moreover, since it takes camera poses as input, it can easily separate camera motion from object motion and does not suffer from jittering in static regions.
- Application: We find that enforcing the query tokens to model motion faithfully is what enables T4GS to support downstream applications such as 4D feature lifting, which no prior feed-forward method has demonstrated due to the lack of consistent motion modeling. Even TokenGS, though query-based, does not ensure that its Gaussians follow consistent trajectories (Table R2, Fig. 7), so features lifted onto them would not remain consistent over time, and such applications were not demonstrated.
Quantitative Comparison with Dynamic TokenGS (NVS & 3D Tracking)
While TokenGS and T4GS focuses on different primary objectives as we explained, we agree with the reviewer that further comparing with TokenGS can better verify our strengths. We first compare with the official dynamic TokenGS model against T4GS on NVS across four dynamic benchmarks and on 3D/2D point tracking. Notably, TokenGS requires ground-truth camera poses as input and uses 81.9K Gaussians (1,024 static + 256 dynamic tokens 64 Gaussians/token), whereas T4GS is pose-free and uses only 2K Gaussians ( fewer).
Table R1: NVS Comparison
| Methods (PSNR↑ / SSIM↑ / LPIPS↓) | Pose | Dycheck | ADT | TUM-Dynamics | NVIDIA |
|---|---|---|---|---|---|
| TokenGS | O | 12.00 / 0.297 / 0.603 | 24.52 / 0.770 / 0.308 | 19.69 / 0.628 / 0.367 | 17.15 / 0.411 / 0.653 |
| Ours | X | 15.64 / 0.397 / 0.444 | 27.70 / 0.816 / 0.202 | 20.84 / 0.668 / 0.277 | 20.62 / 0.497 / 0.384 |
Table R2: 3D & 2D Tracking Evaluation
| Methods (↓ / ↑ / ↑ / ↑ / ↑) | ADT | Drive Track | Panoptic Studio |
|---|---|---|---|
| Movies | 0.518 / 18.5 / 33.8 / 51.9 / 67.0 | 0.072 / 45.7 / 74.6 / 37.9 / 55.9 | 0.0902 / 47.4 / 74.4 / 41.7 / 54.7 |
| TokenGS | 0.516 / 18.6 / 30.7 / 51.8 / 65.5 | 0.066 / 48.6 / 72.6 / 40.1 / 59.5 | 0.1164 / 36.3 / 61.7 / 42.0 / 56.6 |
| Ours | 0.459 / 20.1 / 40.2 / 63.5 / 78.1 | 0.063 / 50.1 / 72.6 / 43.3 / 64.8 | 0.0741 / 74.9 / 86.2 / 53.9 / 67.9 |
Despite using fewer Gaussians and operating without input camera poses, T4GS consistently outperforms TokenGS across all NVS and tracking benchmarks, validating the effectiveness of our motion-aware training recipe and compact bottleneck.
Response to the Reviewer vUYm (3/4)
[W2/Q3] The reviewer has concerns about the fairness of the experimental comparison. The evaluation protocols on ADT, TUM-Dynamics and NVIDIA differ from those used in the baseline papers and closely match the paper's own training configuration in frame stride, number of context frames and resolution. The baselines are run off-the-shelf rather than retrained under the same conditions, and their numbers fall well below those reported in their original papers, by up to about 10 dB on ADT. In addition, no controlled comparison is provided to isolate the effect of the query-based representation.
Since evaluation protocols differ across the compared methods, we built our protocol on NeoVerse with modifications suited to our goal. Given the scarcity of multi-view dynamic videos, we used NVIDIA to evaluate rendering from other cameras in a synchronized multi-view rig. On ADT and TUM-Dynamics, we used fewer context frames with a larger stride to better assess motion modeling. With dense, closely spaced frames, per-pixel methods can rely on near-duplicate neighboring views, so the metrics mostly reflect temporal interpolation rather than motion. All baselines were evaluated with their publicly released models.
That said, we agree that this protocol may appear to favor our method. We therefore additionally evaluate all methods under the original baseline protocols: 4DGT’s protocol on ADT and TUM-Dynamics, and MoVieS’s protocol on NVIDIA.
Table R3: NVS comparision with original evaluation protocol
| Methods (PSNR↑ / SSIM↑ / LPIPS↓) | ADT | TUM-Dynamics | NVIDIA |
|---|---|---|---|
| 4DGT | 29.29 / 0.874 / 0.132 | 19.76 / 0.662 / 0.275 | 16.37 / 0.350 / 0.568 |
| MoVieS | 23.87 / 0.701 / 0.241 | 15.64 / 0.455 / 0.407 | 19.21 / 0.497 / 0.330 |
| NeoVerse | 21.19 / 0.598 / 0.300 | 22.76 / 0.778 / 0.166 | 15.53 / 0.297 / 0.481 |
| TokenGS | 26.91 / 0.833 / 0.245 | 20.73 / 0.686 / 0.328 | 17.73 / 0.430 / 0.661 |
| Ours | 25.91 / 0.756 / 0.270 | 20.65 / 0.643 / 0.310 | 20.78 / 0.500 / 0.384 |
A direct per-pixel counterpart of our model is non-trivial, since per-pixel methods require additional mechanisms to model motion, such as velocity attributes (4DGT, NeoVerse) or deformation fields (MoVieS). Nevertheless, existing results already isolate the role of the representation to a large extent. MoVieS is trained with explicit flow supervision, similar to our 3D flow loss, yet its Gaussians still drift substantially over time and produce overlapping, duplicated Gaussians. Conversely, our variant trained without any motion loss still drifts far less than all per-pixel methods. This indicates that the gain comes from the query-based representation rather than from motion supervision. The comparison with TokenGS further isolates design choices within token-based models.
[W3/Q4] The reviewer finds that several key claims are not sufficiently supported. The central claim that the same query decodes the same Gaussian across time is not directly validated. The tracking evaluations measure an alpha-blended motion field rather than individual Gaussians, and appearance attributes are left unconstrained over time. By contrast, TokenGS [2] directly visualizes the trajectory of each dynamic Gaussian across time as emergent scene flow, whereas this paper provides no such per-Gaussian evidence. Moreover, the claimed robustness to large temporal gaps is contradicted by Tab. 2.
We thank the reviewer for this question. We added Gaussian-wise tracking visualizations for dynamic scenes, including iPhone paper-windmill, to the revised paper (§D.1, Fig. 7), where each Gaussian is colored by its token ID and traced across all frames. We also report 3D tracking metrics below:
Table R2: 3D & 2D Tracking Evaluation
| Methods (↓ / ↑ / ↑ / ↑ / ↑) | ADT | Drive Track | Panoptic Studio |
|---|---|---|---|
| Movies | 0.518 / 18.5 / 33.8 / 51.9 / 67.0 | 0.072 / 45.7 / 74.6 / 37.9 / 55.9 | 0.0902 / 47.4 / 74.4 / 41.7 / 54.7 |
| TokenGS | 0.516 / 18.6 / 30.7 / 51.8 / 65.5 | 0.066 / 48.6 / 72.6 / 40.1 / 59.5 | 0.1164 / 36.3 / 61.7 / 42.0 / 56.6 |
| Ours | 0.459 / 20.1 / 40.2 / 63.5 / 78.1 | 0.063 / 50.1 / 72.6 / 43.3 / 64.8 | 0.0741 / 74.9 / 86.2 / 53.9 / 67.9 |
These results show that our Gaussian queries faithfully follow the motion of dynamic parts over the full sequence, whereas many TokenGS Gaussians cover only part of each trajectory.
Response to the Reviewer vUYm (1/4)
We thank the reviewer for this comment and would like to clarify the novelty of our method compared to C3G and we have revised the paper for clarification of our difference (§3.4). Our primary goal is faithful motion modeling in feed-forward 4D reconstruction to overcome the core failure modes of feed-forward 4D methods (ghosting and view-dependent opacity decay, Fig. 1). We show that per-pixel methods learn temporal interpolation rather than motion, and adopt a compact query-based design to model the global motion of the Gaussian field. The differences from C3G follow from this goal.