acceptodds
Under review as a conference paper at ICLR 2027

Learning Temporally-Aware Compact Gaussians for Feed-Forward 4D Reconstruction

10 views

Abstract

Dynamic scene reconstruction from monocular video remains a fundamental challenge in computer vision. Existing feed-forward methods predict per-pixel 3D Gaussians for each frame, leading to duplicated Gaussians and a view-dependent bias that hinder learning scene motion. We present Temporally-Aware Compact 4D Gaussian Splatting (T4GS), a feed-forward 4D reconstruction framework built upon a compact set of timestamp-conditioned learnable query tokens. Each token aggregates features across the full temporal context and decodes a 3D Gaussian whose position is modulated by the target timestamp, enabling globally coherent motion modeling. The resulting temporally coherent Gaussian representation serves as a motion-aware anchor for camera-controllable video generation and supports 4D feature lifting for point tracking and dynamic scene understanding. Across multiple dynamic benchmarks, T4GS achieves strong novel-view synthesis without input camera poses while using significantly fewer Gaussians and remaining robust to large temporal gaps.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D/4D Vision, Gaussian Splatting, Novel View Synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vUYmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses feed-forward 4D reconstruction from monocular video. It argues that per-pixel Gaussian prediction in existing feed-forward methods leads to ghosting under large motion and over-reliance on temporally nearby views. To address this, the proposed T4GS uses a compact set of learnable queries that aggregate VGGT features across all frames and are decoded into 3D Gaussians at a target time, so that the same query yields a Gaussian trajectory over time. The model is trained with a static-to-dynamic curriculum and additional pseudo-supervised geometry and motion losses. Experiments on four dynamic benchmarks show that T4GS outperforms 4DGT, MoVieS and NeoVerse on most metrics with far fewer Gaussians. The representation is further applied to camera-controllable video generation and 4D feature lifting.

Critical strengths
  1. The paper is well written and easy to follow. The motivation is clearly stated, the failure modes of per-pixel feed-forward 4DGS are well illustrated, and the appendix provides thorough implementation and evaluation details.

  2. The experiments and analyses are extensive. The paper evaluates on four dynamic benchmarks and includes point tracking, diagnostic analyses of opacity persistence and tracking consistency, computational cost comparisons, and ablations on time embedding, losses, token count and data curation. The tracking results on ADT, together with the attention visualizations in Fig. 4 showing that each query attends to consistent regions across target times, provide evidence of better motion consistency than per-pixel methods. The paper also explores two downstream applications, video generation and 4D feature lifting.

Critical weaknesses
  1. The reviewer considers the novelty of this work to be quite limited. The Gaussian decoder, training setup, feature lifting and attention analysis largely follow C3G [1], and the only architectural addition is a relative time embedding. Extending a static model to dynamic scenes by adding time conditioning is an intuitive and fairly standard design, and similar ideas have been explored in prior work, such as the time-conditioned dynamic tokens in TokenGS [2], which is not cited. The main difference lies in where the time condition is injected, yet the paper neither compares with these prior designs nor provides sufficient analysis of why its choice is advantageous.

  2. The reviewer has concerns about the fairness of the experimental comparison. The evaluation protocols on ADT, TUM-Dynamics and NVIDIA differ from those used in the baseline papers and closely match the paper's own training configuration in frame stride, number of context frames and resolution. The baselines are run off-the-shelf rather than retrained under the same conditions, and their numbers fall well below those reported in their original papers, by up to about 10 dB on ADT. In addition, no controlled comparison is provided to isolate the effect of the query-based representation.

  3. The reviewer finds that several key claims are not sufficiently supported. The central claim that the same query decodes the same Gaussian across time is not directly validated. The tracking evaluations measure an alpha-blended motion field rather than individual Gaussians, and appearance attributes are left unconstrained over time. By contrast, TokenGS [2] directly visualizes the trajectory of each dynamic Gaussian across time as emergent scene flow, whereas this paper provides no such per-Gaussian evidence. Moreover, the claimed robustness to large temporal gaps is contradicted by Tab. 2.

[1] H. An, J. Jung, M. Kim, S. Hong, C. Kim, K. Fukuda, M. Jeon, J. Han, T. Narihira, H. Ko, J. Kim, Y. Mitsufuji, and S. Kim. C3G: Learning Compact 3D Representations with 2K Gaussians. CVPR, 2026.

[2] J. Ren, M. J. Tyszkiewicz, J. Huang, and Z. Gojcic. TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens. CVPR, 2026.

Questions for authors
  1. After reading the full paper, the reviewer finds that, compared with C3G [1], the proposed method only adds a time embedding module H(ψ(Δt))\mathcal{H}(\psi(\Delta t)), and its training pipeline appears to be C3G's training further extended with additional dynamic datasets. Could the authors clarify the differences from C3G in network architecture, training pipeline and settings?

  2. Are the authors aware of TokenGS [2]? The reviewer considers it the most closely related work to this paper, yet it is neither cited nor compared.

  3. The training and evaluation protocols in the experiments differ from those used by the compared methods, particularly on ADT, TUM-Dynamics and NVIDIA. Why did the authors choose these protocols, and in what sense are they more reasonable than the protocols used in prior work? Were the baselines evaluated with their publicly released models, or retrained under settings equivalent to those of the proposed method?

  4. Regarding the Gaussians decoded from the same token at different timestamps, the reviewer understands that they should represent the same physical point in the scene and move with it as the object moves. However, the paper does not provide intuitive results to verify this. Could the authors visualize the trajectories of individual Gaussians over time, as TokenGS [2] does with its emergent scene flow visualization (Fig. 9 in TokenGS)?

  5. The dynamic datasets used in the experiments appear to have short temporal spans, with relatively small motion and simple deformation patterns. Could the authors evaluate the method on datasets more commonly used for 4D reconstruction, such as N3DV [3], Panoptic Sports [4] and Technicolor [5]?

  6. For more practical real-world scenarios such as sports games that may last tens of minutes, how could the proposed framework be extended? Given that it uses a fixed set of tokens and processes all context frames jointly, what kind of solution would the authors consider for such long sequences?

[1] H. An, J. Jung, M. Kim, S. Hong, C. Kim, K. Fukuda, M. Jeon, J. Han, T. Narihira, H. Ko, J. Kim, Y. Mitsufuji, and S. Kim. C3G: Learning Compact 3D Representations with 2K Gaussians. CVPR, 2026.

[2] J. Ren, M. J. Tyszkiewicz, J. Huang, and Z. Gojcic. TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens. CVPR, 2026.

[3] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, and Z. Lv. Neural 3D Video Synthesis from Multi-View Video. CVPR, 2022.

[4] J. Luiten, G. Kopanas, B. Leibe, and D. Ramanan. Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis. 3DV, 2024.

[5] N. Sabater, G. Boisson, B. Vandame, P. Kerbiriou, F. Babon, M. Hog, R. Gendrot, T. Langlois, O. Bureller, A. Schubert, and V. Allié. Dataset and Pipeline for Multi-View Light-Field Video. CVPR Workshops, 2017.

7 comments
AuthorsOpenReview ↗

Response to the Reviewer vUYm (1/4)

[W1/Q1] After reading the full paper, the reviewer finds that, compared with C3G [1], the proposed method only adds a time embedding module, and its training pipeline appears to be C3G's training further extended with additional dynamic datasets. Could the authors clarify the differences from C3G in network architecture, training pipeline and settings?

We thank the reviewer for this comment and would like to clarify the novelty of our method compared to C3G and we have revised the paper for clarification of our difference (§3.4). Our primary goal is faithful motion modeling in feed-forward 4D reconstruction to overcome the core failure modes of feed-forward 4D methods (ghosting and view-dependent opacity decay, Fig. 1). We show that per-pixel methods learn temporal interpolation rather than motion, and adopt a compact query-based design to model the global motion of the Gaussian field. The differences from C3G follow from this goal.

  • Training: Naively extending C3G to dynamic data is non-trivial for two reasons. First, we observe that naively training it with the photometric NVS loss alone does not make each query consistently track the same region over time. Queries tend to change the color and opacity of their Gaussians over time, or drift away from the physical surface, rather than tracking it (Fig. 7). Our explicit motion supervision (Ltrack\mathcal{L}_\text{track}, Lflow3D\mathcal{L}^\text{3D}_\text{flow}) prevent this by forcing dynamic regions to be explained by Gaussian motion, while the static term keeps static Gaussians fixed. In the pose-free setting, the model must also disentangle camera motion from object motion on its own, which causes static regions to jitter and training even collapses without static pre-training (Tab. 14). We address this through our training curriculum: static pre-training first anchors the static structure, and multi-view data with coverage sampling then pins each Gaussian to a geometrically consistent position. Static loss term also helps to reduce the jittering.
  • Architecture. Unlike bullet-time approaches, which condition the full pipeline on the target time and thus produce flickering Gaussians with no correspondence across timestamps, we condition time only at the decoder, so each new timestamp requires only a lightweight decoder pass. Specifically, we apply a relative time embedding to the image features while keeping the query embedding fixed. Since the input to each query stays identical across target times, each token decodes the same Gaussian at every timestamp, which directly yields its trajectory. This choice matters: absolute or scaled time embeddings substantially degrade tracking (Tab. 7, ADT PCK 24.9/26.0 vs. 34.2). Likewise, our feature decoder lifts time-conditioned 4D features rather than static 3D ones.
AuthorsOpenReview ↗

Response to the Reviewer vUYm (2/4)

[W1/Q2] Are the authors aware of TokenGS [2]? The reviewer considers it the most closely related work to this paper, yet it is neither cited nor compared.

We thank the reviewer for pointing out TokenGS. We have cited TokenGS and included a comprehensive discussion and experimental comparison in our revised manuscript (§2, §4.2, Tab 1,2,3,4).

Differences in Motivation and Design (TokenGS vs. T4GS)

We would like to first clarify that TokenGS and T4GS focuses on different primary objectives, which leads to different architectural designs and training recipes. While both TokenGS and T4GS depart from pixel-aligned Gaussian prediction via learnable query tokens, they target fundamentally different objectives:

  • Primary Objective: TokenGS focuses on novel-view synthesis, showing that query-based coordinate regression can replace pixel-aligned prediction in both static and dynamic scenes. In contrast, T4GS focuses on motion understanding: it formulates query-based decoding for dynamic scenes to overcome the failure modes of feed-forward 4D methods, namely ghosting and view-dependent opacity decay (Fig. 1). This motion-faithful representation enables three applications: novel-view synthesis, 4D feature lifting, and camera-controllable video generation.
  • Motion Modeling & Training Recipe: A key advantage of query-based decoding is the ability to model scene motion continuously by decoding the same query across timestamps, bypassing brittle per-pixel 3D tracking. However, we observe that naively training a query-based network with photometric novel-view synthesis (NVS) loss alone fails to unlock this property: tokens still drift or alter opacity/color over time rather than tracking a physical surface (Fig. 7), also causing static regions to jitter. To resolve this, we introduce several recipes including relative feature-side time shifting (Δt=t−tb\Delta t = t - t_b), a static-to-dynamic multi-view coverage curriculum, and explicit motion supervision (Ltrack,Lflow3D\mathcal{L}_{track}, \mathcal{L}_{flow}^{3D}), whereas TokenGS simply adopts NVS loss as their primary supervision signal. Moreover, since it takes camera poses as input, it can easily separate camera motion from object motion and does not suffer from jittering in static regions.
  • Application: We find that enforcing the query tokens to model motion faithfully is what enables T4GS to support downstream applications such as 4D feature lifting, which no prior feed-forward method has demonstrated due to the lack of consistent motion modeling. Even TokenGS, though query-based, does not ensure that its Gaussians follow consistent trajectories (Table R2, Fig. 7), so features lifted onto them would not remain consistent over time, and such applications were not demonstrated.

Quantitative Comparison with Dynamic TokenGS (NVS & 3D Tracking)

While TokenGS and T4GS focuses on different primary objectives as we explained, we agree with the reviewer that further comparing with TokenGS can better verify our strengths. We first compare with the official dynamic TokenGS model against T4GS on NVS across four dynamic benchmarks and on 3D/2D point tracking. Notably, TokenGS requires ground-truth camera poses as input and uses 81.9K Gaussians (1,024 static + 256 dynamic tokens ×\times 64 Gaussians/token), whereas T4GS is pose-free and uses only 2K Gaussians (40×40\times fewer).

Table R1: NVS Comparison

Methods (PSNR↑ / SSIM↑ / LPIPS↓)PoseDycheckADTTUM-DynamicsNVIDIA
TokenGSO12.00 / 0.297 / 0.60324.52 / 0.770 / 0.30819.69 / 0.628 / 0.36717.15 / 0.411 / 0.653
OursX15.64 / 0.397 / 0.44427.70 / 0.816 / 0.20220.84 / 0.668 / 0.27720.62 / 0.497 / 0.384

Table R2: 3D & 2D Tracking Evaluation

Methods (EPE3D\text{EPE}^\text{3D}↓ / δ0.053D\delta_{0.05}^\text{3D}↑ / δ0.103D\delta_{0.10}^\text{3D}↑ / AJ2D\text{AJ}^\text{2D}↑ / δavg2D\delta^\text{2D}_\text{avg}↑)ADTDrive TrackPanoptic Studio
Movies0.518 / 18.5 / 33.8 / 51.9 / 67.00.072 / 45.7 / 74.6 / 37.9 / 55.90.0902 / 47.4 / 74.4 / 41.7 / 54.7
TokenGS0.516 / 18.6 / 30.7 / 51.8 / 65.50.066 / 48.6 / 72.6 / 40.1 / 59.50.1164 / 36.3 / 61.7 / 42.0 / 56.6
Ours0.459 / 20.1 / 40.2 / 63.5 / 78.10.063 / 50.1 / 72.6 / 43.3 / 64.80.0741 / 74.9 / 86.2 / 53.9 / 67.9

Despite using 40×40\times fewer Gaussians and operating without input camera poses, T4GS consistently outperforms TokenGS across all NVS and tracking benchmarks, validating the effectiveness of our motion-aware training recipe and compact bottleneck.

AuthorsOpenReview ↗

Response to the Reviewer vUYm (3/4)

[W2/Q3] The reviewer has concerns about the fairness of the experimental comparison. The evaluation protocols on ADT, TUM-Dynamics and NVIDIA differ from those used in the baseline papers and closely match the paper's own training configuration in frame stride, number of context frames and resolution. The baselines are run off-the-shelf rather than retrained under the same conditions, and their numbers fall well below those reported in their original papers, by up to about 10 dB on ADT. In addition, no controlled comparison is provided to isolate the effect of the query-based representation.

Since evaluation protocols differ across the compared methods, we built our protocol on NeoVerse with modifications suited to our goal. Given the scarcity of multi-view dynamic videos, we used NVIDIA to evaluate rendering from other cameras in a synchronized multi-view rig. On ADT and TUM-Dynamics, we used fewer context frames with a larger stride to better assess motion modeling. With dense, closely spaced frames, per-pixel methods can rely on near-duplicate neighboring views, so the metrics mostly reflect temporal interpolation rather than motion. All baselines were evaluated with their publicly released models.

That said, we agree that this protocol may appear to favor our method. We therefore additionally evaluate all methods under the original baseline protocols: 4DGT’s protocol on ADT and TUM-Dynamics, and MoVieS’s protocol on NVIDIA.

Table R3: NVS comparision with original evaluation protocol

Methods (PSNR↑ / SSIM↑ / LPIPS↓)ADTTUM-DynamicsNVIDIA
4DGT29.29 / 0.874 / 0.13219.76 / 0.662 / 0.27516.37 / 0.350 / 0.568
MoVieS23.87 / 0.701 / 0.24115.64 / 0.455 / 0.40719.21 / 0.497 / 0.330
NeoVerse21.19 / 0.598 / 0.30022.76 / 0.778 / 0.16615.53 / 0.297 / 0.481
TokenGS26.91 / 0.833 / 0.24520.73 / 0.686 / 0.32817.73 / 0.430 / 0.661
Ours25.91 / 0.756 / 0.27020.65 / 0.643 / 0.31020.78 / 0.500 / 0.384

A direct per-pixel counterpart of our model is non-trivial, since per-pixel methods require additional mechanisms to model motion, such as velocity attributes (4DGT, NeoVerse) or deformation fields (MoVieS). Nevertheless, existing results already isolate the role of the representation to a large extent. MoVieS is trained with explicit flow supervision, similar to our 3D flow loss, yet its Gaussians still drift substantially over time and produce overlapping, duplicated Gaussians. Conversely, our variant trained without any motion loss still drifts far less than all per-pixel methods. This indicates that the gain comes from the query-based representation rather than from motion supervision. The comparison with TokenGS further isolates design choices within token-based models.

[W3/Q4] The reviewer finds that several key claims are not sufficiently supported. The central claim that the same query decodes the same Gaussian across time is not directly validated. The tracking evaluations measure an alpha-blended motion field rather than individual Gaussians, and appearance attributes are left unconstrained over time. By contrast, TokenGS [2] directly visualizes the trajectory of each dynamic Gaussian across time as emergent scene flow, whereas this paper provides no such per-Gaussian evidence. Moreover, the claimed robustness to large temporal gaps is contradicted by Tab. 2.

We thank the reviewer for this question. We added Gaussian-wise tracking visualizations for dynamic scenes, including iPhone paper-windmill, to the revised paper (§D.1, Fig. 7), where each Gaussian is colored by its token ID and traced across all frames. We also report 3D tracking metrics below:

Table R2: 3D & 2D Tracking Evaluation

Methods (EPE3D\text{EPE}^\text{3D}↓ / δ0.053D\delta_{0.05}^\text{3D}↑ / δ0.103D\delta_{0.10}^\text{3D}↑ / AJ2D\text{AJ}^\text{2D}↑ / δavg2D\delta^\text{2D}_\text{avg}↑)ADTDrive TrackPanoptic Studio
Movies0.518 / 18.5 / 33.8 / 51.9 / 67.00.072 / 45.7 / 74.6 / 37.9 / 55.90.0902 / 47.4 / 74.4 / 41.7 / 54.7
TokenGS0.516 / 18.6 / 30.7 / 51.8 / 65.50.066 / 48.6 / 72.6 / 40.1 / 59.50.1164 / 36.3 / 61.7 / 42.0 / 56.6
Ours0.459 / 20.1 / 40.2 / 63.5 / 78.10.063 / 50.1 / 72.6 / 43.3 / 64.80.0741 / 74.9 / 86.2 / 53.9 / 67.9

These results show that our Gaussian queries faithfully follow the motion of dynamic parts over the full sequence, whereas many TokenGS Gaussians cover only part of each trajectory.

Reviewer_wqS7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper point out two failure modes of the per-pixel feedforward 4DGS: ghosting under large temporal motion and view-dependent bias issue on the temporally closest views. To solve this issue, the paper builds a fixed number Gaussian persisting in all frames on top of a VGGT image encoder and a two transformer layers as a decoder. Training separating static stages and static-and-dynamic stages for stability. During inference, the model can decode the fixed number of Gaussian frames by frames. Besides, the model also investigate the downstream applications including camera-controllable VDM and 4D feature lifting for tracking and dynamic scene understanding.

Critical strengths
  1. This paper clearly discuss the drawbacks of the current per-pixel feedforward 4DGS and propose a effective fix to solve the opacity persistence and tracking consistency issues. To avoid the flickering issues, the decoder-only timestamp injection and static pre-training deliver a stable static Gaussians across frames.
  2. The any-feature 4D lifting comes out naturally and is cheap. The attention maps is reused for 4D feature lifting to be applied on top of an additional feature decoder. Since this feature consistently aggregates knowledge across frames and contain global motion information, the features perform better than baselines in Table 5 and 6.
Critical weaknesses
  1. [Major problem] The model trades details with temporally-aware compact Gaussian representation. The 2048 Gaussians inherently can not represent fine-grained details in the bounded scene and the demos show the blurs compared with other baselines. More importantly, the ablation studies show increasing N from 2048 to 4096 does not improve image quality and make tracking performance worse, which means the proposed method sets the cap for visual quality. Since the method is not scalable, the 4D Gaussians can not be directly used for high-fidelity NVS.
  2. Although combining with a video model can help improve the results, the SOTA video models can already handle the NVS without the additional features proposed in this paper. As a result, this design just gives an additional solution instead of solving the unsolved problem in video model, which limits the novelty for the downstream application.
  3. From the demo, it seems a Gaussian may not keep the geometry consistency across the whole frames. To be specific, the dynamic Gaussian may represent different physical part at different frames instead of serving as a real physical part. Also, the static region still have obvious shifting across time frames. Could the author gives the Gaussian-wise tracking visualization for dynamic cases such as iphone-paper-windmill?
Questions for authors

Besides of the question in Weakness section, I have additional questions:

  1. The evaluation views seem to be close to the input view with a small shift. Do the model output also perform well when the rendering pose has a large shift from the input view? Since the model has already been trained on the multi-view dataset, do this capability emerge under the current design?
  2. The inference input has different distribution with the training ones. In Eq3, the context (images) from other videos are inputted as context as well but in the inference, the model takes the monocular context. Do this difference cause any issues?
6 comments
AuthorsOpenReview ↗

Response to the Reviewer wqS7 (1/2)

[W1] [Major problem] The model trades details with temporally-aware compact Gaussian representation. The 2048 Gaussians inherently can not represent fine-grained details in the bounded scene and the demos show the blurs compared with other baselines. More importantly, the ablation studies show increasing N from 2048 to 4096 does not improve image quality and make tracking performance worse, which means the proposed method sets the cap for visual quality. Since the method is not scalable, the 4D Gaussians can not be directly used for high-fidelity NVS.

We thank the reviewer for this suggestion. We agree that, in our original ablation (Tab. 13 in original paper), increasing the number of Gaussians did not improve rendering quality as expected. Upon investigation, we found that this was due to our initialization: when increasing the number of tokens from a model trained with 2,048 tokens, we randomly initialized the additional tokens. For a fairer scalability study, we instead initialize the 4,096 tokens by copying the 2,048 trained tokens and adding small noise, so that duplicated tokens do not decode identical Gaussians. The results are shown below:

Table R1: Ablation Studies of Number of Tokens

Number of Tokens (PSNR↑ / SSIM↑ / LPIPS↓)DyCheckADTTUM-DynamicsNVIDIA
102415.73 / 0.385 / 0.49327.60 / 0.805 / 0.25020.49 / 0.634 / 0.33920.40 / 0.486 / 0.454
204815.80 / 0.392 / 0.44328.36 / 0.832 / 0.18020.83 / 0.666 / 0.26820.53 / 0.498 / 0.394
409615.70 / 0.393 / 0.43828.65 / 0.837 / 0.17421.10 / 0.681 / 0.25720.64 / 0.499 / 0.392

With this initialization, PSNR consistently increases with the number of Gaussians. On DyCheck, the gain is less pronounced, since metrics are computed only over co-visible pixels; on the other datasets, rendering quality clearly improves as the number of Gaussians grows. This indicates that our framework does scale with the Gaussian budget. We revised this results on the original paper (§D.2, Tab. 15).

[W2] Although combining with a video model can help improve the results, the SOTA video models can already handle the NVS without the additional features proposed in this paper. As a result, this design just gives an additional solution instead of solving the unsolved problem in video model, which limits the novelty for the downstream application.

We thank the reviewer for this comment. We agree that recent video models can perform NVS without explicit 3D input so we have added the results of current SoTA NVS model and revised the paper (§4.3, Tab 5). However, as noted in NeoVerse, models conditioned only on camera parameters cannot guarantee geometric and temporal consistency, which motivates hybrid reconstruction-generation pipelines that use explicit geometry for spatiotemporal consistency and precise viewpoint control. For example, Lyra 2.0 [A] follows a similar philosophy for static scenes: it uses explicit per-frame geometry to maintain 3D consistency over long camera trajectories. T4GS extends this idea to dynamic content. Its Gaussians follow globally consistent motion, so the anchor constrains both static and dynamic regions, enabling more faithful NVS compared with current SoTA NVS model [B] as shown in below Table. In the longer term, we believe such a motion-consistent Gaussian anchor can also support long-horizon video generation. Moreover, our anchor is complementary to advances in video models. As stronger video models emerge, T4GS can serve as their conditioning signal to further improve quality while preserving consistent motion.

Table R2: Vbench evaluation for pose-conditioned NVS Diffusion model

MethodSub. Consist.Back. Consist.Temp. FlickMotion SmoothAeth. QualityImage Quality
NeoVerse0.87950.92300.95060.97980.51660.6392
Camera Anything0.87200.91870.96300.97790.49570.6128
Ours0.93170.95790.96870.98250.53810.6398

[A] Shen, Tianchang, et al. "Lyra 2.0: Explorable generative 3d worlds." arXiv preprint arXiv:2604.13036 (2026).

[B] Li, Yixuan, et al. "CameraAnything: Refilming videos with arbitrary camera control." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

AuthorsOpenReview ↗

Response to the Reviewer wqS7 (2/2)

[W3] From the demo, it seems a Gaussian may not keep the geometry consistency across the whole frames. To be specific, the dynamic Gaussian may represent different physical part at different frames instead of serving as a real physical part. Also, the static region still have obvious shifting across time frames. Could the author gives the Gaussian-wise tracking visualization for dynamic cases such as iphone-paper-windmill?

We thank the reviewer for this question. We have added Gaussian-wise tracking visualizations for dynamic scenes, including iPhone paper-windmill, to the revised paper (Fig. 7), where each Gaussian is colored by its token ID and traced across all frames. We also report 3D tracking metrics below:

Table R3: 3D & 2D Tracking Evaluation

Methods (EPE3DEPE3D↓ / δ0.053Dδ0.053D↑ / δ0.103Dδ0.103D↑ / AJ2DAJ2D↑ / δavg2Dδavg2D↑)ADTDrive TrackPanoptic Studio
Movies0.518 / 18.5 / 33.8 / 51.9 / 67.00.072 / 45.7 / 74.6 / 37.9 / 55.90.0902 / 47.4 / 74.4 / 41.7 / 54.7
TokenGS0.516 / 18.6 / 30.7 / 51.8 / 65.50.066 / 48.6 / 72.6 / 40.1 / 59.50.1164 / 36.3 / 61.7 / 42.0 / 56.6
Ours0.459 / 20.1 / 40.2 / 63.5 / 78.10.063 / 50.1 / 72.6 / 43.3 / 64.80.0741 / 74.9 / 86.2 / 53.9 / 67.9

These results show that our Gaussian queries faithfully follow the motion of dynamic parts over the full sequence, whereas many TokenGS Gaussians cover only part of each trajectory.

[Q1] The evaluation views seem to be close to the input view with a small shift. Do the model output also perform well when the rendering pose has a large shift from the input view? Since the model has already been trained on the multi-view dataset, do this capability emerge under the current design?

We thank the reviewer for this question. Our evaluation already includes renderings from viewpoints that differ substantially from the input. On DyCheck, the context comes from a handheld camera, while the targets are rendered from a separate static camera with an entirely different trajectory (about 30 degree between train and test camera). On NVIDIA, we evaluate on all camera–timestamp pairs not used as input (about 3 degree between train and test camera). T4GS outperforms all baselines on both benchmarks (Tab. 1), with the largest margin on DyCheck (15.64 vs. 12.45 PSNR), where the viewpoint gap is largest.

Regarding multi-view training, this capability is indeed supported by our multi-view training data. Removing multi-view data from training lowers DyCheck performance (Tab. 14, PSNR 15.89 → 15.22). This indicates that multi-view supervision helps the model render well from viewpoints far from the input.

[Q2] The inference input has different distribution with the training ones. In Eq3, the context (images) from other videos are inputted as context as well but in the inference, the model takes the monocular context. Do this difference cause any issues?

We thank the reviewer for this question. Monocular context is already the dominant training setting, where coverage sampling is applied only to Kubric (35% of samples), and only at a single context timestamp. Moreover, the decoder identifies each context frame by its relative timestamp embedding and aggregates features through full self-attention, so extra views at the same timestamp simply act as additional observations.

I thank the authors for detailed and helpful replies! Now, I have some follow-up questions.

[W1] Thanks for figuring out the initialization issue and achieving improvement. I have additional questions about training pipeline and cost. When training the model with different number of tokens, are they trained from scratch or some pretrained weight with common knowledge? When the model scales up, it is costy if the model need training from scratch and can not utilize the pretrained weight with other Ns. Also, could you discuss how the training time and GPU memory scales with the number N?

[Q1] If we imagine the rendering camera is on a sphere with the target object at its center, the DyCheck test cameras only cover a small portion on the sphere. For the larger shift such as 60 or 90 degree away from the train camera, does the result collapse heavily?

It would be very helpful if the authors can reply to these follow-up questions.

Reviewer_H3mWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes T4GS, a feed-forward approach to dynamic Gaussian reconstruction from monocular video without camera-pose inputs. It uses VGGT-initialized features and timestamp-conditioned learned queries to predict approximately 2K Gaussians at a target time. Training combines rendering losses with pseudo depth, normal, and motion supervision. The resulting representation is evaluated for novel-view synthesis, tracking, feature lifting, and diffusion-based video generation. The paper reports strong rendering results with substantially fewer Gaussians than the compared feed-forward methods.

Critical strengths
  1. This method use very few Gaussians (2K) to generate SoTA results.
  2. This method has a diffusion model pipeline to enhance the novel view rendering result.
Critical weaknesses
  1. Missing comparison with a dynamic TokenGS baseline. Both methods use learnable tokens to predict Gaussians, with different numbers of tokens and Gaussians per token. The paper should compare them under the same training and evaluation protocol. In particular, it should examine TokenGS trained with C4G’s approximately 2K Gaussian budget, and C4G trained to predict 64 Gaussians per token as in TokenGS.

  2. Table 3 reports only 2D tracking on ADT, which does not establish the accuracy of the predicted 3D trajectories. Following MoVieS Table 3, the authors should evaluate 3D point tracking on all three TAPVid-3D subsets (ADT, DriveTrack, and Panoptic Studio), and report 3D EPE and distance-threshold accuracy.

  3. The method uses a VGGT backbone, but the paper does not show whether the predicted Gaussians retain VGGT’s geometric accuracy. Following the depth evaluation in 4DGT Table 1, the authors should render depth from their Gaussians on held-out frames of ADT and TUM-Dynamics and report RMSE against ground-truth depth. This would test geometry at unseen timestamps and camera poses along the video trajectory.

  4. Limited DyCheck evaluation. The paper evaluates one 64-frame window in only five scenes. It should evaluate all seven scenes and all 13 available test-camera trajectories, with at least one window per trajectory. Testing additional available windows would give a more reliable picture of novel-view performance. DyCheck dataset description

Questions for authors

Given the VGGT backbone, why not use VGGT’s predicted point maps or point clouds to anchor Gaussian positions, then learn their attributes and motion? Could the authors compare this approach with direct prediction from learnable tokens under the same Gaussian budget?

6 comments
AuthorsOpenReview ↗

Response to the Reviewer H3mW (1/2)

[W1] Missing comparison with a dynamic TokenG. The paper should compare them under the same training and evaluation protocol with same Gaussian budget.

We thank the reviewer for pointing out TokenGS. We have cited TokenGS and included a comprehensive discussion and experimental comparison in our revised manuscript (§2, 4.2, Tab 1,2,3,4).

Differences in Motivation and Design

We would like to clarify the novelty that TokenGS and T4GS focus on different primary objectives, which leads to different architectural designs and training recipes. While both TokenGS and T4GS depart from pixel-aligned Gaussian prediction via learnable query tokens, they target fundamentally different objectives:

  • Primary Objective: TokenGS focuses on novel-view synthesis, showing that query-based coordinate regression can replace pixel-aligned prediction. In contrast, T4GS focuses on motion understanding, overcoming the ghosting and view-dependent opacity decay of feed-forward 4D methods (Fig. 1). This motion-faithful representation enables NVS, 4D feature lifting, and camera-controllable video generation.
  • Motion Modeling & Training Recipe: A key advantage of query-based decoding can model scene motion continuously by decoding the same query across timestamps. However, we observe that naively training a query-based network with photometric NVS loss alone fails to unlock this property: tokens still drift or alter opacity/color over time rather than tracking a physical surface (Fig. 7), also causing static regions to jitter. To resolve this, we introduce several recipes including relative time embedding, dataset curriculum, and explicit motion supervision (Ltrack,Lflow3D\mathcal{L}_{track}, \mathcal{L}_{flow}^{3D}). In contrast, TokenGS relies mainly on the NVS loss, and since it takes camera poses as input, it can readily separate camera from object motion and avoids jittering in static regions.
  • Application: Enforcing faithful motion modeling is what enables T4GS to support downstream applications such as 4D feature lifting, which no prior feed-forward method has demonstrated. Even TokenGS does not ensure consistent Gaussian trajectories (Table R2, Fig. 7), so features lifted onto its Gaussians would not stay consistent over time.

Quantitative Comparison with dynamic TokenGS

While TokenGS and T4GS focus on different primary objectives, we agree that further comparing with TokenGS can better verify our strengths. We first compare with the official dynamic TokenGS model against T4GS on NVS and point tracking. Notably, TokenGS requires camera poses as input and uses 81.9K Gaussians, whereas T4GS is pose-free and uses only 2K Gaussians (40×40\times fewer).

Table R1: NVS Comparison

Methods (PSNR↑/ SSIM↑/LPIPS↓)PoseDycheckADTTUM-DynamicsNVIDIA
TokenGSO12.00/0.297/0.60324.52/0.770/0.30819.69/0.628/0.36717.15/0.411/0.653
OursX15.64/0.397/0.44427.70/0.816/0.20220.84/0.668/0.27720.62/0.497/0.384

Table R2: 3D/2D Tracking Evaluation

Methods (EPE3D\text{EPE}^\text{3D}↓ / δ0.053D\delta_{0.05}^\text{3D}↑ / δ0.103D\delta_{0.10}^\text{3D}↑ / AJ2D\text{AJ}^\text{2D}↑ / δavg2D\delta^\text{2D}_\text{avg}↑)ADTDrive TrackPanoptic Studio
TokenGS0.516 / 18.6 / 30.7 / 51.8 / 65.50.066 / 48.6 / 72.6 / 40.1 / 59.50.1164 / 36.3 / 61.7 / 42.0 / 56.6
Ours0.459 / 20.1 / 40.2 / 63.5 / 78.10.063 / 50.1 / 72.6 / 43.3 / 64.80.0741 / 74.9 / 86.2 / 53.9 / 67.9

Despite using fewer Gaussians and operating without input camera poses, T4GS consistently outperforms TokenGS across all NVS and tracking benchmarks, validating the effectiveness of our motion-aware training recipe and compact bottleneck.

Controlled Capacity Analysis

Finally, following the reviewer's suggestion, we also examine both frameworks under a matched Gaussian budget. Due to the limited time, all variants below are initialized from their respective static backbones and trained for 10K steps on the same dynamic training mixture and evaluated under the same protocol with 2K Gaussian budget.

Table R3: NVS Comparison with same Gaussian budget at 10K Training Steps

MethodsPoseDycheckADTTUM-DynamicsNVIDIA
TokenGSO15.87 / 0.377 / 0.53626.02 / 0.774/ 0.41318.53 / 0.565 / 0.48020.58 / 0.525 / 0.566
OursX15.96 / 0.395 / 0.44926.55 / 0.787 / 0.21319.78 / 0.625 / 0.29620.50 / 0.494 / 0.397

While T4GS outperforms TokenGS on most metrics under the matched 2K budget, matching Gaussian capacity alone cannot fully compare the two methods, as they also differ in architecture and training. TokenGS uses a 6-layer ViT encoder with Plücker ray inputs and a 24-layer DETR-style decoder with separate static/dynamic tokens, whereas T4GS uses a VGGT-initialized encoder and a lightweight 2-layer self-attention decoder.

AuthorsOpenReview ↗

Response to the Reviewer H3mW (2/2)

[W2] Table 3 reports only 2D tracking on ADT, which does not establish the accuracy of the predicted 3D trajectories. Following MoVieS Table 3, the authors should evaluate 3D point tracking on all three TAPVid-3D subsets, and report 3D EPE and distance-threshold accuracy.

We thank the reviewer for this suggestion. We have evaluated 3D point tracking on all three TAPVid-3D subsets (ADT, DriveTrack, and Panoptic Studio), reporting 3D EPE and δ0.05/δ0.10 following MoVieS, along with 2D metrics (AJ, δavg), and revised the paper accordingly (§4.2, Tab 3).

Since T4GS is pose-free, its Gaussians are predicted in a normalized coordinate frame, unlike pose-required methods such as MoVieS, which operate in the metric frame given by ground-truth poses. Evaluating 3D trajectories therefore requires aligning both scale and coordinate frame to the ground truth. To avoid alignment choices that could favor any particular method, we initially reported only 2D tracking.

For this evaluation, we express GT tracks in the coordinate frame of the first context camera and align the scale, applying the same alignment to all methods and re-evaluating all baselines under this identical protocol. As shown below, T4GS achieves the best tracking results, further confirming that our Gaussian queries faithfully model the motion of dynamic points. The trajectories of Gaussian means visualized in Fig. 7 qualitatively support this.

Table R2: 3D/2D Tracking Evaluation

Methods (EPE3D\text{EPE}^\text{3D}↓ / δ0.053D\delta_{0.05}^\text{3D}↑ / δ0.103D\delta_{0.10}^\text{3D}↑ / AJ2D\text{AJ}^\text{2D}↑ / δavg2D\delta^\text{2D}_\text{avg}↑)ADTDrive TrackPanoptic Studio
Movies0.518 / 18.5 / 33.8 / 51.9 / 67.00.072 / 45.7 / 74.6 / 37.9 / 55.90.0902 / 47.4 / 74.4 / 41.7 / 54.7
TokenGS0.516 / 18.6 / 30.7 / 51.8 / 65.50.066 / 48.6 / 72.6 / 40.1 / 59.50.1164 / 36.3 / 61.7 / 42.0 / 56.6
Ours0.459 / 20.1 / 40.2 / 63.5 / 78.10.063 / 50.1 / 72.6 / 43.3 / 64.80.0741 / 74.9 / 86.2 / 53.9 / 67.9

[W3] The method uses a VGGT backbone, but the paper does not show whether the predicted Gaussians retain VGGT’s geometric accuracy. Following the depth evaluation in 4DGT Table 1, the authors should render depth from their Gaussians on held-out frames of ADT and TUM-Dynamics and report RMSE against ground-truth depth. This would test geometry at unseen timestamps and camera poses along the video trajectory.

We thank the reviewer for this suggestion. T4GS does not use VGGT’s depth or point-map heads; its Gaussians are decoded freely from learnable queries rather than as pixel-aligned depth, so preserving geometry is not automatic. Following 4DGT, we render depth at held-out frames of ADT and TUM-Dynamics and report RMSE against ground truth.

Table R4: Depth Evaluation

Methods (RMSE↓)ADTTUM-Dynamics
4DGT0.681.21
Movies0.891.40
Neoverse (World Mirror)0.340.67
Ours (VGGT)0.671.35
TokenGS0.781.70
VGGT (Pseudo GT)0.501.22

T4GS achieves depth RMSE competitive with per-pixel baselines while outperforming them on 3D tracking (Table R2). In dynamic regions, it achieves clearly lower depth error, as per-pixel methods render from ghosted or faded Gaussians while ours follow the scene motion.

[W4] Limited DyCheck evaluation. The paper evaluates one 64-frame window in only five scenes. It should evaluate all seven scenes and all 13 available test-camera trajectories, with at least one window per trajectory. Testing additional available windows would give a more reliable picture of novel-view performance.

Our DyCheck setting follows Shape of Motion and NeoVerse as noted in §B.1, which evaluate on the same five scenes (apple, block, paper-windmill, spin, and teddy) which is well preprocessed by Shape of Motion, We adopted this setting to ensure a fair comparison with these prior works.

[Q1] Given the VGGT backbone, why not use VGGT’s predicted point maps or point clouds to anchor Gaussian positions, then learn their attributes and motion? Could the authors compare this approach with direct prediction from learnable tokens under the same Gaussian budget?

We thank the reviewer for this suggestion. Our goal is a compact representation for understanding 4D scenes and their motion. VGGT’s dense point maps would require heuristic pruning to reach a ~2K budget, placing Gaussians independently of scene motion. They also exist only at observed timestamps and duplicate moving objects across views, reintroducing view-dependent Gaussians. Indeed, NeoVerse, built on VGGT, requires extra motion priors such as per-Gaussian velocity, yet its Gaussians still persist only around their source frames (Tab. 10, 11). In contrast, our queries learn where to place Gaussians and how they move without such priors, and the limited budget compels each to model its region’s motion over time.

Dear reviewer H3mW,

We sincerely appreciate your time and thoughtful efforts in reviewing our manuscript. As the discussion period will end in two days, we would like to gently remind you in case you have any remaining comments. We believe that we have sincerely and successfully addressed your concerns, supported by the corresponding additional experimental results. If you have any questions or concerns, please do not hesitate to let us know. We remain fully available for further discussion and are prepared to conduct additional experiments to address any remaining concerns during the discussion period.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Responce

We thank all reviewers for their careful and constructive feedback. The reviewers recognized several strengths of our work: the analysis of failure modes in current methods (wqS7, vUYm), strong performance with far fewer Gaussians (H3mW), applications to video diffusion and feature lifting (H3mW, wqS7, vUYm), the architecture design (wqS7), and the clarity of writing (vUYm). Based on the reviewers' feedback, we have revised the paper as follows, with all changes highlighted in red:

  • Related work and comparison with TokenGS (§2, §4.2, Tab 1,2,3,4)
  • Clarification of our method’s novelty (§3.4)
  • 3D tracking evaluation (§4.2, Tab 3)
  • Visualization of movement of Gaussian means (§C.4,Fig. 7)
  • Add pose-conditioned diffusion novel view synthesis model (§4.3, Tab 5)
  • Additional ablation studies of the number of tokens (§D.2, Tab. 15)
  • Novel view synthesis results on Additional datasets (§D.1, Tab. 14)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.