acceptodds
Under review as a conference paper at ICLR 2027

Trajectory-Anchored Compact Gaussians for Feed-Forward 4D Reconstruction

Abstract

Dynamic scene reconstruction from monocular video requires a representation that supports both novel-view rendering and correspondence through time. Feed-forward methods learn reconstruction priors across scenes, enabling inference on unseen videos without costly per-scene optimization. However, current feed-forward methods rely heavily on dense pixel-aligned architecture which produces redundant primitives. On the other hand, recent compact query-based methods can struggle to maintain accurate temporal correspondence. We present a feed-forward framework for compact, structured 4D Gaussian reconstruction without input camera poses. Our central idea is to use dense correspondence to organize Gaussians around a sparse set of explicit motion trajectories. We first predict dense B-spline trajectories with a geometry-anchored parameterization that preserves each trajectory's source point position. Grouping trajectories by spatial proximity, motion, and appearance yields representative anchors. Given learnable tokens paired with the anchor trajectories, a query-based decoder aggregates image evidence to generate child Gaussians, whose local offsets capture geometry and motion beyond each anchor. This hierarchy supports rendering and dense 3D correspondence with a compact Gaussian representation whose size is independent of the number of input pixels. Across the evaluated benchmarks, our full model outperforms the evaluated pixel-aligned baselines in most rendering metrics while using significantly fewer Gaussians. The same compact representation outperforms the evaluated query-based baselines across all reported rendering and Gaussian-tracking metrics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Reconstruction, Feedforward, Gaussian Splatting, 4D Gaussians, Monocular videos

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mLrLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a feed-forward method for reconstructing compact 4D Gaussians from unposed monocular video. The authors point out that existing pixel-aligned methods create a Gaussian for every pixel, leading to redundancy and ghosting, while query-based methods are compact but do not explicitly track motion, resulting in weak correspondence. To address this, the method uses a three-stage design. First, a DA3-based backbone predicts per-pixel 3D points and cameras, and a trajectory is predicted for every pixel as a B-spline displacement that, through a null-space constraint, always passes through the source point at its source time. Second, these dense trajectories are clustered based on 3D proximity, relative-distance variation, and color to extract dynamic anchors, which are combined with static anchors to form a small set of anchor trajectories. Third, a query-based decoder attaches a learnable token to each anchor, reads image features, and generates child Gaussians that move along the anchor trajectory with additional time-varying offsets. Training is done in two stages, where the foundation model is trained first and then frozen while the decoder is trained. As a result, the method performs rendering and tracking with far fewer Gaussians than dense pixel-aligned methods.

Critical strengths
  1. Hierarchical design that separates motion tracking from rendering.

Existing feed-forward 4D methods either create a Gaussian per pixel, which tracks motion well but duplicates surfaces, or decode Gaussians from a few queries, which is compact but weak at tracking. This paper combines both by predicting per-pixel trajectories and using a selected subset as a scaffold for the Gaussians. With far fewer Gaussians, it achieves the highest PSNR and SSIM among external baselines and outperforms other query-based methods on all tracking metrics. Showing that a representation can be compact without losing motion tracking is the main reason I view this paper positively.

  1. Simple yet effective null-space constraint (NSC).

Trace Anything’s absolute control points let the learned motion shift the trajectory’s starting position. This paper instead splits each trajectory into a source point and a displacement, using a null-space basis so that the displacement is exactly zero at the observation time by construction, at negligible extra cost. The ablation shows that removing it degrades both rendering and tracking, and the idea can be readily adopted by other trajectory-based methods.

Critical weaknesses
  1. Heavy reliance on external papers for describing key components.

The paper combines several existing methods, including TraceAnything, SuperTrajectory, TokenGS, and LocusGS, and many of its components are described only by reference. For example, the DPC procedure and static anchor selection in anchor extraction, the geometric bias and the separate image encoder in the decoder, and the geometry, reprojection, static, and rendering losses in Stage I are stated to follow prior work without concrete definitions. Since the contribution relies heavily on combining existing components, this lack of detail not only hurts reproducibility but also makes it difficult to assess which parts are new contributions of this paper.

  1. No efficiency evaluation despite the emphasis on compactness.

Although the final number of Gaussians is greatly reduced, the compact variant first predicts trajectories for every pixel and then additionally runs clustering, a separate image encoder, and a query-based decoder, so its computational cost may actually exceed that of the dense variant. Moreover, since the positions and rotations of the child Gaussians are decoded again for each target time, it is unclear whether the Gaussian count fully reflects the actual storage size of the representation. Nevertheless, no measurements of inference time, memory, or storage are provided. Since the Gaussian count is used as a key point of comparison, the paper should show that compactness translates into actual efficiency for its central claim to be convincing.

Questions for authors
  1. Efficiency

Could the authors compare the inference time, memory, and rendering speed of the full pipeline, including dense trajectory prediction, clustering, the image encoder, and the decoder, against the baselines? In addition, since the positions and rotations of the child Gaussians are decoded again for each target time, how does the actual storage size scale with the number of frames? I would like to understand whether “compact” brings practical efficiency benefits beyond the number of Gaussians.

  1. Scope of contribution in the decoder and anchor extraction

The query-based decoder builds on TokenGS and LocusGS, and the anchor extraction builds on SuperTrajectory and GroupFlow. The paper presents several changes, such as replacing LocusGS’s static anchors with time-dependent anchors kept fixed during decoding, adding a color term to the grouping criterion, and separating the static and dynamic budgets. Beyond these, could the authors clearly identify which elements are newly designed compared to existing methods?

Reviewer_R1eNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies feed-forward 4D reconstruction from monocular videos, aiming to achieve compact scene representations while preserving rendering quality and temporal correspondence. The authors propose a hierarchical framework that predicts dense B-spline trajectories, extracts representative key trajectories based on spatial proximity, motion, and appearance, and generates child Gaussians through a query-based decoder. A geometry-anchored trajectory parameterization preserves source point positions while enabling smooth motion modeling.

Critical strengths
  1. The paper clearly identifies the trade-off between compact scene representations and accurate temporal correspondence in feed-forward 4D reconstruction.

  2. The hierarchical design effectively combines trajectory prediction, key trajectory extraction, and query-based Gaussian decoding to balance compactness and reconstruction quality.

  3. The paper is well-written and presents the proposed framework and technical details in a coherent and understandable manner.

Critical weaknesses
  1. The method requires two-stage training, including foundation model training on eight datasets and an additional 5K-step warm-up followed by 30K-step decoder training. Reporting the total training time and computational cost would help clarify the practical overhead of the proposed framework.

  2. The current ablations do not separately isolate the effects of spatial proximity and temporal motion consistency. Evaluating these criteria independently would provide a clearer understanding of their respective contributions.

  3. Although the proposed method substantially reduces the number of Gaussians, the computational cost of dense trajectory prediction and key trajectory extraction remains unclear. Reporting inference latency, peak memory usage, and scalability with video resolution and length would provide a more complete assessment of practical efficiency.

Questions for authors
  1. How much time and GPU resources are required for Stage I and Stage II, respectively? Reporting these costs would help clarify the training overhead of the proposed framework.

  2. The key trajectory extraction module combines spatial proximity and temporal motion consistency. An ablation that removes each criterion individually, with other settings fixed, would help establish their respective contributions to tracking and rendering performance.

  3. The compact representation substantially reduces the number of Gaussians, but its overall inference efficiency remains unclear. Please provide inference latency and peak GPU memory usage, together with an analysis of how these costs scale with video resolution and sequence length.

Reviewer_zxoYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a trajectory-anchored framework for compact feed-forward 4D Gaussian reconstruction from monocular videos without input camera poses. The method first predicts dense geometry-anchored B-spline trajectories, extracts representative trajectory anchors through clustering, and generates child Gaussians using a query-based decoder. Experiments demonstrate competitive novel-view synthesis and tracking performance with significantly fewer Gaussians.

Critical strengths
  1. The paper addresses an important problem in feed-forward 4D reconstruction: achieving compact representations while preserving rendering quality and temporal correspondence.

  2. The trajectory-anchored Gaussian representation is intuitive and well-motivated. The combination of explicit motion trajectories and local Gaussian decoding provides a reasonable way to separate global motion from local geometry and appearance.

  3. The experimental results are promising. The compact model achieves competitive rendering quality with approximately 74× fewer Gaussians than dense baselines, while outperforming existing query-based methods on Gaussian tracking.

Critical weaknesses
  1. The paper demonstrates a reduction in the number of Gaussians but does not report end-to-end inference time, GPU memory consumption, or rendering speed. Since the method still requires complex modules, it is unclear whether the compact representation leads to real computational benefits.

  2. Table 2 evaluates Gaussian tracking for the compact model, but Table 5 only reports standard 3D point tracking results for Ours (dense). Since maintaining accurate correspondence is a central motivation, the authors should also report APD and AJ for Ours (compact) to demonstrate how much tracking accuracy is preserved after compression.

  3. A key motivation is to eliminate redundant Gaussians corresponding to the same physical surfaces observed across different frames. However, Section 3.3 adopts image-local trajectory clustering, and it is unclear how trajectories corresponding to the same surface across different input frames are merged into shared anchors. Could the authors clarify this mechanism and provide visualizations or analysis demonstrating that the proposed grouping effectively reduces cross-frame redundancy?

  4. In Eq. 4, child Gaussian positions depend on time-varying offsets that are not spatially bounded. Although the anchor trajectories are continuous and motion supervision is applied, it remains unclear how consistently the child Gaussians follow their corresponding anchors over time. Could the authors provide an analysis or visualizations of the temporal consistency of child offsets and their deviations from the anchor trajectories?

Questions for authors

Please refer to the weaknesses. I would be willing to reconsider my rating if the authors can satisfactorily address these concerns with clear explanations and supporting evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.