acceptodds
Under review as a conference paper at ICLR 2027

PointSplatV2 : Query-Based Feed-Forward 4D Gaussian Reconstruction

Abstract

On-the-fly dynamic reconstruction is crucial for real-time spatial computing and immersive applications, requiring 4D representations that render stably from free viewpoints. Existing feed-forward methods rapidly reconstruct scenes by predicting Gaussian representations aligned with individual input observations, such as views or timesteps. This observation-wise discrete prediction prevents them from directly consolidating multi-view and temporal observations into a shared continuous, compact 4D representation, resulting in temporal flickering and redundant Gaussian primitives. Conversely, optimization-based methods achieve spatiotemporal consistency and compactness, but their costly per-scene optimization limits low-latency applications. To bridge this gap, we introduce PointSplatV2, a query-based feed-forward framework for reconstructing 4D Gaussians. Given multi-view temporal observations, PointSplatV2 first uses a multi-view image encoder to construct a latent field from the input frames. It then builds key-spatiotemporal queries that sparsely yet completely cover the spatial projection of the 4D scene at a key timestep. Each query then aggregates features from this latent field via cross-attention and directly decodes a 4D Gaussian representation. This simple yet effective formulation gathers temporal observations onto shared queries to avoid dense per-frame discrete prediction, and lets the continuous representation render stably across time. To extend this formulation to longer sequences, we introduce the Boundary-Aware Sequential Training (BAST) strategy, which enforces temporal coherence at sequence interfaces. Experiments show that PointSplatV2 outperforms the evaluated feed-forward 4D baseline in quality and compactness at low latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Feed-forward reconstruction, 4DGS

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7J2kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

PointSplatV2 is a feed-forward framework for reconstructing dynamic scenes from sparse multiview videos. It uses geometry-guided spatiotemporal queries to aggregate observations into a compact, continuous 4D Gaussian representation. The method also introduces boundary-aware sequential training to encourage consistency between neighboring temporal chunks. Experiments demonstrate a promising balance between rendering quality, representation compactness, and reconstruction efficiency.

Critical strengths
  1. PointSplatV2 produces detailed reconstructions while substantially reducing redundant Gaussian primitives. The reported results demonstrate a useful trade-off between visual quality, storage requirements, and reconstruction efficiency, particularly for dynamic foreground reconstruction.

  2. Geometrically anchored queries provide a sensible way to aggregate observations across views and time into a shared dynamic representation. The supplementary boundary analysis and robustness experiments help characterize the method’s behavior, while explicit accounting of preprocessing and prediction costs makes the efficiency evaluation more informative.

Critical weaknesses
  1. The methodological novelty is insufficiently differentiated and experimentally isolated. TokenGS [1] is particularly relevant because it already combines Gaussian queries with dynamic reconstruction, yet it is neither discussed nor evaluated. AnchorSplat [2] and C3G [3] also establish closely related approaches to geometry-supported and compact token-based Gaussian prediction, respectively. Furthermore, the motivation should distinguish pixel-aligned prediction from independent per-frame outputs: 4DGT [4] demonstrates that pixel-aligned primitives can still define continuous temporal representations. Although these methods do not subsume PointSplatV2’s complete formulation, the current comparisons do not establish how much improvement comes specifically from the proposed spatiotemporal queries, rather than the pretrained backbone, geometric support, or output parameterization. Comparisons using matched inputs, backbones, and Gaussian budgets would clarify the significance of the remaining contribution.

  2. The evidence for the central temporal-consistency contribution is not sufficiently controlled. In Table 3, “No BAST fine-tuning” is the Stage-1 checkpoint, whereas the full model receives another 100k training steps with multichunk batches. Consequently, the 2.24 dB improvement cannot be attributed specifically to the overlap-consistency loss. Appendix A.4 varies inference overlap using the same trained model and does not resolve this confound. Temporal evaluation also needs stronger validation. The standard VBench Temporal Flickering implementation measures adjacent-frame pixel differences and explicitly assumes static videos; it does not compensate for genuine motion. Higher scores can therefore reflect reduced motion or detail rather than more accurate dynamics. The paper acknowledges smoothing bias, and its overlap MSE is useful, but agreement between two predictions does not establish motion accuracy. Ground-truth-relative or motion-compensated temporal errors are needed.

[1] Jiawei Ren et al. “TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens.” CVPR, 2026.

[2] Xiaoxue Zhang et al. “AnchorSplat: Feed-Forward 3D Gaussian Splatting with 3D Geometric Priors.” CVPR, 2026.

[3] Honggyu An et al. “C3G: Learning Compact 3D Representations with 2K Gaussians.” CVPR, 2026.

[4] Zhen Xu et al. “4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos.” NeurIPS, 2025.

Questions for authors
  1. How does PointSplatV2 compare with TokenGS under matched multiview inputs and training conditions? What benefit comes specifically from predicting explicit 4D parameters once per chunk instead of decoding time-conditioned 3D Gaussians?

  2. Can you report ground-truth-relative temporal errors or motion-compensated consistency, including actual chunk transitions and held-out intermediate timestamps? How exactly was VBench Temporal Flickering applied to moving scenes?

  3. Please specify the Neural3DV training/test split. Are evaluation scenes unseen during model training, or are only views or frames held out?

Reviewer_boVDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PointSplatV2, a feed-forward method for reconstructing compact and temporally consistent 4D Gaussian Splatting (4DGS) representations from sparse multi-view videos. The method aims to address the redundancy and temporal inconsistency caused by existing approaches that predict Gaussians independently for each frame. First, the method leverages DA3 to construct a latent field that integrates multi-view image features, camera information, and temporal information. It then generates voxel-based sparse queries using the visual hull at the middle frame for the foreground and the depth map of the first frame for the background. Each query aggregates information across multiple frames through cross-attention, and a 4DGS decoder directly predicts Gaussian attributes, including position, color, and velocity. Finally, the paper introduces BAST to reduce discontinuities between chunks in long sequences by enforcing consistency between rendered results from overlapping frames. Experimental results show that the proposed method achieves higher reconstruction quality than existing feed-forward 4D reconstruction methods while reducing the number of Gaussians and processing time.

Critical strengths
  1. The paper clearly identifies the problems of redundant Gaussians and temporal inconsistency in existing feed-forward 4D reconstruction methods, and the proposed methodology is well designed to address these issues. Instead of independently predicting Gaussians for multiple frames, integrating them into a single continuous 4D representation through sparse queries is a reasonable approach. Overall, the problem formulation and the proposed methodology are well aligned.

  2. The approach of separating foreground and background regions and generating queries using the visual hull and depth, respectively, is also reasonable. Furthermore, introducing BAST to address the challenges of extending short-chunk reconstruction to longer sequences is a natural design choice. The individual modules are well connected, and the overall pipeline appears to be well designed and coherent.

  3. The experimental results demonstrate substantially higher reconstruction quality than L4GM while significantly reducing the number of Gaussians. The paper also provides ablation studies and analyses of geometry perturbation and boundary consistency, which support the effectiveness of the proposed methodology to some extent.

Critical weaknesses
  1. Although the paper provides an ablation study demonstrating the effectiveness of query construction, the number of Gaussians differs significantly across the evaluated settings. Therefore, it is difficult to determine whether the performance improvement comes from the query construction strategy itself or simply from differences in the number or distribution of Gaussians. A comparison under the same Gaussian budget would help clarify this issue.

  2. Although BAST is a reasonable approach, it remains somewhat unclear how effectively it maintains temporal consistency in long sequences. The current ablation without BAST also removes Stage-2 fine-tuning entirely, and the analysis of long sequences is limited to a maximum of 25 frames. Furthermore, since the method relies on the foreground visual hull and background depth, its robustness under severe occlusions or fast motion, where the estimated geometry may be incomplete or inaccurate, remains unclear.

Questions for authors
  1. Query Construction: In the current ablation study, the number of Gaussians varies significantly depending on the query construction strategy. Could the authors compare the proposed approach with random sampling, depth-based sampling, or learnable queries under the same Gaussian budget? Additionally, how does the reconstruction performance change with different voxel sizes and numbers of queries?

  2. Temporal Anchors and Occlusion: The method generates foreground queries based on the middle frame and background queries based on the first frame. However, occluded regions or fast-moving objects may not receive sufficient queries if they are not properly observed in these reference frames. How would the results change if different timesteps were used or geometry from multiple frames were combined? In particular, could the authors analyze the reconstruction performance in regions undergoing disocclusion?

  3. Effectiveness of BAST: The ablation study without BAST also removes Stage-2 training entirely. Could the authors provide a comparison that removes only the BAST loss while keeping the same training setting? Additionally, could the authors further analyze whether BAST merely improves image quality at chunk boundaries or also improves the actual Gaussian motion and temporal consistency?

  4. Long Sequences: Since the proposed method generates independent 4DGS representations for each chunk, Gaussian identities and motions are not explicitly connected across chunks. Does this approach remain stable for substantially longer sequences? Could the authors provide an analysis of how boundary discontinuity, reconstruction quality, the number of Gaussians, and processing time change as the sequence length increases?

  5. Fast Motion and Robustness: The proposed method relies on geometric priors such as the visual hull and DA3 depth estimates. How much is the reconstruction quality affected when these geometric estimates become inaccurate due to fast motion or severe occlusions? An analysis of reconstruction performance under different motion magnitudes and occlusion levels would help demonstrate the robustness of the proposed methodology.

Reviewer_uT12OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PointSplatV2, a feed-forward method for reconstructing 4D Gaussian representations from sparse-view videos. Instead of predicting Gaussians for each frame separately, the method uses spatial queries to aggregate information from multiple views and timesteps within a 4-frame chunk. It also introduces Boundary-Aware Sequential Training to improve consistency between neighboring chunks. Experiments on DNA-Rendering and Neural3DV show better results than L4GM, while using fewer Gaussians than existing feed-forward 3D methods.

Critical strengths
  1. Using shared spatial queries across multiple frames is a reasonable design and helps reduce redundant Gaussians.

  2. The method achieves good reconstruction quality with much fewer Gaussians, and has relatively fast inference in the foreground setting.

  3. The paper includes useful ablations on query construction, chunk overlap, and robustness to input noise.

Critical weaknesses
  1. The temporal consistency improvement is not well demonstrated. The VBench flickering scores are very close across different methods, and PointSplatV2 actually has the lowest score on Neural3DV. Also, PointSplat still performs better in PSNR, SSIM, and LPIPS, so it is not clear whether the proposed 4D representation really improves reconstruction quality beyond reducing the number of Gaussians.

  2. The baseline comparisons are somewhat limited. L4GM is the only feed-forward 4D baseline, but it was originally designed for a different setting (object-centric) and may not be well optimized for this task. PointSplat and GPS-Gaussian are also missing from the full-image comparison.

  3. The BAST ablation is not fully convincing, since the model without BAST also has 100k fewer training steps. It is difficult to tell how much of the improvement actually comes from the boundary loss. The method also relies on accurate camera poses, masks and depth estimation, and the robustness experiments show a considerable performance drop with small pose errors. Some important design choices, such as the number of Gaussians per query and chunk length, are not there.

Questions for authors
  1. Are the training and testing scenes on Neural3DV completely disjoint? Could the authors clarify the dataset splits and the number of training sequences for both datasets?

  2. Could the authors report more meaningful temporal consistency metrics, such as flow-warped error? The current flickering metric shows very small differences, and PointSplatV2 is even worse than the baselines on Neural3DV.

  3. Could the authors compare BAST against a model trained for the same number of steps but without the boundary loss? Also, how does the method perform with longer chunks, such as 8 or 16 frames, especially for fast or nonlinear motion?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We deeply appreciate the time and effort you have dedicated to providing detailed and insightful feedback on our work. We have carefully addressed each comment and revised the manuscript accordingly. The main revisions are summarized below:

  • Stronger baseline comparisons: We added dynamic TokenGS to the quantitative comparisons on DNA-Rendering and Neural3DV and updated the qualitative comparisons (Tables 1–2 and Figures 3–6).
  • Expanded ablations: We evaluated fine-tuning without the BAST loss, compared query constructions under a matched Gaussian budget, and examined explicit 4D parameterization against per-frame 3DGS outputs (Table 3).
  • Additional temporal evaluation: We introduced a ground-truth-relative, motion-compensated error metric in the main comparisons and added full-sequence, chunk-transition, and intermediate-timestamp evaluations in the supplementary material (Tables 1–2 and 6–8).
  • Cross-dataset generalization: We added quantitative and qualitative evaluations on SelfCap, Google Immersive, and ENeRF-Outdoor (Table 5 and Figure 9).
  • Long-sequence evaluation: We evaluated reconstruction quality, temporal error, runtime, memory consumption, and representation size on sequences of up to 150 frames (Tables 9–10).
  • Expanded related work: We extended the discussion of related Gaussian prediction methods, including TokenGS, AnchorSplat, and C3G.
  • Backbone-matched query ablation: We compared geometry-anchored queries with learnable tokens while keeping the DA3 backbone, inputs, training data, training schedule, and Gaussian budget per chunk fixed (Table 3).
  • No-motion baseline: We introduced a no-motion baseline for intermediate-timestamp evaluation, comparing two renderings from the same 4D prediction against the same held-out frame and target view (Table 8).

These revisions were made in response to the reviewers' valuable suggestions, including broader baseline comparisons and discussion of related methods (Reviewers uT12 and 7J2k), more controlled ablations (Reviewers uT12, boVD, and 7J2k), additional evaluations of temporal accuracy and intermediate-timestamp reconstruction (Reviewers uT12, boVD, and 7J2k), long-sequence performance (Reviewer boVD), and cross-dataset generalization (Reviewers uT12 and 7J2k). We sincerely thank all three reviewers again for their constructive feedback, which has helped us improve the clarity and rigor of our evaluation and better demonstrate the strengths and limitations of our method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.