- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes QuaST for streaming 4D Gaussian reconstruction. The authors argue that residual-based updates change each Gaussian's attributes (center, rotation, scale, etc.) independently, so neighborhood motion and individual Gaussian shape can become inconsistent over time. QuaST instead uses a shared local Jacobian, estimated from relative center motion on a persistent motion-domain graph (built with SAM 2 and DINOv2), to both regularize neighborhood deformation (Graph-QC and Local-J losses) and guide covariance evolution (stop-gradient shape transport loss and per-Gaussian temporal shape control). Experiments on six DyNeRF/N3DV scenes and three SelfCap scenes compare against 3DGStream, HiCoM, QUEEN, and MoRGS. QuaST gives a small mean PSNR gain on N3DV over QUEEN and larger gains on SelfCap, along with fewer Gaussians than the baselines. Ablations on two SelfCap scenes support the contribution of each component.
Critical strengths
-
The main idea is clear and principled. Existing streaming 4D reconstruction methods update each Gaussian separately and lack regularization. This paper uses a local Jacobian estimated from neighboring motion as the regularization, with careful design details.
-
The paper tests on SelfCap, a long video benchmark with sequences of up to 1200 frames. The qualitative results show clear improvements over the baselines. The method is also efficient, using fewer Gaussians than the baselines.
Critical weaknesses
QuaST's core idea is to model local correlation, both among neighboring Gaussians' motion and between motion and shape, through a shared local deformation map. Two closely related lines are missing from the discussion. (a) Graph-structured deformation priors for Gaussians (e.g., MoSca [1], OriGS [2]) also use graph structure to couple neighboring motion, but drive deformation through anchors. QuaST claims to avoid motion-driving anchors, yet the reason why this design is better is not clearly shown. (b) High-dimensional Gaussians (4DGS [3], 7DGS [4]) encode space-time correlation in a joint covariance, which ties motion and shape within a primitive. This is a direct alternative to the paper's premise that independent attribute residuals are the problem. Although these methods are offline (not streaming), the paper should discuss the differences and, where feasible, add reference comparisons or a controlled alternative. Moreover, the tables compare only against 3DGStream, HiCoM, QUEEN, and MoRGS (the last only on SelfCap), while recent streaming methods the paper itself cites, such as Instant Gaussian Stream and Recon-GS, are absent.
[1] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds
[2] Orientation-anchored Hyper-Gaussian for 4D Reconstruction from Casual Videos
[3] Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting
[4] 7DGS: Unified Spatial-Temporal-Angular Gaussian Splatting
Questions for authors
Does the core method design also apply to general (offline) 4D reconstruction (e.g., on top of a deformation-field method), or does it depend on streaming-specific parts? Any insight on this from the authors would be appreciated.
Response to Reviewer NxJQ: Deformation Priors, Spacetime Baselines, and Offline Applicability
We thank Reviewer
NxJQfor recognizing QuaST's principled formulation, careful design, and strong long-horizon results. We expanded the related work (Sec. 2) and added quantitative baselines (Tables 1–4) in the revised PDF (changes highlighted in blue). We address offline applicability below.1. Graph/Anchor-Driven Deformation Priors (Weakness (a))
SelfCap, 1,200 frames (Table 4); entries are PSNR↑ / SSIM↑ / LPIPS↓:
Methodological distinction. MoSca uses a sparse motion scaffold; Mango-GS learns multi-frame motion on sparse control nodes and transfers it through learned affinities; ReCon-GS uses hierarchical Anchor Gaussians. OriGS instead uses an orientation field and orientation-aware Hyper-Gaussians (Sec. 2). QuaST introduces no separate motion-driving anchors: its graph vertices are reconstructed Gaussians. Persistent edges provide neighborhoods and confidence weights for Graph-QC and Local-J (Eqs. (3)–(10)), rather than interpolating anchor-predicted motion. The local Jacobian is estimated from relative center displacements (Eq. (6)) and, with stop-gradient, supplies the covariance-transport target (Eq. (11)).
Quantitative comparison. MoSca and OriGS primarily target casual monocular video, so we added multiview baselines Mango-GS (non-streaming) and ReCon-GS (streaming), reproduced under the same SelfCap camera/resolution/evaluation protocol. At 1,200 frames, QuaST exceeds ReCon-GS by 4.73 / 6.55 / 7.31 dB and Mango-GS by 10.32 / 17.17 / 8.77 dB on Bike/Corgi/Yoga. Its DyNeRF mean exceeds them by 0.64 / 1.91 dB, respectively (Table 1).
2. High-Dimensional Spacetime Gaussians and Recent Streaming Methods (Weaknesses (b), (c))
DyNeRF, 300 frames, PSNR (Table 1):
Representation and objective. 4DGS models spacetime correlation with 4D Gaussian primitives; 4D Scaffold-GS uses 4D anchor features. QuaST retains standard 3D Gaussians and instead enforces motion–shape compatibility during causal optimization. On DyNeRF, QuaST reaches 32.33 dB, above 4DGS (31.59; +0.74 dB) and 4D Scaffold-GS (31.42; +0.91 dB). Regarding 7DGS, we contacted its authors to request access to their implementation but have not received a response. Therefore, we could not conduct a reproducible comparison.
Streaming coverage. ReCon-GS is now evaluated at all three SelfCap horizons (Tables 2–4) and on all six DyNeRF scenes (Table 1). IGS follows a different protocol: AGM-Net is pretrained across four N3DV sequences and tested on only two held-out sequences (Cut Roasted Beef and Sear Steak), whereas QuaST optimizes each of the six scenes causally without cross-scene pretraining. We clarify this protocol difference here rather than comparing incompatible six-scene averages.
3. Applicability to General Offline 4D Reconstruction (Question 1)
The underlying principle may also extend to offline reconstruction. While QuaST's persistent motion-domain graph, parent–child Gaussian correspondences, and local Jacobian estimation are designed for causal frame-by-frame optimization, the principle of motion–shape compatibility is not inherently limited to streaming settings. For example, offline deformation-field methods could potentially incorporate local deformation gradients to guide covariance transport during joint multi-frame optimization.
Future extension. In future work, we plan to investigate how QuaST's local Jacobian regularization and covariance transport can be integrated into existing offline 4D reconstruction frameworks, and whether this integration can further improve reconstruction quality and geometric consistency.