acceptodds
Under review as a conference paper at ICLR 2027

QuaST: Quasiconformal Jacobian-Guided Shape Transport for Stable Streaming 4D Gaussian Reconstruction

Abstract

Streaming 4D Gaussian reconstruction calls for local motion and Gaussian shape to evolve coherently across frames, yet existing residual-based update methods do not explicitly guarantee this. We present QuaST, a quasiconformal Jacobian-guided shape transport framework that models relative center motion and covariance evolution through a shared local deformation map. QuaST constructs a persistent motion-domain graph and estimates rank-aware local Jacobians from relative center motion. A quasiconformal-inspired prior constrains anisotropic distortion within local neighborhoods. A stop-gradient interface lets neighborhood motion guide covariance transport while routing the transport loss only to current Gaussian scale and rotation. Furthermore, temporal shape control limits excessive anisotropic changes of individual Gaussians across consecutive frames. QuaST integrates these constraints into optimization while retaining standard Gaussian rendering, without a learned deformation field or motion-driving anchors. QuaST is strictly causal: the update at frame uses only the current training images and states stored from frames up to , with no future-frame access. Experiments on DyNeRF and SelfCap show that QuaST improves novel-view reconstruction and maintains stable performance over extended captures of up to 1,200 frames.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Gaussian Splatting, Dynamic Scene Reconstruction, Streaming Reconstruction, Quasiconformal Regularization, Jacobian-Guided Shape Transport, Temporal Consistency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NxJQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QuaST for streaming 4D Gaussian reconstruction. The authors argue that residual-based updates change each Gaussian's attributes (center, rotation, scale, etc.) independently, so neighborhood motion and individual Gaussian shape can become inconsistent over time. QuaST instead uses a shared local Jacobian, estimated from relative center motion on a persistent motion-domain graph (built with SAM 2 and DINOv2), to both regularize neighborhood deformation (Graph-QC and Local-J losses) and guide covariance evolution (stop-gradient shape transport loss and per-Gaussian temporal shape control). Experiments on six DyNeRF/N3DV scenes and three SelfCap scenes compare against 3DGStream, HiCoM, QUEEN, and MoRGS. QuaST gives a small mean PSNR gain on N3DV over QUEEN and larger gains on SelfCap, along with fewer Gaussians than the baselines. Ablations on two SelfCap scenes support the contribution of each component.

Critical strengths
  1. The main idea is clear and principled. Existing streaming 4D reconstruction methods update each Gaussian separately and lack regularization. This paper uses a local Jacobian estimated from neighboring motion as the regularization, with careful design details.

  2. The paper tests on SelfCap, a long video benchmark with sequences of up to 1200 frames. The qualitative results show clear improvements over the baselines. The method is also efficient, using fewer Gaussians than the baselines.

Critical weaknesses

QuaST's core idea is to model local correlation, both among neighboring Gaussians' motion and between motion and shape, through a shared local deformation map. Two closely related lines are missing from the discussion. (a) Graph-structured deformation priors for Gaussians (e.g., MoSca [1], OriGS [2]) also use graph structure to couple neighboring motion, but drive deformation through anchors. QuaST claims to avoid motion-driving anchors, yet the reason why this design is better is not clearly shown. (b) High-dimensional Gaussians (4DGS [3], 7DGS [4]) encode space-time correlation in a joint covariance, which ties motion and shape within a primitive. This is a direct alternative to the paper's premise that independent attribute residuals are the problem. Although these methods are offline (not streaming), the paper should discuss the differences and, where feasible, add reference comparisons or a controlled alternative. Moreover, the tables compare only against 3DGStream, HiCoM, QUEEN, and MoRGS (the last only on SelfCap), while recent streaming methods the paper itself cites, such as Instant Gaussian Stream and Recon-GS, are absent.

[1] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds

[2] Orientation-anchored Hyper-Gaussian for 4D Reconstruction from Casual Videos

[3] Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

[4] 7DGS: Unified Spatial-Temporal-Angular Gaussian Splatting

Questions for authors

Does the core method design also apply to general (offline) 4D reconstruction (e.g., on top of a deformation-field method), or does it depend on streaming-specific parts? Any insight on this from the authors would be appreciated.

1 comment
AuthorsOpenReview ↗

Response to Reviewer NxJQ: Deformation Priors, Spacetime Baselines, and Offline Applicability

We thank Reviewer NxJQ for recognizing QuaST's principled formulation, careful design, and strong long-horizon results. We expanded the related work (Sec. 2) and added quantitative baselines (Tables 1–4) in the revised PDF (changes highlighted in blue). We address offline applicability below.

1. Graph/Anchor-Driven Deformation Priors (Weakness (a))

SelfCap, 1,200 frames (Table 4); entries are PSNR↑ / SSIM↑ / LPIPS↓:

SceneMango-GS (Offline Nodes)ReCon-GS (Streaming Anchors)QuaST (Streaming, Ours)
Bike15.56 / 0.695 / 0.44221.15 / 0.772 / 0.43725.88 / 0.893 / 0.233
Corgi10.10 / 0.579 / 0.56220.72 / 0.705 / 0.52527.27 / 0.828 / 0.331
Yoga10.86 / 0.659 / 0.55112.32 / 0.702 / 0.59419.63 / 0.855 / 0.361

Methodological distinction. MoSca uses a sparse motion scaffold; Mango-GS learns multi-frame motion on sparse control nodes and transfers it through learned affinities; ReCon-GS uses hierarchical Anchor Gaussians. OriGS instead uses an orientation field and orientation-aware Hyper-Gaussians (Sec. 2). QuaST introduces no separate motion-driving anchors: its graph vertices are reconstructed Gaussians. Persistent edges provide neighborhoods and confidence weights for Graph-QC and Local-J (Eqs. (3)–(10)), rather than interpolating anchor-predicted motion. The local Jacobian is estimated from relative center displacements (Eq. (6)) and, with stop-gradient, supplies the covariance-transport target (Eq. (11)).

Quantitative comparison. MoSca and OriGS primarily target casual monocular video, so we added multiview baselines Mango-GS (non-streaming) and ReCon-GS (streaming), reproduced under the same SelfCap camera/resolution/evaluation protocol. At 1,200 frames, QuaST exceeds ReCon-GS by 4.73 / 6.55 / 7.31 dB and Mango-GS by 10.32 / 17.17 / 8.77 dB on Bike/Corgi/Yoga. Its DyNeRF mean exceeds them by 0.64 / 1.91 dB, respectively (Table 1).

2. High-Dimensional Spacetime Gaussians and Recent Streaming Methods (Weaknesses (b), (c))

DyNeRF, 300 frames, PSNR (Table 1):

MethodCoffee MartiniCook SpinachCut BeefFlame SalmonFlame SteakSear SteakMean
4DGS (Yang et al., 2024a)28.3933.0332.5228.9233.9232.7231.59
4D Scaffold-GS (Cho et al., 2026)29.2832.9333.2228.0032.8432.2631.42
QuaST (Ours)28.9733.4833.8529.4334.1634.1232.33

Representation and objective. 4DGS models spacetime correlation with 4D Gaussian primitives; 4D Scaffold-GS uses 4D anchor features. QuaST retains standard 3D Gaussians and instead enforces motion–shape compatibility during causal optimization. On DyNeRF, QuaST reaches 32.33 dB, above 4DGS (31.59; +0.74 dB) and 4D Scaffold-GS (31.42; +0.91 dB). Regarding 7DGS, we contacted its authors to request access to their implementation but have not received a response. Therefore, we could not conduct a reproducible comparison.

Streaming coverage. ReCon-GS is now evaluated at all three SelfCap horizons (Tables 2–4) and on all six DyNeRF scenes (Table 1). IGS follows a different protocol: AGM-Net is pretrained across four N3DV sequences and tested on only two held-out sequences (Cut Roasted Beef and Sear Steak), whereas QuaST optimizes each of the six scenes causally without cross-scene pretraining. We clarify this protocol difference here rather than comparing incompatible six-scene averages.

3. Applicability to General Offline 4D Reconstruction (Question 1)

The underlying principle may also extend to offline reconstruction. While QuaST's persistent motion-domain graph, parent–child Gaussian correspondences, and local Jacobian estimation are designed for causal frame-by-frame optimization, the principle of motion–shape compatibility is not inherently limited to streaming settings. For example, offline deformation-field methods could potentially incorporate local deformation gradients to guide covariance transport during joint multi-frame optimization.

Future extension. In future work, we plan to investigate how QuaST's local Jacobian regularization and covariance transport can be integrated into existing offline 4D reconstruction frameworks, and whether this integration can further improve reconstruction quality and geometric consistency.

Reviewer_mJL2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

QuaST aims to address the problem of streaming 4D Gaussian reconstruction, with a core focus on fostering synergistic consistency between the motion of Gaussian centers and the evolution of their covariances. The method constructs a persistent motion-field graph and estimates rank-aware local deformation Jacobians based on the displacements of neighboring Gaussians. These Jacobians serve not only to constrain local distortion but also to provide an independent objective for covariance propagation, thereby updating the scale and rotation of the Gaussian components. Furthermore, the method incorporates temporal shape constraints to limit excessive anisotropic scale variations between consecutive frames. Experimental results demonstrate that the approach achieves strong performance relative to the methods used for comparison.

Critical strengths
  1. The motion–shape compatibility formulation has a clear geometric motivation, with explicit treatment of surface-induced rank deficiency and gradient routing.
  2. Long-sequence evaluations include per-frame quality curves and matched late-frame visual comparisons, providing useful evidence of sustained reconstruction fidelity.
  3. Progressive ablations show measurable gains from covariance transport, while also reporting Gaussian counts and training costs.
Critical weaknesses
  1. The comparison protocol does not isolate the method’s contribution. QuaST uses 15 epochs per predictive frame on DyNeRF, whereas the cited QUEEN-l results use 10, despite an average improvement of only approximately 0.14 dB. This discrepancy should be explicitly disclosed and addressed through matched-budget experiments. It is also unclear whether the baselines share the same semantic masks, gate initialization, and static-update policies.
  2. The baseline coverage is incomplete and outdated. Important recent streaming methods, particularly Instant Gaussian Stream[1] and ReCon-GS[2], are missing from the experiments despite being cited. The current comparisons do not sufficiently establish QuaST’s position in the contemporary quality–latency–storage trade-off.
  3. The ablations do not establish the necessity of the central technical choices. Progressive component addition shows conditional improvements but does not separate the proposed geometric objectives from semantic preprocessing or demonstrate superiority over simpler regularizers. Particularly important controls are a semantic-prior-matched baseline, local-rigidity/ARAP and rotation-only transport on the same graph, and targeted tests of graph persistence, rank-aware estimation, and gradient detachment.

Reference: [1] Instant Gaussian Stream: Fast and Generalizable Streaming of Dynamic Scene Reconstruction via Gaussian Splatting, CVPR 2025. [2] ReCon-GS: Continuum-Preserved Gaussian Streaming for Fast and Compact Reconstruction of Dynamic Scenes, NeurIPS 2025.

Questions for authors
  1. Can you disclose all protocol differences and provide a matched-budget QUEEN comparison with identical preprocessing and update policies?
  2. Can you add comparisons with Instant Gaussian Stream and ReCon-GS?
  3. Can you provide controlled alternatives that isolate the benefits of the proposed Jacobian-based transport from semantic priors, simpler geometric regularization, and implementation choices?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mJL2: Matched-Budget Comparison and Recent Baselines

We thank Reviewer mJL2 for the thoughtful review and for recognizing QuaST's clear geometric motivation, surface rank-deficiency handling, and sustained long-horizon fidelity. We have addressed every point with new experiments and explicit summary comparisons in the updated PDF (highlighted in blue).

1. Matched-Budget QUEEN Comparison & Protocol Disclosure (W1 & Q1)

We fully disclose all protocol details in Section 4 and Appendix D.1 (Table 12) and provide a direct 15-epoch matched-budget reproduction of official QUEEN on DyNeRF (500 first-frame epochs, 15 predictive-frame epochs, cam00 held out, 1352 × 1014 resolution, and the same evaluation pipeline):

Method (15-epoch budget, DyNeRF)Coffee MartiniCook SpinachCut BeefFlame SalmonFlame SteakSear SteakMean PSNR↑
QUEEN (reproduced)29.0233.3233.6129.4233.9334.2332.26
QuaST (Ours)28.9733.4833.8529.4334.1634.1232.33
  • Under identical 15-epoch budgets, QuaST achieves 32.33 dB mean PSNR vs. 32.26 dB for reproduced QUEEN (+0.07 dB), outperforming QUEEN on 4 of 6 scenes (+0.24 dB on Cut Beef, +0.23 dB on Flame Steak, +0.16 dB on Cook Spinach, +0.01 dB on Flame Salmon; Tables 1 & 12).
  • Both methods share the exact same dataset splits, resolutions, and evaluation pipeline (Section 4, Appendix A.9), while long-horizon 1,200-frame evaluations (Tables 2–4) further highlight QuaST's drift-free gains (+3.26 dB on Bike, +1.35 dB on Corgi, +1.72 dB on Yoga over QUEEN).

2. Recent Streaming & 4D Baselines (W2 & Q2)

We have added ReCon-GS (NeurIPS 2025 streaming continuum-preserved Gaussians) and Mango-GS (multi-frame node-guided 4DGS; offline/non-streaming) across the 1,200-frame SelfCap streams (Table 4) as well as DyNeRF (Table 1):

Scene (SelfCap 1,200 frames, Table 4)ReCon-GS (Streaming)Mango-GS (Non-Streaming)QuaST (Streaming, Ours)
Bike (PSNR↑ / SSIM↑ / LPIPS↓)21.15 / 0.772 / 0.43715.56 / 0.695 / 0.44225.88 / 0.893 / 0.233
Corgi (PSNR↑ / SSIM↑ / LPIPS↓)20.72 / 0.705 / 0.52510.10 / 0.579 / 0.56227.27 / 0.828 / 0.331
Yoga (PSNR↑ / SSIM↑ / LPIPS↓)12.32 / 0.702 / 0.59410.86 / 0.659 / 0.55119.63 / 0.855 / 0.361
  • ReCon-GS & Mango-GS (Tables 1–4): While already achieving higher PSNR across all evaluated SelfCap horizons than existing streaming baselines (QUEEN, 3DGStream, MoRGS), QuaST also surpasses the newly added streaming baseline ReCon-GS by +4.73 dB on Bike, +6.55 dB on Corgi, and +7.31 dB on Yoga at 1,200 frames (Table 4), as well as by +0.64 dB mean PSNR on DyNeRF (32.33 vs. 31.69 dB, Table 1). Even against the non-streaming node-guided baseline Mango-GS, QuaST leads by +10.32 / +17.17 / +8.77 dB on 1,200-frame SelfCap (Table 4) and by +1.91 dB on DyNeRF (Table 1).
  • Instant Gaussian Stream (IGS): IGS follows a cross-scene pretraining protocol (its AGM-Net is trained on four N3DV scenes and evaluated only on two held-out scenes, Cut Roasted Beef and Sear Steak), whereas QuaST performs per-sequence causal optimization without cross-scene motion pretraining. To avoid mixing two-scene pretraining numbers with six-scene per-sequence averages, we explicitly clarify this protocol distinction and use ReCon-GS as the direct per-sequence streaming baseline across all sequences.
AuthorsOpenReview ↗

Response to Reviewer mjL2 : Controlled Geometric Ablations

3. Controlled Alternatives Isolating QuaST's Geometric Design (W3 & Q3)

We added all requested semantic-prior, local-rigidity, rotation-only, and architectural ablations on SelfCap (Sec. 4.4 / Tables 6–8, App. D.2–D.5 / Tables 13–16):

Scene (SelfCap 600 frames, Table 6)Semantic (QUEEN + SAM/DINO)Semantic + ARAP (32-NN, wt 0.03)QuaST (Ours)
Bike (PSNR↑ / SSIM↑ / LPIPS↓)25.84 / 0.886 / 0.24925.78 / 0.888 / 0.24625.89 / 0.893 / 0.233
Corgi (PSNR↑ / SSIM↑ / LPIPS↓)22.71 / 0.798 / 0.37526.44 / 0.813 / 0.36227.27 / 0.827 / 0.339
Yoga (PSNR↑ / SSIM↑ / LPIPS↓)18.01 / 0.809 / 0.41119.10 / 0.857 / 0.35119.65 / 0.854 / 0.375
Scene (SelfCap 1,200 frames)Rotation-Only ST (Table 8)Uniform 3D Reg. (Table 13)KNN Persistent (Table 14)Filtered Framewise (Table 14)No Stop-Grad (Table 7)QuaST (Full)
Bike (PSNR↑ / SSIM↑ / LPIPS↓)25.83 / 0.894 / 0.23525.72 / 0.893 / 0.23725.79 / 0.894 / 0.23725.87 / 0.893 / 0.23625.76 / 0.893 / 0.23725.88 / 0.893 / 0.233
Corgi (PSNR↑ / SSIM↑ / LPIPS↓)27.13 / 0.830 / 0.33326.97 / 0.829 / 0.33427.12 / 0.831 / 0.33327.09 / 0.831 / 0.33327.11 / 0.829 / 0.33527.27 / 0.828 / 0.331
Yoga (PSNR↑ / SSIM↑ / LPIPS↓)19.56 / 0.856 / 0.35919.57 / 0.856 / 0.36219.60 / 0.857 / 0.35719.80 / 0.859 / 0.35919.89 / 0.858 / 0.36119.63 / 0.855 / 0.361
  • Semantic-prior-matched & ARAP controls (Sec. 4.4 / Table 6): On SelfCap (600 frames), adding the exact same SAM/DINO cues to QUEEN (Semantic) yields 25.84 / 22.71 / 18.01 dB PSNR on Bike/Corgi/Yoga, and adding dynamic-region ARAP (Semantic + ARAP) yields 25.78 / 26.44 / 19.10 dB. QuaST reaches 25.89 / 27.27 / 19.65 dB, surpassing Semantic by +0.05 / +4.56 / +1.64 dB and Semantic + ARAP by +0.11 / +0.83 / +0.55 dB (mean +0.50 dB) with superior SSIM and LPIPS on Bike and Corgi. This confirms that estimating an unconstrained observable Jacobian and coupling it to covariance evolution outperforms local rigidity alone.

  • Full-Jacobian vs. rotation-only transport (Sec. 4.4 / Table 8): Replacing the full Jacobian in Eq. (11) with its nearest proper rotation lowers PSNR across all three SelfCap scenes (25.88 vs. 25.83 dB on Bike, 27.27 vs. 27.13 dB on Corgi, 19.63 vs. 19.56 dB on Yoga), proving the value of transporting anisotropic scale and shear.

  • Rank-aware estimation, graph choices, & gradient detachment (Tables 7, 13–17): Rank-aware subspace estimation improves PSNR by +0.16 / +0.30 / +0.06 dB over uniform 3D regression (Table 13); geometric/co-motion screening improves PSNR on all three SelfCap scenes (+0.09 / +0.15 / +0.03 dB over plain KNN, Table 14); QuaST adapts seamlessly across persistent and framewise graph backends (25.88/27.27/19.63 vs. 25.87/27.09/19.80 dB on SelfCap, Table 14; 32.33 vs. 32.32 dB on DyNeRF, Table 15), with persistence adding +0.18 dB under Corgi's frequent occlusions while immediate framewise updates react slightly faster to Yoga's rapid occlusion-free articulations; and detaching the transport target improves PSNR on 6/9 sequences (+0.12 dB on Bike, +0.16 dB on Corgi, and 32.33 vs. 32.27 dB mean on DyNeRF; Tables 7 & 17), while Table 5 verifies that Shape Transport itself adds +0.42 dB PSNR and +0.018 SSIM on Yoga (where joint updates in the un-detached variant absorb first-order residuals under extreme limb articulations).

Reviewer_A9yvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QuaST augments a QUEEN-style streaming 4DGS with four structural regularizers built on a persistent graph of dynamic Gaussians. Edge log-stretch spread (Graph-QC), a rank-truncated least-squares local Jacobian with anisotropy and scale terms (Local-J), covariance transport of the parent Gaussian by the detached Jacobian (ST), and a per-Gaussian parent-to-child log-scale condition number (TQC) are added to the photometric objective so that center motion and Gaussian shape stay mutually consistent. Updates are strictly causal. Experiments compare against 3DGStream, HiCoM, QUEEN and MoRGS on DyNeRF (300 frames) and SelfCap (600/900/1,200 frames).

Critical strengths

The long-horizon evidence is the paper's most convincing part. Under a strictly causal protocol, prefix-averaged PSNR varies by only 0.02--0.04 dB from 600 to 1,200 frames on all three SelfCap scenes, while QUEEN degrades (e.g., Corgi 26.13 to 25.92, Yoga 19.47 to 17.91), which supports the claim that the structural terms slow drift rather than buy a one-off gain.

The efficiency claim is also real: the full model reaches higher quality than its own base with 31.6%/41.7% fewer Gaussians and lower training time (5.30 h vs. 9.65 h on Yoga), and the paper reports the negative cases honestly.

Critical weaknesses
  1. The "quasiconformal" framing is not supported by the objectives. Dilatation is K=σmax⁡/σmin⁡K=\sigma_{\max}/\sigma_{\min}; the three quantities actually used are the variance of log singular values (Eq. 9), a weighted 90--10 percentile of edge log-stretch spread (Eqs. 4--5), and a max--min of parent-to-child log-scale differences (Eq. 14). None bounds KK, and the Limitations section concedes that no global homeomorphism is established. The concept is a naming device rather than a mechanism. Covariance transport by a deformation gradient is standard continuum mechanics and is exactly the PhysGaussian construction the paper cites; the novelty reduces to estimating the gradient from reconstructed neighborhoods plus a stop-gradient routing rule.
  2. There is no ablation of the stop-gradient in Eq. 11, of the graph screening tests against plain KNN, or of the rank threshold; and no comparison against a plain ARAP or local-rigidity term on QUEEN, which is the obvious alternative explanation for the gains.
  3. The only timing reported is pure training time over frames 2–T, which Appendix A.9 defines to exclude initialization, data loading, evaluation and output storage. SAM 2 propagation and DINOv2 feature matching run over 21–23 training cameras on every frame, so their cost is absent.
Questions for authors
  1. Which objective bounds the dilatation? Please give the relation between Eq. (9) and log⁡K=log⁡(σmax⁡/σmin⁡)\log K=\log(\sigma_{\max}/\sigma_{\min}): for rank-three neighborhoods the variance of log singular values is not a function of KK alone (e.g. σ=(2,1,1)\sigma=(2,1,1) and σ=(2,2,1)\sigma=(2,\sqrt{2},1) both give K=2K=2 but Eq. (9) gives 0.1070.107 and 0.0800.080). If Eq. (5) is the intended bound, why is it a 0.1--0.9 quantile spread with a dead zone at log⁡1.5\log 1.5 rather than a bound on KK?
  2. Please ablate (i) the stop-gradient in Eq. (11), (ii) the graph screening tests against plain KNN, and (iii) the rank threshold. 3.The reported time excludes SAM 2 propagation and DINOv2 matching, which run on 21--23 training cameras per frame. Please report total wall-clock per predictive frame including them; how much of the 5.30 h versus 9.65 h gain over QUEEN remains once they are counted?
2 comments
AuthorsOpenReview ↗

Response to Reviewer A9yv: Controlled Ablations and Wall-Clock Timing

2. New Controlled Ablations (W2 & Q2)

We have added all requested ablations to the main paper and Appendix D (Tables 7, 13–17):

SelfCap Scene (1,200 frames)Without Stop-Grad (Table 7)Uniform 3D Reg. (Table 13)KNN, Persistent (Table 14)Filtered, Framewise (Table 14)QuaST (Full, default)
Bike (PSNR↑ / SSIM↑ / LPIPS↓)25.76 / 0.893 / 0.23725.72 / 0.893 / 0.23725.79 / 0.894 / 0.23725.87 / 0.893 / 0.23625.88 / 0.893 / 0.233
Corgi (PSNR↑ / SSIM↑ / LPIPS↓)27.11 / 0.829 / 0.33526.97 / 0.829 / 0.33427.12 / 0.831 / 0.33327.09 / 0.831 / 0.33327.27 / 0.828 / 0.331
Yoga (PSNR↑ / SSIM↑ / LPIPS↓)19.89 / 0.858 / 0.36119.57 / 0.856 / 0.36219.60 / 0.857 / 0.35719.80 / 0.859 / 0.35919.63 / 0.855 / 0.361
Local-J τ_rank Sensitivity (SelfCap 600 frames, Table 16)τrank=0.01τ_{rank} = 0.01$τ_{rank }= 0.05 $(Default)τrank=0.10τ_{rank} = 0.10Max Scene Spread
Bike (PSNR↑ / SSIM↑ / LPIPS↓)25.89 / 0.893 / 0.23525.89 / 0.893 / 0.23325.72 / 0.894 / 0.2360.17 dB / 0.001 / 0.003
Corgi (PSNR↑ / SSIM↑ / LPIPS↓)27.15 / 0.828 / 0.34127.27 / 0.827 / 0.33927.10 / 0.828 / 0.3410.17 dB / 0.001 / 0.002
Yoga (PSNR↑ / SSIM↑ / LPIPS↓)19.73 / 0.855 / 0.37419.65 / 0.854 / 0.37519.67 / 0.853 / 0.3760.08 dB / 0.002 / 0.002

Stop-gradient in Eq. (11) (Sec. 4.4 / Table 7, App. D.6 / Table 17). Detaching the transport Jacobian establishes a one-way optimization relationship ("motion teacher → shape student"), so the shape-transport loss updates current Gaussian scale and rotation without perturbing center tracking. Retaining stop-gradient improves PSNR on 6 of 9 evaluated sequences: +0.12 dB on Bike (25.88 vs. 25.76), +0.16 dB on Corgi (27.27 vs. 27.11), and DyNeRF mean 32.27 to 32.33 dB (4/6 scenes, up to +0.22 dB on Flame Steak). On Yoga, Table 5 confirms Shape Transport adds +0.42 dB PSNR and +0.018 SSIM over +Local-J; removing stop-gradient improves PSNR (19.89 vs. 19.63), possibly because joint updates absorb linearization residuals under strong articulation.

Graph screening vs. plain KNN & graph-backend adaptability (App. D.3–D.4 / Tables 14–15). On SelfCap (1,200 frames), geometric and co-motion screening improves PSNR across all three scenes over plain KNN (+0.09 / +0.15 / +0.03 dB on Bike/Corgi/Yoga) and lowers LPIPS on Bike/Corgi; KNN has lower LPIPS on Yoga (0.357 vs. 0.361). Temporal edge persistence is a graph-maintenance choice rather than our core contribution. The 3-frame confirmation window helps Corgi under occlusion (+0.18 dB vs. framewise), whereas Yoga benefits from immediate framewise updates (19.80 vs. 19.63 dB). On DyNeRF, the alternative graphs average 32.32 vs. 32.33 dB for QuaST.

Rank-aware estimation & rank threshold (App. D.2 / Table 13, App. D.5 / Table 16). Compared with uniform 3D regression on the same graph, rank-aware subspace estimation improves PSNR by +0.16 / +0.30 / +0.06 dB on Bike/Corgi/Yoga and reduces LPIPS across all three. Sweeping τrank∈0.01,0.05,0.10τ_{rank} ∈ {0.01, 0.05, 0.10} confirms stability: variations ≤ 0.17 dB PSNR, 0.002 SSIM, 0.003 LPIPS. The default 0.05 gives Corgi's best PSNR/LPIPS and joint-best PSNR on Bike without per-scene tuning.

3. Preprocessing & Wall-Clock Timing (W3 & Q3)

Appendix D.7 (Table 18) reports forward-only SAM2/GroundingDINO/DINOv2 preprocessing on training views only, excluding held-out test cameras, using one A100 80 GB GPU with four concurrent processes over 1,200 frames:

  • Yoga (22 cameras): 54.05 min (2.70 s/frame). With 5.30 h predictive-frame training, the sequential sum is 6.20 h (18.62 s/frame) vs. 9.65 h (28.97 s/frame) for QUEEN, retaining a 35.7% reduction, +1.72 dB PSNR, and 31.6% fewer Gaussians.
  • Corgi (23 cameras): 87.70 min (4.39 s/frame). The sequential sum is 7.37 h (22.13 s/frame) vs. 6.49 h (19.49 s/frame) for QUEEN: a 2.65 s/frame overhead, with +1.35 dB PSNR and 41.7% fewer Gaussians.
  • Bike (21 cameras): 52.43 min (2.62 s/frame) preprocessing.

These are preprocessing-inclusive predictive-frame totals; initialization, data loading, evaluation, and output storage remain excluded (Appendix A.9). RGB-only mask extraction can run asynchronously during streaming, but no overlap benefit is assumed here.

AuthorsOpenReview ↗

Response to Reviewer A9yv : Quasiconformal Objectives and Local Dilatation

We thank Reviewer A9yv for the insightful mathematical and experimental questions, and for highlighting QuaST's long-horizon stability (0.02–0.04 dB PSNR variation from 600 to 1,200 frames) and compactness (31.6%–41.7% fewer Gaussians). All new experiments and clarifications have been incorporated into the updated PDF (highlighted in blue).

1. Mathematical Relation Between Quasiconformal Objectives and Local Dilatation K (W1 & Q1)

QuaST controls observable directional distortion at three complementary structural levels rather than imposing the same bound three times.

Local-J (Eq. (9)) — Exact Rank-2 Identity and Tight Rank-3 Bounds.

Let the singular values of the local Jacobian on the observable rank-2 or rank-3 subspace be

σ1≥⋯≥σr>0,r∈2,3.\sigma_1 \ge \cdots \ge \sigma_r > 0, \qquad r \in \\{2,3\\}.

Define the log-stretches and dilatation by

uk=log⁡σk,Kr=σ1σr.u_k = \log\sigma_k, \qquad K_r = \frac{\sigma_1}{\sigma_r}.

For rank-2 surface neighborhoods, Eq. (9) is an exact monotonic function of dilatation:

ℓaniso=14(log⁡K2)2.\ell_{\mathrm{aniso}} = \frac14(\log K_2)^2.

For rank-3 neighborhoods, let

L=log⁡K3=u1−u3,m=u1+u32.L = \log K_3 = u_1-u_3, \qquad m = \frac{u_1+u_3}{2}.

Eq. (9) then decomposes exactly as

ℓaniso=16L2+29(u2−m)2.\ell_{\mathrm{aniso}} = \frac16 L^2 + \frac29(u_2-m)^2.

Since the middle log-stretch lies between the extrema,

16(log⁡K3)2≤ℓaniso≤29(log⁡K3)2.\frac16(\log K_3)^2 \le \ell_{\mathrm{aniso}} \le \frac29(\log K_3)^2.

Consequently,

K3≤exp⁡!(6ℓaniso).K_3 \le \exp\\!\left(\sqrt{6\ell_{\mathrm{aniso}}}\right).

Thus, Eq. (9) is a smooth anisotropy measure yielding an explicit upper bound on K3 while regularizing the middle principal stretch toward the geometric mean of the extreme stretches.

Graph-QC (Eq. (5)) — Empirical Stretch Control on Observed Graph Edges.

For an observed graph edge, the normalized direction is

vij=xij∥xij∥2.\mathbf{v}_{ij} = \frac{\mathbf{x}_{ij}}{\lVert\mathbf{x}_{ij}\rVert_2}.

Its log-stretch is

dij=log⁡∥yij∥2+ϵ∥xij∥2+ϵ.d_{ij} = \log\frac{\lVert\mathbf{y}_{ij}\rVert_2+\epsilon}{\lVert\mathbf{x}_{ij}\rVert_2+\epsilon}.

Under the local linear approximation,

dij≈log⁡∥Fivij∥2.d_{ij} \approx \log\lVert\mathbf{F}_i\mathbf{v}_{ij}\rVert_2.

This samples expansion along observed edges, whereas full dilatation takes extrema over all unit directions. Graph-QC instead uses the weighted 90th–10th percentile spread:

Di=Q0.9(dij;wij)−Q0.1(dij;wij).D_i = Q_{0.9}(\\{d_{ij}\\};\\{w_{ij}\\})-Q_{0.1}(\\{d_{ij}\\};\\{w_{ij}\\}).

The corresponding dead-zone penalty is

ReLU⁡(Di−log⁡K0)2,K0=1.5.\operatorname{ReLU}(D_i-\log K_0)^2, \qquad K_0=1.5.

Percentiles prevent isolated outlier edges from dominating gradients; the dead zone permits non-rigid articulation without enforcing strict conformality. This is empirical directional control, not a hard bound on full-sphere dilatation.

Temporal-QC (Eq. (14)) — Per-Gaussian Log-Dilatation.

Define the shape factor and parent-to-child shape map as

Ait=R(qit)diag⁡(sit).\mathbf{A}_i^t=\mathbf{R}(\mathbf{q}_i^t)\operatorname{diag}(\mathbf{s}_i^t).

Ji,shapet=Ait(Ap(i)t−1)−1.\mathbf{J}_{i,\mathrm{shape}}^t=\mathbf{A}_i^t(\mathbf{A}_{p(i)}^{t-1})^{-1}.

Its singular values are the corresponding parent-to-child axis-scale ratios. Therefore,

κilog⁡=max⁡kΔlog⁡si,k−min⁡kΔlog⁡si,k.\kappa_i^{\log}=\max_k\Delta\log s_{i,k}-\min_k\Delta\log s_{i,k}.

κilog⁡=log⁡K(Ji,shapet).\kappa_i^{\log}=\log K(\mathbf{J}_{i,\mathrm{shape}}^t).

This measures each Gaussian's exact log-dilatation across consecutive frames. Eq. (14) penalizes values exceeding the tolerance KT; it does not impose a hard inequality.

Distinction from PhysGaussian and ARAP (Sec. 2, Sec. 4.4; Tables 6 and 8).

PhysGaussian uses deformation gradients from continuum material constitutive models and an MPM solver. QuaST requires neither: it estimates local Jacobians from reconstructed Gaussian center trajectories on a persistent motion-domain graph (Eq. (6)), then uses a detached target (Eq. (11)) so neighborhood motion guides covariance evolution.

Table 6 shows QuaST exceeds QUEEN + SAM/DINO + ARAP by +0.11 / +0.83 / +0.55 dB PSNR (mean +0.50 dB) on Bike/Corgi/Yoga. Table 8 shows full-Jacobian covariance transport improves PSNR over rotation-only transport in all three scenes, supporting the benefit of propagating estimated scale and shear beyond local rigidity.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and New Experimental Evidence

We sincerely thank the Area Chair and Reviewers A9yv, mJL2, and NxJQ for their constructive feedback and for recognizing QuaST's principled motion–shape compatibility formulation, representation compactness, and sustained long-horizon fidelity over 1,200-frame sequences.

We have uploaded an updated PDF with all revisions highlighted in blue, incorporating matched-budget comparisons, recent streaming and 4D baselines, controlled mechanism ablations, and end-to-end wall-clock breakdowns:

Expanded Baselines & Matched-Budget Protocol (Sec. 2, Sec. 4.1–4.3, Tables 1–4, App. D.1 / Table 12):

  • Added matched-budget QUEEN (15 epochs/predictive frame) on DyNeRF, where QuaST achieves the highest mean PSNR (32.33 vs. 32.26 dB, outperforming QUEEN on 4/6 scenes).

  • Added four recent representative baselines—ReCon-GS, Mango-GS, 4DGS, and 4D Scaffold-GS—covering hierarchical streaming anchors, node-guided deformation priors, and 4D spacetime primitives. QuaST outperforms all of them by +0.64 to +1.91 dB on DyNeRF (Table 1) and outperforms the two evaluated at 1,200 frames on SelfCap (ReCon-GS and Mango-GS) by +4.73 to +17.17 dB (Table 4).

Controlled Mechanism & Component Ablations (Sec. 4.4, Tables 6–8, App. D.2–D.6 / Tables 13–17):

  • Semantic-prior & ARAP controls (Table 6): QuaST surpasses QUEEN + SAM/DINO by +0.05 / +4.56 / +1.64 dB and Semantic + ARAP by +0.11 / +0.83 / +0.55 dB (mean +0.50 dB) on Bike/Corgi/Yoga, verifying that gains arise from coupled local Jacobian estimation and covariance transport rather than semantic masks or local rigidity alone.

  • Full-Jacobian vs. rotation-only transport (Table 8): Full-Jacobian covariance transport improves PSNR across all three SelfCap scenes over rotation-only transport, confirming the benefit of propagating anisotropic scale and shear.

  • Stop-gradient routing (Tables 7 & 17): Detaching the transport Jacobian improves PSNR on 6/9 evaluated sequences (including +0.12 dB on Bike, +0.16 dB on Corgi, and 32.33 vs. 32.27 dB mean on DyNeRF), while progressive ablations (Table 5) confirm that Shape Transport itself adds +0.42 dB PSNR and +0.018 SSIM on Yoga.

  • Rank-aware estimation, threshold sensitivity, & graph backends (Tables 13–16): Rank-aware subspace estimation improves PSNR by +0.16 / +0.30 / +0.06 dB over uniform 3D regression (Table 13); τ_rank ∈ [0.01, 0.10] exhibits strong stability with within-scene variations ≤ 0.17 dB PSNR, ≤ 0.002 SSIM, and ≤ 0.003 LPIPS (Table 16); geometric/co-motion screening improves fidelity across all SelfCap scenes (Table 14), and QuaST adapts seamlessly across both framewise and persistent graph backends (Tables 14–15).

End-to-End Preprocessing & Wall-Clock Cost (App. D.7 / Table 18):

  • Reported forward-only SAM2/GroundingDINO/DINOv2 preprocessing times on training views (2.62–4.39 s/frame). Including all preprocessing, QuaST still reduces end-to-end wall-clock time on Yoga by 35.7% relative to QUEEN (6.20 h vs. 9.65 h; 18.62 s vs. 28.97 s per predictive frame).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.