acceptodds
Withdrawn from ICLR 2027

Stab4D: 4D Gaussian Splatting for Dynamic Video Stabilization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstruction-based video stabilization queries camera views that are close to, but not identical to, the observed shaky trajectory, thus creating a mismatch between observed-view optimization and stabilized-view deployment. For example, misplaced primitives can reproduce raw frames yet become visible as floaters after trajectory smoothing in Gaussian reconstruction. To address this issue, we introduce Stab4D, a local 4D Gaussian framework that shares geometry over short temporal windows while accommodating dynamic content with evidence-guided temporal lifespans. Stab4D contains Contribution-Guided Depth (CDepth) as its core component, which uses renderer-native alpha-compositing contribution to route monocular depth evidence to rendering-relevant Gaussian primitives, with detached soft weighting and fixed-topology refinement after densification. Besides, a geometry-certified stable-view tube transfers reliable neighboring RGB-D evidence to the camera neighborhood between raw and stabilized trajectories, while unsupported exposure is conservatively suppressed. We also construct a seven-scene controlled dynamic benchmark with paired shaky/stable RGB, shared oracle geometry, and a common stable target trajectory. Under this protocol, Stab4D achieves overall stronger stable-view reconstruction quality than competing reconstruction baselines while maintaining comparable raw-view fidelity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video stabilization, 4D Gaussian splatting, dynamic scene reconstruction, Gaussian splatting, novel-view synthesis, geometry regularization, monocular reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2JrbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Stab4D addresses dynamic video stabilization by reconstructing local 4D Gaussian fields over overlapping temporal windows and rendering them along a smoothed camera trajectory. The method combines primitive-wise spline motion with evidence-guided temporal lifespans. Contribution-Guided Depth assigns depth supervision to individual Gaussians using their alpha-compositing contributions, while stable-view tube supervision transfers neighboring RGB-D observations to nearby target views. An additional objective suppresses newly exposed contributions without reliable observational support. Evaluation includes real NUS videos and a seven-scene synthetic dynamic benchmark with shared oracle geometry and stabilized target trajectories. The reported results show improvements in stable-view PSNR and SSIM, although perceptual quality metrics exhibit trade-offs.

Critical strengths
  1. The controlled evaluation isolates a consequential source of error. Providing reconstruction baselines with the same camera parameters, depth, and stabilized target trajectory helps separate representation quality from geometry estimation and trajectory smoothing. Reporting both raw-view and stable-view reconstruction quality directly tests whether fitting observed frames transfers to the views required for stabilization.
  2. The CDepth ablation provides targeted empirical evidence. In Table 3, CDepth achieves a stable-view PSNR of 31.925 dB, compared with 30.487 dB for rendered-depth supervision, while raw-view PSNR remains essentially unchanged. This suggests that the proposed geometric refinement can improve nearby-view rendering without simply improving the fit to the input frames.
Critical weaknesses

The paper should include comparisons with more dynamic Gaussian reconstruction methods, such as Temporal Gaussian Hierarchy[1], which uses a multiscale temporal representation of 4D Gaussians for dynamic reconstruction.

[1] Xu Z, Xu Y, Yu Z, et al. Representing long volumetric video with temporal gaussian hierarchy[J]. ACM Transactions on Graphics (TOG), 2024, 43(6): 1-18.

Questions for authors
  1. During reconstruction, the video is divided into multiple windows, and a local 4D Gaussian representation is reconstructed within each window. Could this affect reconstruction speed? The method should be compared with global 4D Gaussian reconstruction in terms of both reconstruction speed and rendering quality metrics.
  2. How is Gaussian overlap or stacking handled where local 4D Gaussian representations from different windows are stitched together? The 4D Gaussian field in Figure 2 appears to contain noticeable stitching seams. These overlapping Gaussians could produce floaters in novel views.
Reviewer_82HaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Stab4D addresses monocular dynamic video stabilization by optimizing a local 4D Gaussian representation and rendering it along a smoothed camera trajectory. The method combines primitive-wise B-spline motion and evidence-guided temporal lifespans with contribution-guided depth regularization, neighboring-frame RGB-D supervision along a raw-to-stable camera tube, and suppression of unsupported newly exposed Gaussian contributions. Evaluation on NUS and a seven-scene controlled Blender benchmark reports improved average NUS stability and stronger stable-view PSNR/SSIM; the controlled comparisons use shared camera/depth inputs and target trajectories, with component ablations examining the proposed design.

Critical strengths
  1. Evidence-guided temporal lifespan modeling. The paper introduces a reasonable strategy that uses cross-frame visibility and depth consistency to guide Gaussian lifespans, allowing persistent structures to share geometric information while adapting to transient dynamic content. Treating occluded and low-confidence observations as unknown rather than negative evidence is a thoughtful design, and the ablation results demonstrate the effectiveness of adaptive temporal support.
  2. The authors construct a seven-scene dynamic Blender benchmark with paired shaky/stable views, ground-truth geometry, and controlled camera trajectories. The careful calibration of synthetic camera shake using real-world motion statistics makes the benchmark a valuable contribution for evaluating reconstruction-based video stabilization.
  3. The paper presents comprehensive experiments on both real-world and synthetic datasets, including comparisons with multiple video stabilization methods, component ablations, and detailed analyses of temporal lifespans, depth regularization, and optimization settings. These experiments demonstrate considerable effort and provide useful evidence for evaluating the proposed method.
Critical weaknesses

1.Temporal consistency across local windows: The method optimizes separate local 4D Gaussian fields, which may contain different numbers of primitives and different motion trajectories. Although the paper mentions fusion between overlapping windows, it does not clearly explain how this fusion is performed or how temporal consistency is maintained across window boundaries. Differences in geometry and motion between adjacent windows may introduce flickering, ghosting, or discontinuities. Since the method is based on 4DGS, continuous 4D visualizations showing Gaussian geometry and motion across multiple windows, together with targeted evaluations of temporal consistency at window boundaries, would help validate the effectiveness of the proposed representation. 2. Necessity and reliability of stable-view tube supervision: Since the stabilized camera trajectory typically lies close to the original shaky trajectory, it is unclear whether sampling additional intermediate virtual views is necessary, or whether directly supervising the final stabilized views would achieve similar results. Moreover, the pseudo RGB-D targets rely on geometry estimated by VGGT/VGGT4D, making the supervision potentially sensitive to depth and camera pose errors. As the controlled benchmark primarily uses ground-truth geometry, the robustness of this supervision to inaccurate geometric estimates remains insufficiently validated. 3. The proposed 4DGS representation combines Gaussian temporal lifespans with B-spline motion trajectories, which appears conceptually similar to ExGS. However, the paper lacks a direct comparison with ExGS, particularly in terms of reconstructed geometry and temporal motion consistency. Although the primary application is video stabilization, the method also performs monocular dynamic 4D reconstruction. Therefore, comparisons with representative monocular dynamic reconstruction methods, such as MoSca and Shape of Motion, would help better evaluate the effectiveness and advantages of the proposed 4D representation.

Questions for authors
  1. Availability of supplementary results: Since this work is based on a 4D Gaussian representation, could the authors provide the stabilized videos, lateral camera sweeps, and Gaussian PLY files mentioned in Appendix G? These materials would help assess the temporal consistency and geometric quality of the reconstructed dynamic scenes beyond the selected qualitative images.
  2. Dataset availability: Could the authors clarify whether the proposed seven-scene controlled dynamic benchmark will be publicly released, including the paired shaky/stable videos, camera poses, depth maps, and evaluation protocols? Making this dataset available would facilitate reproducibility and future comparisons.
Reviewer_WNWfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Stab4D frames video stabilization as a nearby-view video novel view synthesis (NVS) task. One common approach is reconstruction-based stabilization, where the input video is reconstructed (e.g., with Gaussians) and rendered at the smoothed trajectory. However, since the reconstruction is optimized at the observed views, which slightly deviate from the rendered views, artifacts such as floaters become visible when rendering from the smoothed trajectory. Stab4D is a local 4D Gaussian Splatting (4DGS) framework that initializes from VGGT/VGGT4D depth, poses, and confidence. It addresses the floaters problem with Contribution-Guided Depth (CDepth), where each Gaussian's own depth is independently supervised against the (estimated) monocular depth of its most contributing pixel, preventing correct composited depth from hiding Gaussians with incorrect individual depths that show up with NVS. This is further refined by optimizing the 4DGS to match RGB-D from nearby frames, warped to cameras along a stable-view tube and kept only at pixels passing consistency checks. Stab4D is validated on NUS and on a new seven-scene synthetic benchmark with GT geometry.

Critical strengths
  1. Well-motivated and validated: The paper clearly identifies that representations fitted at shaky input cameras have problems when actually rendered at nearby stabilized cameras for video stabilization, which is a specific failure mode of reconstruction-based stabilization. The effectiveness of Stab4D on addressing this particular failure mode is clearly shown in the paper: while Stab4D and RStab show similar PSNR for the shaky (input) views, Stab4D is ~2.3dB above RStab for PSNR for stable (nearby) views in Table 2. Table 3 also shows how, after adding the evidence-guided lifespan, each subsequent component incrementally raises the stable (nearby-view) PSNR while raw (input-view) PSNR stays about constant. The controlled benchmark clearly shows Stab4D's effectiveness at addressing this input-to-rendered view transfer.
Critical weaknesses
  1. The evaluations don't establish real-world robustness and performance: The paper provides two types of evaluations: NUS (real-world videos and frontends) and the controlled benchmark (synthetic videos and GT geometry). The former uses estimated geometry and is thus closest to real-world video stabilization, where we usually don't have GT geometry, but the baselines generally don't use the same geometry frontend as each other or as Stab4D (which the authors identify in L078–079 as well). The latter provides each baseline with the same geometry and isolates representation/rendering quality, but the GT geometry means the evaluation does not test real-world performance. The fairest comparison is to use the same geometry frontend for each baseline and Stab4D (VGGT4D, for example), so we can isolate each method properly but still evaluate them under real-world conditions. Moreover, different geometry frontends should also be tested here (Pi3/Pi3X, Depth Anything 3, VGGT-Ω, and even earlier methods such as CUT3R and MegaSaM) to test each method's robustness to different geometry frontends and their potential artifacts.
  2. Main components similar to SparseGS: Two of Stab4D's main components (CDepth and tube supervision) have very similar analogues in SparseGS. SparseGS supervises (approximately) the depth of the dominant Gaussian per pixel (Stab4D does dominant pixel per Gaussian), and prunes Gaussians where the dominant Gaussian's depth and the blended alpha depth disagree, which is close to CDepth's motivation. SparseGS also has a masked depth warping loss on nearby virtual views, similar to Stab4D's tube supervision. Although the paper is explicit that CDepth "does not redefine this failure mode" (L143–144) when citing SparseGS in Related Work, the authors have not clearly distinguished CDepth, which is framed as a key component and contribution of Stab4D, from SparseGS's mode depth selection, and the tube supervision is not discussed relative to SparseGS's depth warping loss at all. A Table 3 ablation (and ideally, following weakness #1, a real-world ablation) replacing CDepth with SparseGS's mode depth supervision, along with an ablation replacing stable-view tube supervision with SparseGS's depth warping loss, will help clarify this.
Questions for authors

Clarification requests:

  1. Missing supplementary materials: Appendix G mentions that "the supplementary material provides additional stabilized videos and frame-level comparisons on both NUS and the controlled dynamic benchmark" (L1131–1132). However, on OpenReview there doesn't seem to be an attached supplementary material zip that can be downloaded and viewed, making it very difficult to visually judge the stabilization and visual quality of Stab4D as compared to the baselines. Where can we find these materials?
  2. Tube parameters and ablation: Currently, tube supervision samples λ∈Λ⊂(0,1]\lambda \in \Lambda \subset (0, 1], yet the final stabilized video is meant to be rendered at the stabilized target poses. Because of this, why not just do tube supervision only at λ=1\lambda = 1? Λ⊂(0,1]\Lambda \subset (0, 1] is also never specified, so how is Λ\Lambda chosen, and why even define a subset instead of just sampling from the full (0,1](0, 1]? Some ablations on only doing tube supervision on stabilized poses and also on a more widely sampled tube can justify this choice.
  3. LPIPS gap: Table 2 shows RStab obtaining lower LPIPS than Stab4D at both shaky views and stable views, though Stab4D is better than all baselines on PSNR and SSIM. The paper explains this as due to "different behavior across complementary fidelity metrics" (L376). Since LPIPS is the most perceptual metric of the three (the most high-level compared to the lower-level/local PSNR and SSIM) and is thus more sensitive to blur and texture artifacts, can the authors explain where this gap comes from? For example, is Stab4D's outputs smoother than RStab's in certain regions, and do these gaps come from specific regions (e.g., tube-supervised, exposure-suppressed, or moving/dynamic subjects)? Showing either per-scene LPIPS or heatmaps/crops showing where the LPIPS gaps are at can help.

Future work:

  1. Application to monocular-to-stereo video synthesis: Synthesizing stereo videos from monocular videos is also a near-view novel view synthesis problem. Can Stab4D be applied to mono-to-stereo video synthesis? If the authors have the bandwidth to run this, how does Stab4D perform on mono-to-stereo benchmarks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.