acceptodds
Withdrawn from ICLR 2027

SplatFlux: Feed-Forward 4D Gaussian Reconstruction and Generation from Video Latents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This work presents SplatFlux, a feed-forward latent-to-4DGS framework for both video-based 4D reconstruction and conditional 4D generation. Our key observation is that observed videos and conditionally generated content can be represented in the same pretrained video latent space, allowing both tasks to be formulated as latent-to-4DGS decoding. To construct compact and temporally coherent 4D scenes, SplatFlux introduces a static-dynamic decoupled 4D Gaussian representation that explicitly predicts dynamic probabilities for Gaussian primitives. This representation separates static scene structures from moving regions, enabling complete 4D scene construction in a feed-forward manner through static aggregation and dynamic motion interpolation. To learn static-dynamic decomposition and dynamic Gaussian motion without costly annotations, we further develop an auxiliary distillation strategy that provides pseudo dynamic masks and optical-flow cues. Extensive experiments across 3D and 4D reconstruction and generation demonstrate that SplatFlux achieves state-of-the-art performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Feed-Forward Gaussian Splatting, 4D Reconstruction, 4D Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ac5SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SplatFlux decodes per-frame Wan VAE latents into Gaussian scenes for reconstruction and conditional generation. It aggregates static Gaussians across frames and interpolates dynamic Gaussians using predicted velocities, allowing intermediate-time rendering without another network prediction. Training uses geometry, rendering, mask, and flow supervision. a fine-tuned Wan2.1 supplies generation latents.

Critical strengths
  • The shared decoder connects reconstruction and generation through one representation. Independent frame encoding and generator adaptation make the latent interface explicit.
  • NVIDIA Dynamic Scene PSNR reaches 21.56 versus NeoVerse's 18.83, with LPIPS of 0.1918 versus 0.2881 (Table 1). Six-view static reconstruction also improves over DA3, 22.17 versus 20.86 PSNR (Table 8).
  • The ablations support the representation design: removing static-dynamic decoupling drops PSNR to 13.94; removing mask or flow supervision gives 18.41 or 18.62 (Table 5). These are substantial losses from the full model's 21.56.
Critical weaknesses
  • Generation comparisons use different video backbones (Tables 3-4). Without an RGB-generation-then-reconstruction baseline using the same Wan model and samples, gains cannot be attributed to latent decoding. Appearance and motion scores alone do not establish better 4D geometry.
  • DAVIS splits one captured sequence into input and evaluation frames, testing temporal reconstruction along that camera path rather than independently observed simultaneous viewpoints. DyCheck also pools held-out-camera and same-camera timestamps. Separate view/time results would better support arbitrary-view and arbitrary-time claims.
  • Equation 4 combines dynamic Gaussians from adjacent frames without explaining competing primitives, occlusion, or visibility changes. Dynamic thresholds and voxel-aggregation settings are unspecified. Fast motion and wider temporal gaps would be informative stress tests for local linear interpolation.
Questions for authors
  1. With the same Wan model and samples, how does latent decoding compare with RGB generation followed by reconstruction, including geometry-sensitive metrics?
  2. How do results differ across held-out views, times, and joint view/time pairs? How are adjacent Gaussian sets combined when visibility changes?
  3. What are latency, throughput, memory, and Gaussian counts at matched resolutions and sequence lengths? Please include the dynamic threshold and aggregation settings.
Reviewer_adLqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a unified framework for 4D Gaussian reconstruction and generation. The unified framework is based on the Wan VAE latent space, with an additional decoder for Gaussian attributes and motion-related properties prediction. To train the model which requires motion-related supervision like mask and velocity, the method applies foundation/pre-trained models to provide pseudo labels. The idea of unified latent space for different tasks is interesting, and the paper is easy to follow.

Critical strengths

I think the idea of using Wan latent space for different tasks is interesting, as it can naturally extends from 4DGS reconstruction to 4DGS generation with a simple framework.

Critical weaknesses
  1. Though the paper claims that the framework could support both 4DGS reconstruction and generation, the core of the method is a decoder attached after the Wan latents, since the decoder mainly predicts 4DGS properties from a set of given Wan latents no matter from VAE or DiT. Given a set of latents, the contribution is essentially a feed-forward 4DGS network whose input is latent compared to RGB frames of other works. Therefore, I think the idea is interesting to some extent, but not that novel.
  2. I think the experiments are not convincing. Table 1 mainly assesses the novel view synthesis performance. I believe that there are a bunch of recent works that could support 4DGS reconstruction, no matter optimization-based or feed-forward ones. The authors should provide more comparisons.
Questions for authors
  1. I am not familiar the typical method for 4DGS reconstruction, but I think predicting mask and velocity might not be that novel.

  2. I am a little bit curious about how is the performance if we remove the pretrained DA3 weights from the decoder? As the input space is totally different, I am wondering if it is useful or not.

Reviewer_c44FOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a shared Wan-latent-to-4DGS decoder for both reconstruction from observed videos and text- or image-conditioned 4D generation. It predicts per-frame Gaussians with dynamic probabilities and bidirectional 3D velocities, aggregates static Gaussians across frames, and interpolates dynamic Gaussians locally. Training uses rendering and geometry losses together with pseudo dynamic masks and optical flow. The evaluation includes novel-view synthesis from real videos and perceptual metrics for conditionally generated scenes.

Critical strengths
  1. A useful shared decoding formulation. The same decoder accepts latents from observed videos and from a conditional video generator, yielding an explicit 4DGS scene in either case. Direct conditional generation of a renderable representation is the paper’s clearest contribution.

  2. Strong real-video reconstruction evidence. The paper reports improved novel-view synthesis on several benchmarks, particularly NVIDIA Dynamic Scene and DAVIS. Its ablations also show substantial benefits from static–dynamic decomposition and the auxiliary mask and flow supervision.

Critical weaknesses
  1. Long-term motion modeling remains unverified. Although the decoder processes multiple frames, dynamic Gaussians are propagated mainly within local temporal neighborhoods. It is unclear whether the representation can maintain persistent 3D trajectories beyond adjacent-frame interpolation. Please report optical-flow or 3D tracking results at increasing temporal gaps, with MoVieS and NeoVerse as comparisons.

  2. The usable novel-view range and geometric quality of the 4DGS are insufficiently demonstrated. Tables 2–3 mainly measure perceptual quality, while the demos show limited viewpoint changes. Please include wider novel views and occluded regions. For example, showing the NVIDIA Balloon1 scene at the novel-view camera would help determine whether SplatFlux improves the partial balloon occlusion problem compared with the MoVieS and NeoVerse results.

  3. Novel-view and novel-time evaluations are mixed. The DyCheck score combines cross-camera results from camera 1 with held-out frames from camera 0, while DAVIS evaluates held-out frames from a single video. Please report these settings separately for novel time synthesis and novel view synthesis. The current mixing setting can not let us truly understand the real performance under the large novel view angle in DyCheck.

  4. Table 2 includes Wan + AnySplat through VIST3A, but Tables 2–3 do not include "Wan + MoVieS" or "Wan + NeoVerse". The same Wan-generated latents could be decoded directly by SplatFlux or converted to RGB frames and reconstructed by these methods. Comparing fixed-time 3D results and full-sequence 4D results would clarify the benefit of direct latent-to-4DGS decoding.

Questions for authors

The paper describes bidirectional linear velocities, whereas the supplementary code defaults to a four-control-point spline. Which motion parameterization was used for the checkpoints in Tables 1–3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.