acceptodds
Under review as a conference paper at ICLR 2027

MoVE-4D: Feed-forward Video-to-4D Generation via 4D Structured Gaussians

Abstract

Generating dynamic 4D assets from monocular videos in a feed-forward manner has rapidly progressed by extending pretrained 3D foundation models to time. However, existing paradigms still treat 3D representations as the atomic unit and introduce temporal modeling externally, either through canonical-shape deformation or per-frame generation coupled by temporal attention. Both designs leave 3D structure unshared at the representation level, limiting motion expressivity in deformation-based methods and leaving a persistent quality–efficiency trade-off in per-frame methods. We propose MoVE-4D, a feed-forward video-to-4D framework built on a 4D-native representation that shares 3D structure across time at the representation level itself. Specifically, we introduce structured 4D Gaussians on motion-occupied voxels, where sparse voxel anchors are reused across frames within a temporal window and time-dependent Gaussian attributes capture local dynamics. MoVE-4D generates this representation in two stages: it first predicts motion-occupied regions through voxel-wise occupancy merging, then populates them with structured 4D Gaussians decoded from a Compact Temporal Structured LATent that aggregates multi-frame visual features anchored at the same voxel. Extensive experiments show that MoVE-4D delivers a Pareto-optimal quality–efficiency trade-off, surpassing existing feed-forward methods in visual quality and spatiotemporal consistency.

Primary area:
generative models
Keywords:
Video-to-4D, 4D Generation, 4D Gaussians

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3FACOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This manuscript presents a feed-forward 4D generation framework, dubbed MOVE-4D, which adopts a TRELLIS-style two-stage generation approach. In the first stage, the model generates motion-occupied voxels from the input video for each temporal group comprising several consecutive frames. In the second stage, a video-conditioned generator transforms these voxels into structured 4D Gaussians. To train the VAEs and flow-based generative models used for motion occupancy and CT-SLAT, the author curates a dataset DYNOBJ4D from ObjaverseXL and Objaverse.

Critical strengths
  1. By applying temporal compression at the representation level, the proposed method achieves a substantial speedup over its two-stage baseline, while the provided quantitative and qualitative results demonstrate that it can achieve comparable or even better generation quality.
Critical weaknesses
  1. The paper lacks detailed descriptions of the architectures and model sizes.
  2. The provided video results in the supplementary material do not show what is the input video in each case.
Questions for authors
  1. Compared with prior 4D generation work also adopting a TRELLIS-style two-stage pipeline, how much efficiency improvement does each stage contribute? It would be helpful to provide a stage-wise runtime breakdown.
Reviewer_C2PZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MoVE-4D, a feed-forward system that converts a monocular object video into a directly renderable dynamic 4D Gaussian representation. For each temporal group of τ\tau frames, Stage 1 predicts the union of occupied voxels using an occupancy VAE and a video-conditioned rectified-flow generator. Stage 2 aligns multi-frame visual features on this shared support, aggregates them into a Compact Temporal Structured LATent (CT-SLAT), and decodes each active voxel into native 4D Gaussian primitives. Cross-group temporal attention is used in both generators. The model is trained on roughly 20K animated Objaverse assets and evaluated on DYNOBJ4D, ACTIONBENCH, and a 20-video DAVIS subset. The reported results improve all appearance metrics on DYNOBJ4D, approach the strongest geometry-specific method on ACTIONBENCH, and require 20 seconds for a 24-frame input.

Critical strengths
  1. I find the representation well matched to the stated source of redundancy. A single motion-occupied support is reused within each temporal group, while native 4D Gaussians retain time-varying geometry and appearance. This is more expressive than deforming one canonical asset and potentially more economical than generating an independent 3D representation at every frame. The comparisons across τ\tau, mean versus learned aggregation, and temporal-attention variants provide evidence that the benefit does not come from temporal downsampling alone.

  2. The empirical evaluation is unusually broad for a feed-forward video-to-4D paper. MoVE-4D is evaluated for novel-view appearance, temporal quality, geometry, inference time. It achieves the best reported DYNOBJ4D appearance scores, remains close to the geometry-specialized ActionMesh on CD-3D/CD-4D, and is substantially faster than SS4D. Fine-tuning the closest baseline on the authors' data, testing group-boundary behavior, and disclosing a fast-motion failure case make the comparison more convincing.

Critical weaknesses
  1. I think that interpreting M(g)M^{(g)} as representing a shared 4D structure is not sufficient.CT-SLAT aggregates observations by fixed voxel coordinate rather than by motion-aware correspondence. The same material point moves across voxels, whereas different body parts may occupy the same voxel at different times. Moreover, the group support M(g)=max⁡t∈TgOtM^{(g)}=\max_{t\in\mathcal{T}_g} O_t retains only the spatial union and discards occupancy time, order, direction, and repeated visits. Without scene-flow warping, point tracking, or explicit correspondence, the aggregation can mix unrelated features under articulation, self-contact, or large motion. It is difficult for stage 2 to infer precisely the temporal information removed by Stage 1.

  2. The compactness assumption depends on motion overlap and a fixed temporal scale. The union support is compact only when the per-frame occupancies overlap substantially; with large displacement it can approach the sum of the per-frame supports, eliminating much of the intended saving. A fixed τ=8\tau=8 also imposes one compression rate on slow and fast motion. In addition, a single 4D Gaussian most naturally represents a locally smooth space-time trajectory, while the fixed allocation of 32 Gaussians per voxel does not adapt to motion complexity. Abrupt acceleration, topology changes, or repeated crossings may therefore be oversmoothed or require inefficient duplication. The limited comparison among τ∈{1,8,12}\tau\in\{1,8,12\} does not establish robustness across motion speeds or frame rates.

  3. Cross-group continuity is encouraged but not enforced. Each temporal group has separately decoded occupancy, CT-SLAT, and Gaussian sets that are merged at rendering time. Cross-group attention is a soft communication mechanism, but there is no explicit identity, scene-flow, velocity, geometry-continuity, or boundary loss, I think it's necessary to incorporate these losses, and perhaps we will achieve even more SOTA results.

Questions for authors
  1. How does CT-SLAT distinguish different material points that visit the same voxel at different times, and how does it associate one moving point across different voxels?

  2. What prevents independently decoded temporal groups from producing identity switches, trajectory discontinuities, or Gaussian overlap at group boundaries?

I believe the main flaw of the paper lies in the fact that there are some design issues that were not taken into account regarding the continuity and smoothness of the movement of different parts of the object in the temporal dimension. If in the subsequent rebuttal stage, the author can provide some theoretical explanations and experimental illustrations for this problem, I will actively consider giving it a higher score.

Reviewer_HP5DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MoVE-4D, a feed-forward video-to-4D framework that reuses motion-occupied voxel anchors across time and models dynamic content through time-dependent Gaussian attributes. A two-stage pipeline first generates motion-occupied regions, then decodes structured 4D Gaussians from a Compact Temporal Structured Latent (CT-SLAT). The experiments report favorable visual quality, temporal consistency, and inference efficiency.

Critical strengths
  1. Representation-level sharing addresses redundant spatial modeling across video frames.
  2. CT-SLAT aggregates multi-frame information at shared voxel locations while allowing local temporal variation.
  3. The reported reconstruction quality and inference time compare favorably with the evaluated feed-forward baselines.
  4. Experiments examine motion occupancy, temporal attention, aggregation functions, and a fine-tuned SS4D baseline.
Critical weaknesses
  1. Voxel supports, temporal aggregation, and time-dependent Gaussians have precedents. Matched-capacity comparisons would clarify what is gained specifically from the proposed shared-voxel representation, beyond pipeline integration.
  2. In Table 3, replacing predicted occupancy with ground-truth occupancy increases PSNR from 18.525 to 24.849. This large difference deserves a more detailed study of occupancy errors and their downstream effects.
  3. The central efficiency argument depends on temporal sharing, yet the experimental focus is on short sequences. Memory, Gaussian count, speed, and quality should be measured as video length and motion amplitude increase.
  4. More evaluation on dynamic scenes with reference novel-view observations would help support the claimed generalization.
Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.