acceptodds
Withdrawn from ICLR 2027

LEG4D: Language Embedded Generative 4D Gaussian Splatting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing complete and semantically meaningful 4D scenes from monocular videos is inherently challenging due to limited observations of moving objects. Existing dynamic reconstruction methods mainly recover visible geometry and appearance, while generative approaches often require costly score-distillation optimization and lack dense semantic representations. We present LEG4D, a Language-Embedded Generative 4D Gaussian Splatting framework for complete, semantic, and editable dynamic scene reconstruction. Given a monocular video and object masks, LEG4D employs a pretrained 3D object generation model to complete partially observed objects as object-level Gaussians. The generated Gaussians are integrated into the observed scene through pose- and geometry-aware alignment, multi-view appearance adaptation, and object replacement. A motion-aware dynamic Gaussian module subsequently optimizes the fused scene while maintaining Gaussian correspondences and trajectories across time. We further distill language-aligned features from observed frames into the dynamic Gaussians, producing a complete, temporally consistent, and language-embedded 4D representation. Experiments demonstrate that LEG4D improves both dynamic scene completeness and language-grounding accuracy over prior approaches. The resulting representation additionally supports object-level querying, localization, removal, and localized editing in dynamic scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Scene Reconstruction, Language-Embedded Representation, Generative 3D Completion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZXrdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LEG4D addresses complete and language-queryable dynamic scene reconstruction from monocular videos and object masks. It uses pretrained SAM 3D to complete partially observed objects, aligns the generated objects with the scene, and adapts their appearance using MV-Adapter. The completed objects are incorporated into a dynamic Gaussian representation through shared object-level motion coefficients. Language features are subsequently distilled into the reconstructed scene, with shared object embeddings transferring semantic identity to unobserved regions. Experiments evaluate novel-view synthesis on two synthetic datasets and text-based querying on one dataset, alongside qualitative demonstrations of object removal and appearance editing.

Critical strengths
  1. This paper addresses a problem: in monocular dynamic reconstruction, it is difficult to reconstruct regions of objects that have not been observed for an extended period. Integrating generative object completion techniques into a 4D representation that can be queried via language is both practical and meaningful.
  2. LEG4D improves new-view rendering on Mesh4D-6, boosts the mIoU for text queries on Kubric4D-25, and supports object querying, removal, and appearance editing.
Critical weaknesses
  1. The model’s methodological innovation is limited, and the performance ablation is unclear. The model’s core architecture relies heavily on existing techniques, including SAM3D, dynamic Gaussian motion modeling, MV-Adapter, and language feature distillation. The main contribution of this paper lies in combining these techniques with object-level motion/semantic sharing; however, the lack of targeted ablation experiments fails to demonstrate the extent to which these new designs contribute beyond a simple combination of existing components.
  2. Validation with real-world experimental data is insufficient. Both evaluation datasets consist of synthetic data, and several claimed capabilities are demonstrated primarily through qualitative means.
Questions for authors

Under conditions where object masking, generative priors, appearance supervision, and preprocessing are all identical, how does the simplified combination of base components perform compared to LEG4D? Please include targeted ablation studies to isolate the effects of alignment, object-level motion sharing, and semantic sharing, respectively, to clarify the contribution of this method beyond the pre-trained components.

Which views and timestamps were used for the semantic results in Table 2? Do these improvements hold true in object regions not observed in the training videos? Given the synthetic nature of the dataset, can the available geometric and motion information support direct metrics for completeness and motion/temporal consistency?

How does LEG4D generalize to real monocular videos with imperfect masking, depth estimates, and correspondences? Please provide results from real-world scenarios, as the current evaluation is based solely on synthetic data.

Reviewer_2ntzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In the paper, the authors propose LEG4D, a pipeline for reconstructing complete, language-embedded 4D Gaussian scenes from monocular videos with object masks. Partially observed objects are completed with a pretrained 3D generative model (SAM 3D), aligned to the scene, and their appearance is adapted using multi-view images generated by MV-Adapter. The fused scene is animated with rigid motion components, and language features are distilled into the Gaussians, with one shared embedding per completed object. Experiments on two synthetic datasets (Kubric4D-25 and Mesh4D-6) show improvements in novel-view synthesis and text-based querying over reconstruction, generative, and semantic-field baselines. According to the authors, the approach also supports object removal and appearance editing.

Critical strengths
  • The LEG4D pipeline stages are clearly described, and the source code is provided, which supports the reproducibility of the results.

  • Within the evaluated setting, LEG4D outperforms all baselines on all reported metrics, in both novel-view synthesis and text-based querying.

Critical weaknesses
  • The method mainly combines existing components. Although the integration is carefully designed, the contribution is largely an engineering one.

  • Experiments use only synthetic datasets, although the method is motivated by real monocular videos. No results are shown on real-world benchmarks such as DAVIS. Ablations are restricted to removing MV-Adapter. Object removal and appearance editing are demonstrated only in the teaser figure (Fig. 1).

Questions for authors
  • Could the authors evaluate their method on real-world datasets (e.g., DAVIS) and extend the ablation study?

  • How do the authors explain the large PSNR drop on Mesh4D-6 without MV-Adapter, and why is it described as a slight degradation?

Reviewer_T2MwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LEG4D, a framework for reconstructing complete, language-queryable dynamic Gaussian scenes from monocular videos and object masks. The method uses SAM3D to generate complete object representations, aligns them with the reconstructed scene, and adapts their appearance using MV-Adapter. The completed objects are incorporated into a dynamic Gaussian representation with shared motion coefficients. Language features are subsequently distilled into the fixed reconstruction, with shared object embeddings transferring semantic supervision to unobserved regions. Experiments on two synthetic datasets evaluate novel-view synthesis and text-based object querying, alongside qualitative demonstrations of object removal and appearance editing.

Critical strengths
  1. The paper addresses a worthwhile problem. Completing partially observed objects while retaining language-based access to them could make dynamic reconstructions more useful for querying and editing. The pipeline provides a practical way to bring object-level generative priors into a persistent dynamic scene representation.
  2. Sharing motion coefficients and semantic embeddings across completed objects is a reasonable structural choice. In particular, a shared semantic embedding allows supervision from visible regions to reach otherwise unobserved parts of an object. The reported improvements in language-query segmentation and the qualitative editing examples are encouraging. I also appreciate that the limitations section explicitly acknowledges the restrictions concerning non-rigid deformation and time-varying semantics.
Critical weaknesses
  1. The incremental contribution is not sufficiently established. The pipeline combines several closely related existing techniques: motion bases resembling those in Shape of Motion, SAM-based CLIP feature distillation and scene-specific compression as in LangSplat, and object-centric generative completion for dynamic reconstruction. Such a combination can be valuable, but the experiments need to show what the proposed integration adds. The main quantitative ablation removes MV-Adapter, leaving the effects of object replacement, alignment refinement, shared motion coefficients, and shared semantic embeddings largely unresolved. Without controlled comparisons, I find it difficult to distinguish the contribution of the proposed designs from the benefits of the pretrained components. The description of “language-guided object completion” also needs clarification, since language learning appears to occur after geometry reconstruction, with geometry held fixed.
  2. The evaluation does not directly establish several central claims. Aggregate query segmentation scores do not establish temporal semantic consistency. Both datasets are synthetic, making geometric and motion ground truth potentially available, yet the paper does not report direct evaluations of surface completeness, unseen regions, or trajectories. The large semantic improvement is also difficult to interpret without comparing semantic methods on the same reconstructed geometry and documenting their access to object masks and other priors. These gaps limit the evidence for complete and temporally consistent semantic reconstruction beyond the demonstrated rendering and querying results.
Questions for authors
  1. Could the authors isolate the contributions of object completion, alignment refinement, shared motion coefficients, and shared semantic embeddings? For the semantic component, a comparison on identical reconstructed geometry would help determine whether the improvement comes from better geometry, object-level feature sharing, or the language-learning procedure.
  2. On page 6, the object-to-scene transformation is written as T_coarse^(-1) T_o*. Given the stated definitions, should this instead be T_t^(-1) T_o*, where T_t maps scene coordinates to camera coordinates? Please clarify the implemented transformation chain, including anisotropic scaling and the camera-space depth correction.
  3. Table 1 reports a PSNR decrease from 19.64 to 8.467 on Mesh4D-6 when MV-Adapter is removed, whereas the text describes the degradation as slight. Is 8.467 correct? If so, what causes this substantial failure, and is it concentrated in particular scenes?
  4. Can the authors provide direct evidence for geometric completeness and temporal consistency, particularly on regions unobserved during training? Please also clarify whether semantic evaluation uses held-out views, how ground-truth masks and query thresholds are obtained, and which input priors are available to each baseline.
  5. What stages are included in the reported 1.74× training speedup over GenMOJO? Please provide absolute runtimes and clarify whether preprocessing, object generation, alignment, multi-view appearance generation, and semantic optimization are included.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.