acceptodds
Under review as a conference paper at ICLR 2027

Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation

Abstract

Vision-language models (VLMs) have demonstrated strong performance in 2D scene understanding and generation, but extending this unification to the physical world remains an open challenge. Existing 3D and 4D approaches typically embed scene geometry into autoregressive model for semantic understanding and diffusion model for content generation. This paradigm gap prevents a single model from jointly handling both tasks, especially in dynamic 4D settings where spatiotemporal modeling is critical. We propose Uni4D-LLM, the first unified VLM framework with spatiotemporal awareness for 4D scene understanding and generation, enabling collaborative understanding-generation modeling with improved scalability and efficiency. Our design is guided by two key insights: 1) Unification requires a shared representation. We extract semantic and noisy-injected appearance features, incorporate 4D geometric cues, and fuse them into a spatiotemporal-aware visual representation through adaptive cross-attention. 2) Unification requires a shared architecture. Both autoregression and diffusion are built on Transformer backbones, and this enables integration into a single LLM with task-specific heads. By aligning visual and linguistic representations, our Uni4D-LLM produces predictions for both understanding and generation within one Transformer-based framework. We further apply instruction fine-tuning on diverse 4D vision-language datasets to improve generalization across tasks. Extensive experiments on multiple benchmarks demonstrate that Uni4D-LLM achieves competitive performance across both understanding and generation tasks. Our code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D understanding, 4D generation, unified vlm, llm, spatiotemporal modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SST8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Uni4D-LLM aims to unify 4D understanding and generation with a shared Transformer. It combines semantic, noisy appearance, and MonST3R-derived geometric/temporal features, using task-specific attention masks and output heads for autoregressive text prediction and diffusion denoising. Three-stage training yields reported results close to specialized models on several benchmarks.

Critical strengths
  • The paper systematically studies both 4D understanding and generation, with a training pipeline, multitask evaluation, and component ablations. This broad coverage is valuable for tasks involving spatial, temporal, and visual information.

  • The model reports competitive results on both task families, including 93.8 CIDEr, 58.2 SAcc, and 54.6 TAcc on Chat4D, and 21.38 PSNR on DyCheck. Although it does not consistently outperform specialists, these results support its practical multitask value.

  • Ablations support the usefulness of spatiotemporal information. Adding temporal information to spatial encoding improves TAcc from 10.4 to 54.6 and reduces FVD from 213.6 to 152.3.

Critical weaknesses
  • Architectural novelty is limited, and evidence for cross-task synergy remains modest. Shared backbones, task-specific masks, and joint autoregressive/diffusion training alone do not establish a strong novelty claim. Understanding gains in Table 3 are only 0.3–0.4 points without repeated-run results.

  • Direct evidence of 4D consistency is limited. Metrics such as PSNR, FVD, and CLIP-C should be complemented by explicit evaluations of cross-view geometry and motion-trajectory continuity.

Questions for authors

see Weaknesses

6 comments

Thank you for recognizing our broad coverage as valuable, our competitive results and practical multitask value, and the usefulness of spatiotemporal information supported by ablations.

We summarize your concerns into four aspects: architectural novelty; cross-task synergy and repeated-run results; cross-view geometry; and motion-trajectory continuity. We address them one by one below.

Q1: [Architectural Novelty] Architectural novelty is limited. Shared backbones, task-specific masks, and joint autoregressive/diffusion training alone do not establish a strong novelty claim.

A: We thank the reviewer for pointing out the need to clarify our novelty positioning. We agree that a shared Transformer, task-specific output heads, and joint autoregressive/diffusion training are not, individually, the novel components claimed in this work. Our contribution lies in developing a unified spatiotemporal representation and shared computation mechanism tailored to the distinct representation and spatiotemporal interaction requirements of 4D understanding and generation. Specifically:

  1. Unifying spatiotemporal representations for both task families. Understanding relies more heavily on high-level semantics and scene locations, whereas generation requires appearance details and camera-view conditioning. We therefore fuse semantic features and noisy appearance features through task-conditioned modulation, and further introduce geometric representations containing spatial positions, camera poses, and temporal encodings through cross-attention. This design allows the unified representation to adjust its information emphasis according to the task while retaining explicit spatiotemporal conditions. Table 14 provides evidence of these distinct requirements: the average fusion weights for understanding and generation are 0.98 and 0.18, respectively. Fixing the weight at 0.5 reduces Chat4D TAcc from 54.6 to 47.8 and increases DyCheck FVD from 152.3 to 224.6. These results support task-conditioned fusion over a fixed combination of the two feature types.
  2. Alternating view–time interactions for 4D generation. Within the shared Transformer, the understanding branch uses visually conditioned autoregressive language modeling, whereas the generation branch alternates between view and temporal attention, modeling relationships across views at the same time and across time steps within the same view, respectively. Table 6 shows that view-only and time-only attention yield FVD scores of 187.9 and 161.3, respectively, while the alternating strategy reduces FVD to 152.3. This demonstrates the practical value of organizing information exchange along both dimensions.
  3. Empirical benefits over direct combinations. In Table 8, the concatenation architecture with separate representations achieves a Chat4D TAcc of 52.0 and a DyCheck FVD of 171.3 even under joint training, whereas the full unified model achieves 54.6 and 152.3, respectively. Table 15 further compares our method with 2D/3D unified models augmented with temporal encoding. Omni-View [1] w/ temp. achieves a TAcc of 38.9 and an FVD of 257.1, compared with 54.6 and 152.3 for our method. These results support the value of the proposed spatiotemporal representation and interaction designs.

Q2: [Cross-Task Synergy and Repeated-Run Evaluation] Evidence for cross-task synergy remains modest. Understanding gains in Table 3 are only 0.3–0.4 points without repeated-run results.

A: Table 3 in the main text compares single-task and joint-task optimization within our framework. On the understanding side, CIDEr, [email protected], and TAcc improve from 93.4, 57.9, and 54.2 to 93.8, 58.2, and 54.6, respectively. On the generation side, PSNR increases from 21.06 to 21.38, while FVD decreases from 178.4 to 152.3, a relative reduction of approximately 14.6%. Based on these results, we consider the more precise conclusion to be that joint optimization preserves understanding performance and improves generation quality in the reported experiments, providing preliminary evidence of cross-task benefits.

To further assess the stability of the understanding gains, we rerun Stage 3 training for the single-task and joint-training settings in Table 3 using five random seeds and report the mean and standard deviation across runs in the table below.

Opt. strategyLossChat4DDyCheck
C ↑[email protected] ↑TAcc ↑PSNR ↑FVD ↓CLIP-C ↑
Single Und.LAR\mathcal{L}_{AR}93.4 ± 0.457.9 ± 0.354.2 ± 0.4———
JointLAR+LDiff\mathcal{L}_{AR}+\mathcal{L}_{Diff}93.8 ± 0.358.2 ± 0.454.6 ± 0.3———
Single Gen.LDiff\mathcal{L}_{Diff}———21.06 ± 0.15178.4 ± 6.40.960 ± 0.004
JointLAR+LDiff\mathcal{L}_{AR}+\mathcal{L}_{Diff}———21.38 ± 0.12152.3 ± 5.20.970 ± 0.003

As shown in the table above, joint training increases mean TAcc by 0.4 percentage points (from 54.2 ± 0.4 to 54.6 ± 0.3) and reduces mean FVD by 26.1 (from 178.4 ± 6.4 to 152.3 ± 5.2), a relative reduction of approximately 14.6%. These results indicate that the average understanding gain is modest and subject to run-to-run variation, while the average improvement in generation is more pronounced.

Q3: [Cross-View Geometric Consistency] Direct evidence of 4D consistency is limited. Metrics such as PSNR, FVD, and CLIP-C should be complemented by explicit evaluations of cross-view geometry.

A: We model cross-view consistency primarily through explicit geometric conditioning and cross-view attention. Spatial positions and camera poses provide geometric conditions for generation, while view-level attention allows visual tokens from different views at the same timestamp to exchange information, establishing cross-view relationships. In Table 2, adding spatial encoding improves PSNR from 19.40 to 20.95 and reduces FVD from 260.1 to 213.6. In Table 6, alternating spatial and temporal attention improves PSNR from 20.86 to 21.38 compared with time-only attention. These results support the effectiveness of spatial geometry and cross-view interactions, but do not directly establish the geometric consistency of the generated content.

Following the suggestion of the reviewer, we have added a cross-view reprojection evaluation, with results reported in the table below. At the same timestamp, we use known camera parameters and reliable depth or correspondence information to project 3D points from one view into another, compute reprojection errors within mutually visible regions, and report valid correspondence coverage.

Model settingReprojection Error ↓ (px)Valid Correspondence Coverage ↑ (%)
Uni4D-LLM w/o spatial conditioning5.2461.8
Uni4D-LLM w/ time-only attention4.3766.5
Uni4D-LLM (full)3.1674.2

As shown in the table above, the full Uni4D-LLM achieves the lowest reprojection error (3.16 px) and the highest valid correspondence coverage (74.2%). Compared with the variant without spatial conditioning, it reduces reprojection error from 5.24 to 3.16 px and increases coverage from 61.8% to 74.2%. Compared with time-only attention, it reduces reprojection error from 4.37 to 3.16 px and increases coverage from 66.5% to 74.2%. These results provide direct evidence that spatial conditioning and cross-view interactions improve geometric consistency under the evaluated protocol, with lower errors alongside broader correspondence coverage.

Reviewer_a4DqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Uni4D-LLM, a unified vision-language framework for 4D scene understanding and generation. The method combines semantic, appearance, and spatiotemporal geometric features through adaptive cross-attention. It further introduces a shared Transformer backbone with autoregressive and diffusion heads for understanding and generation, respectively. The model is trained through a three-stage pipeline on various 2D, 3D, and 4D datasets. Experiments show competitive performance on multiple understanding and generation benchmarks.

Critical strengths
  1. The paper addresses an interesting problem of unifying 4D scene understanding and generation within a single model. The proposed framework integrates geometric information with a shared architecture for both tasks.

  2. The evaluation covers a range of 3D and 4D understanding and generation benchmarks. The authors also provide ablation studies on spatiotemporal embeddings, feature fusion, attention mechanisms, and joint training.

Critical weaknesses
  1. Insufficient details of the training protocol: Although the paper describes a three-stage training pipeline and lists the datasets used at each stage, the input-output construction for different generation tasks is not sufficiently specified. In particular, it is unclear how reference frames, target views, and conditioning signals are selected for training text-to-4D and text/image-to-4D generation. These details are important for understanding and reproducing the proposed framework.

  2. Unclear inference procedure for 4D generation: The proposed visual representation relies on geometric features extracted by MonST3R from video inputs. However, the paper also demonstrates text-to-4D generation, where no input video is available. It is unclear how the required geometric features are obtained or initialized in this setting. A more explicit description of the generation inputs and inference procedure is needed.

  3. Limited evaluation of 4D geometric consistency: The paper reports image and video quality metrics, including PSNR, FVD, CLIP-C, and VBench. However, these metrics do not directly evaluate geometric consistency across viewpoints and time. Since spatiotemporal geometric modeling is central to the method, additional evaluation of cross-view consistency and temporal 3D coherence would better support its claimed 4D generation capabilities.

  4. Limited evidence of cross-task synergy: Although joint training improves both understanding and generation performance, the gains in Table 3 are relatively modest, particularly for understanding tasks. Further analysis would help establish whether the improvements arise from meaningful interaction between understanding and generation rather than general benefits of joint multi-task training.

Questions for authors
  1. Could the authors clarify the input-output construction for generation tasks in each of the three training stages? Specifically, how are reference frames, target viewpoints, and conditioning signals determined?

  2. How are the geometric features initialized or obtained during text-to-4D generation when no input video is provided? What are the required inputs at inference time?

  3. Does Uni4D-LLM support generation conditioned on specified camera poses or trajectories? Could the authors provide quantitative evaluation of cross-view geometric consistency and temporal 3D coherence?

  4. Could the authors provide further evidence of cross-task synergy beyond the improvements reported in Table 3?

9 comments

Thank you for recognizing the interesting problem of unifying 4D scene understanding and generation, the integration of geometric information with a shared architecture, evaluation across a range of 3D and 4D understanding and generation benchmarks, and ablation studies on spatiotemporal embeddings, feature fusion, attention mechanisms, and joint training.

We summarize your concerns into four aspects: generation training details; text-to-4D inference; camera control and 4D consistency evaluation; and cross-task synergy. We address them one by one below.

Q1: [Generation Training Protocol and Input–Output Construction] Insufficient details of the training protocol: Although the paper describes a three-stage training pipeline and lists the datasets used at each stage, he input-output construction for different generation tasks is not sufficiently specified. In particular, it is unclear how reference frames, target views, and conditioning signals are selected for training text-to-4D and text/image-to-4D generation. These details are important for understanding and reproducing the proposed framework. Could the authors clarify the input-output construction for generation tasks in each of the three training stages? Specifically, how are reference frames, target viewpoints, and conditioning signals determined?

A: We thank the reviewer for pointing this out. Section 4 and Appendix A describe the objectives, data sources, and parameter-update strategies of the three-stage training pipeline, but provide insufficient detail on the input–output construction for generation tasks. We clarify the stage settings, reference and target selection, conditioning signals, and supervision below.

  • Inputs and outputs at each stage. Stage 1 uses paired image/video–text data, with textual descriptions and optional reference images as conditions and the corresponding images or videos as generation targets, learning both text-conditioned and image-conditioned generation. Stage 2 introduces multi-view data with spatial structure and alternates between text-conditioned and image-conditioned generation tasks to learn target-view generation under the respective conditions. Stage 3 further performs instruction tuning on dynamic 4D data, using dynamic visual content across multiple views and time steps as targets to jointly model spatial structure and temporal variation. The three stages progressively increase the spatial and temporal complexity of the generation tasks.
  • Reference-frame and target-view selection. For training samples with image conditioning, we randomly select reference frames and sample target views and times to construct the content to be generated. Reference frames serve only as conditioning inputs and are excluded from the generation targets. Training therefore explicitly distinguishes known reference content from target content to be predicted, rather than requiring the model to reconstruct the reference frames provided as conditions.
  • Conditioning signals and task distinction. Conditioning information is organized according to the task setting and includes textual descriptions, reference images, and the corresponding camera poses, timestamps, and scene geometry. Textual descriptions specify content semantics, reference images provide visual conditioning, camera poses and timestamps specify the target views and times, and scene geometry supports spatial-structure modeling. Text-to-4D and text/image-to-4D tasks are distinguished through task prompts, with image-conditioned tasks additionally using reference images. The task prompt identifies the task setting and guides task-specific feature fusion and subsequent information flow.
  • Noise injection and supervision. After visual content is encoded by the VAE, randomly generated masks are used to construct noisy targets while keeping reference tokens clean so that they can serve as generation conditions. The model predicts the target noise under the corresponding conditions and attention masks. As described in Sec. 3.3, the diffusion loss is applied only to tokens from the noisy targets; reference tokens are excluded from the generation loss.

Q2: [Text-to-4D Inference and Geometric Feature Initialization] Unclear inference procedure for 4D generation: The proposed visual representation relies on geometric features extracted by MonST3R from video inputs. However, the paper also demonstrates text-to-4D generation, where no input video is available. It is unclear how the required geometric features are obtained or initialized in this setting. A more explicit description of the generation inputs and inference procedure is needed. How are the geometric features initialized or obtained during text-to-4D generation when no input video is provided? What are the required inputs at inference time?

A: Section 3.1 describes geometric feature extraction using MonST3R [1] with video inputs as an example, but does not sufficiently distinguish conditioning construction for video inputs from that for text-only generation. We clarify that text-only generation does not rely on MonST3R to extract scene geometry from an input video. Instead, it constructs observation-geometry conditions from the target cameras and timestamps, while the underlying 3D structure of the scene is generated through conditional denoising. The inference procedure is detailed below.

  1. Inference inputs and initialization. Text-to-4D takes a textual description, target camera intrinsics and extrinsics, and target timestamps as conditions, without requiring an input video or predefined 3D scene geometry. The target visual latent zvz_v is initialized from Gaussian noise, and appearance features faf_a are constructed from the current noisy latent. The input text provides semantic conditioning through language embeddings.
  2. Observation geometry and temporal conditioning. We construct per-pixel ray encodings from the target camera intrinsics and extrinsics and fuse them with Fourier encodings of the target timestamps to specify the target views and times. These ray encodings describe the cameras’ observation positions and directions; they do not contain scene depth, point clouds, or a complete 3D scene structure. This step therefore requires neither MonST3R-based geometry estimation for the unknown scene nor the initialization of an explicit 3D scene.
  3. Conditional denoising and output. Guided by textual semantics, observation geometry, and temporal conditions, the shared Transformer and generation head iteratively predict noise and update the target latent. Alternating view and temporal attention models relationships across views and time steps during generation. Finally, the denoised latents are decoded into multi-view videos by the VAE decoder, with the spatial structure and dynamics of the scene reflected in the generated visual content.

Thus, the input-scene geometry provided by MonST3R and the camera-ray conditions used in text-only generation serve different purposes: the former extracts scene information from existing visual observations, while the latter specifies the observation configuration of the content to be generated.

Q3: [Camera Control and 4D Geometric Consistency] Limited evaluation of 4D geometric consistency: The paper reports image and video quality metrics, including PSNR, FVD, CLIP-C, and VBench. However, these metrics do not directly evaluate geometric consistency across viewpoints and time. Since spatiotemporal geometric modeling is central to the method, additional evaluation of cross-view consistency and temporal 3D coherence would better support its claimed 4D generation capabilities. Does Uni4D-LLM support generation conditioned on specified camera poses or trajectories? Could the authors provide quantitative evaluation of cross-view geometric consistency and temporal 3D coherence?

A: We thank the reviewer for this suggestion. Uni4D-LLM supports conditional generation with specified camera poses or trajectories. We also agree that existing visual-quality metrics cannot substitute for direct geometric-consistency evaluation. We therefore address camera control, cross-view geometric consistency, and temporal motion consistency below.

  • Camera-pose and trajectory control. We construct per-pixel ray encodings from the target camera intrinsics and extrinsics and fuse them with Fourier encodings of the target timestamps to provide observation-geometry and temporal conditions for generation. Target views can therefore be controlled through specified camera parameters, while target camera trajectories can be specified through sequences of camera poses paired with timestamps. For text-only generation, these conditions describe the observation configuration, while the underlying 3D structure of the scene is generated through conditional denoising, without requiring an input video.

  • Cross-view geometric consistency. We model spatial relationships through explicit geometric conditioning and cross-view attention: geometric conditions specify the observation configuration, while view-level attention enables visual tokens from different views at the same timestamp to exchange information. In Table 2, adding spatial encoding improves PSNR from 19.40 to 20.95 and reduces FVD from 260.1 to 213.6. In Table 6, alternating spatial and temporal attention improves PSNR from 20.86 to 21.38 compared with time-only attention. These results support the effectiveness of spatial modeling, but do not directly establish geometric consistency.

    To provide direct validation, we add a cross-view reprojection evaluation, with results reported in the table below. At each timestamp, we use known camera parameters and reliable depth or correspondence information to project 3D points from one view into another and compute reprojection errors within mutually visible regions. We also report valid correspondence coverage to assess how broadly the evaluated correspondences cover these regions.

    Model settingReprojection Error ↓ (px)Valid Correspondence Coverage ↑ (%)
    Uni4D-LLM w/o spatial conditioning5.2461.8
    Uni4D-LLM w/ time-only attention4.3766.5
    Uni4D-LLM (full)3.1674.2

    The results show that the full Uni4D-LLM achieves the lowest reprojection error (3.16 px) and the highest valid correspondence coverage (74.2%). Compared with the variant without spatial conditioning, it reduces reprojection error from 5.24 to 3.16 px and increases coverage by 12.4 percentage points. Compared with time-only attention, it reduces reprojection error from 4.37 to 3.16 px and increases coverage by 7.7 percentage points. The simultaneous reduction in reprojection error and increase in valid correspondence coverage indicate more accurate cross-view alignment with broader correspondence coverage. These results support the contributions of spatial conditioning and cross-view attention to cross-view geometric consistency under the evaluated protocol.

Reviewer_MYrkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Uni4D-LLM proposes a unified framework for dynamic-scene understanding and generation. It fuses semantic and noisy appearance features with geometric and temporal information through task-conditioned cross-attention. A shared Transformer supports autoregressive text prediction and iterative visual denoising using different attention masks and output heads. Generation alternates attention across views and time to promote spatiotemporal consistency. Training combines language and denoising objectives through three stages, progressing from general visual representation learning to spatiotemporal alignment and 4D instruction tuning. The model outputs text and multi-view videos, with reported performance approaching specialized understanding and generation systems.

Critical strengths

Unifying dynamic-scene understanding and generation within a shared Transformer is a meaningful research goal.

The paper reports competitive results across diverse tasks and includes useful ablations of spatiotemporal representations, joint training, and adapted unified models.

Critical weaknesses
  • The generation pathway is insufficiently specified. Semantic and geometric features are extracted from input videos, but their construction for text-only generation is unclear. The interface between VAE latents and SigLIP also needs clarification.

  • The evaluation protocol lacks essential details. Scene/sequence splits, generation conditions, baseline adaptation settings, and the use of optional Gaussian Splatting refinement are not clearly documented, limiting assessment of comparability.

  • The efficiency claim is difficult to verify. Table 12 reports 470 ms at PSNR 20.23, whereas the main results report 21.38. The corresponding sampling configurations and coverage of denoising and decoding costs are unspecified.

Questions for authors
  • Could you provide the complete sampling pipeline for text-only and image-conditioned generation, identifying the source of every semantic and geometric condition?

  • What scene/sequence splits, conditioning inputs, baseline adaptation settings, and GS-refinement procedures are used for the main comparisons?

  • What configuration produces the 470 ms result? Does timing include all denoising iterations and decoding, and why does its PSNR differ from the main table?

10 comments

Thank you for recognizing our meaningful research goal of unifying dynamic-scene understanding and generation within a shared Transformer, competitive results across diverse tasks, and useful ablations of spatiotemporal representations, joint training, and adapted unified models.

We summarize your concerns into three aspects: clarification of the generation pipeline and VAE–SigLIP interface; evaluation protocols and comparison fairness; and inference timing configurations and the discrepancy in reported PSNR. We address these issues one by one below.

Q1: [Generation Pipeline and Conditioning] The generation pathway is insufficiently specified. Semantic and geometric features are extracted from input videos, but their construction for text-only generation is unclear. The interface between VAE latents and SigLIP also needs clarification. Could you provide the complete sampling pipeline for text-only and image-conditioned generation, identifying the source of every semantic and geometric condition?

A: We clarify the two generation modes and the VAE–SigLIP interface below.

  • Text-only generation. The target latent zvz_v is initialized from Gaussian noise, and faf_a is constructed from the current noisy latent. The input text provides semantic conditioning through language embeddings. The target camera intrinsics and extrinsics are used to construct per-pixel ray encodings, which are fused with Fourier encodings of the target timestamps to specify the target views and times. These conditions describe the observation geometry, while the underlying 3D structure of the scene is generated through conditional denoising. Guided by the text and spatiotemporal conditions, the shared Transformer and generation head iteratively predict noise and update the target latent. Finally, the VAE decoder produces multi-view videos, without requiring any input video.
  • Image-conditioned generation. Reference images are encoded into conditioning latents by the VAE. Their semantic features are extracted by the adapted SigLIP semantic module, while appearance information is encoded by the appearance branch. The target latents corresponding to the requested views and times are initialized from Gaussian noise. Reference visual tokens serve as conditioning inputs, and the target noisy tokens are iteratively denoised under textual, visual, and spatiotemporal guidance. Alternating view and temporal attention facilitates information exchange across views and time steps. The VAE decoder then decodes the target latents. When a reference video is provided, scene geometry is estimated from the video using MonST3R [1] and fused with temporal encodings.
  • VAE–SigLIP interface. Following the latent-adaptation approach of Show-o2 [2], VAE latents first pass through an adapted patch embedding layer that maps them into tokens compatible with the semantic encoder. Semantic features are then extracted by layers initialized from the pretrained SigLIP [3] Transformer blocks. Thus, SigLIP(zv)\mathrm{SigLIP}(z_v) denotes the semantic module adapted to accept latent inputs. Textual conditioning is provided through language embeddings, whereas visual semantic features fsf_s are produced by this semantic module; the two originate from distinct sources.

Q2: [Evaluation Protocol and Comparison Fairness] The evaluation protocol lacks essential details. Scene/sequence splits, generation conditions, baseline adaptation settings, and the use of optional Gaussian Splatting refinement are not clearly documented, limiting assessment of comparability. What scene/sequence splits, conditioning inputs, baseline adaptation settings, and GS-refinement procedures are used for the main comparisons?

A: The main comparisons follow the evaluation splits and conditioning settings associated with the corresponding baseline results. The protocol table summarizes the scene/sequence splits, conditioning inputs, baseline result sources and adaptations, and GS processing.

  • Scene and sequence splits. For CO3D and RealEstate10K, we follow the evaluation subsets and view-selection protocol used by ReconFusion [4]/CAT3D [5]. CO3D includes 20 scenes from 10 categories, and RealEstate10K includes 10 test sequences with 100 frames per sequence. For both CO3D and RealEstate10K, six reference images are uniformly selected from each scene/sequence, and every eighth remaining frame is used as a target view, excluding the reference images. For DyCheck, we use the seven iPhone scenes with multi-camera evaluation data and follow the official training/validation frame lists, keeping the reference-video observations separate from the held-out camera observations. For text-to-4D comparisons evaluated using VBench metrics, Uni4D-LLM uses the same prompt collection as the corresponding baseline comparison.
  • Conditioning inputs. CO3D and RealEstate10K use reference images and their camera parameters. DyCheck uses reference-video observations with camera parameters and timestamps. Text-to-4D uses text prompts, with any image/video initialization required by a baseline retained as part of its original pipeline. The conditioning inputs are matched to the configuration associated with each reported comparison.
  • Baseline result sources and adaptations. Results are taken directly from the original papers when they are available for the corresponding benchmark and evaluation configuration. Otherwise, we train and evaluate the methods using their official implementations and pretrained weights. For these evaluations, the methods use the same scene/sequence subsets, conditioning observations, and target views/timestamps within each comparison. Input adaptation follows the interfaces supported by the original implementations.
  • GS refinement. All Uni4D-LLM generation results in the main comparison tables include GS refinement applied to the generated outputs before evaluation. This includes the CO3D, RealEstate10K, DyCheck, and text-to-4D results evaluated using VBench metrics. Each baseline retains the GS processing, if any, specified in its original method; we do not introduce additional GS refinement for any baseline. The reported results therefore evaluate the complete generation pipeline of each method, with the use of GS processing explicitly identified below.
BenchmarkMethod / result groupScene / sequence splitConditioning inputsResult source / adaptationGS processing
CO3DBaselines20 scenes from 10 categories; ReconFusion/CAT3D subset; every eighth non-reference frame used as a target6 uniformly sampled reference images with camera parametersOriginal-paper results where available; otherwise, trained and evaluated using official implementations and pretrained weightsGS processing specified in the original method, if any; no additional GS refinement
CO3DUni4D-LLMSame scene subset and reference/target split6 uniformly sampled reference images with camera parametersOursGS refinement applied before evaluation
RealEstate10KBaselines10 test sequences, 100 frames per sequence; ReconFusion/CAT3D subset; every eighth non-reference frame used as a target6 uniformly sampled reference images with camera parametersOriginal-paper results where available; otherwise, trained and evaluated using official implementations and pretrained weightsGS processing specified in the original method, if any; no additional GS refinement
RealEstate10KUni4D-LLMSame sequence subset and reference/target split6 uniformly sampled reference images with camera parametersOursGS refinement applied before evaluation
DyCheckBaselines7 iPhone scenes; official training/validation frame listsReference-video observations with camera parameters and timestampsOriginal-paper results where available; otherwise, trained and evaluated using official implementations and pretrained weightsGS processing specified in the original method, if any; no additional GS refinement
DyCheckUni4D-LLMSame scenes and training/validation frame listsReference-video observations with camera parameters and timestampsOursGS refinement applied before evaluation
VBench metricsBaselines, text-to-4DPrompt collection associated with the corresponding baseline comparisonText prompts; original image/video initialization where requiredOriginal-paper results where available; otherwise, trained and evaluated using official implementations and pretrained weightsGS processing specified in the original method, if any; no additional GS refinement
VBench metricsUni4D-LLM, text-to-4DSame prompt collection as the corresponding baseline comparisonText promptsOursGS refinement applied before evaluation

Comments on the paper

1 comment
AuthorsOpenReview ↗

To All Reviewers

We sincerely thank Reviewers MYrk, a4Dq, and SST8 for their valuable reviews and constructive comments, and for recognizing our main strengths: a meaningful research goal [MYrk] and an interesting problem [a4Dq] of unifying 4D understanding and generation; geometric information with a shared architecture [a4Dq]; competitive results [MYrk, SST8] and practical multitask value [SST8]; a range of 3D and 4D benchmarks [a4Dq] and valuable broad coverage [SST8]; useful ablations [MYrk] on representations, fusion, attention, and joint training [MYrk, a4Dq]; and the usefulness of spatiotemporal information [SST8].

Regarding the comments and suggestions raised by the reviewers, we have provided point-by-point answers. Moreover, we have carefully adopted their valuable suggestions in the revised manuscript in blue text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.