acceptodds
Desk rejected at ICLR 2027

ST-Gen4D: Embedding 4D Spatiotemporal Cognition into World Model for 4D Generation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative models have achieved success in producing apparently coherent 2D videos, but remain challenging in the physical world due to lack of 4D spatiotemporal scale. Typically, existing 4D generative models directly embed macro scale constraints to enhance overall spatiotemporal consistency. However, these methods only ensure global appearance coherence and fail to reveal the local dynamics of the physical world. Our insight is that global appearance structure and local dynamic topology empower 4D spatiotemporal cognition, thereby enabling 4D generation with spatiotemporal regularities. In this work, we propose ST-Gen4D, a 4D generation framework with 4D spatiotemporal cognition-based world model. Our model is guided by four key designs: 1) Spatiotemporal representation. We encode various modalities into multiple representations as a feature basis. 2) Spatiotemporal cognition. We sculpture these representations into global appearance graph and local dynamic graph, and fuse them via semantic-bridged spatiotemporal fusion to obtain a 4D cognition graph. 3) Spatiotemporal reasoning. We utilize a world model to derive future state based on the 4D cognition. 4) Spatiotemporal generation. We leverage the derived cognition as condition to guide latent diffusion for 4D Gaussian generation. By deeply integrating 4D intrinsic cognition with generative priors, our model guarantees the structural rationality and topological consistency of 4D generation. Moreover, we propose ST-4D datasets by aggregating public 4D datasets and self-built subset. Extensive experiments demonstrate the superiority of our ST-Gen4D across 3D and 4D generation tasks. Our code will be made publicly available once the paper is accepted.

Primary area:
generative models
Keywords:
4D scene generation, world model, diffusion model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VEYyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ST-Gen4D, a framework for 4D generation based on structured spatiotemporal cognition. The method extracts multiple representations, including semantic, spatial, temporal, logical, and action tokens. These representations are organized into semantic, global-appearance, and local-dynamics graphs, which are fused into a spatiotemporal cognition graph.

An action-conditioned autoregressive world model predicts future cognition states, and these predicted states are subsequently used to condition a latent diffusion model for generating 4D Gaussian representations. The authors additionally construct ST-4D by aggregating multiple public 3D/4D datasets together with self-built data.

Experiments report strong performance on several text/image-conditioned 3D and 4D generation benchmarks.

Critical strengths
  1. Key Issues and Intuitive Representation Design. In 4D generation tasks, explicitly distinguishing between global appearance structure and local dynamic information is well-justified, as both cross-view structural consistency and temporal coherence are crucial.

  2. Comprehensive Component-Level Evaluation. We conducted ablation studies on semantic/spatial/temporal tokens, graph construction, global and local graph branches, the semantic backbone network, and fusion strategies. The results strongly demonstrate the effectiveness of the structured representation for the final generation task.

Critical weaknesses
  1. The paper fails to sufficiently distinguish and articulate its innovations relative to existing "cognition-guided generation" and "4D world modeling" methods. The overall pipeline follows a common paradigm: constructing structured semantic/geometric or spatiotemporal representations and then using them to guide the generation process. Existing cognition-guided generation methods already utilize structured cognitive representations to steer Gaussian generation, while recent 4D world models are capable of predicting future dynamic scene representations. The paper's primary novelty appears to lie in combining global appearance, local dynamics, and autoregressive future cognitive prediction; however, the distinction between its unique contributions and the aforementioned existing paradigms remains inadequately defined.

  2. The core world model inference component lacks direct validation. A key claim of the paper is that future cognitive states predicted by the world model improve 4D generation outcomes. However, a crucial ablation study appears to be missing: a comparison between the "current cognition → generator" path and the "current cognition → world model → future cognition → generator" path. Furthermore, the paper provides no direct quantitative assessment regarding the accuracy of future cognitive predictions, graph node evolution, relationship/topology prediction, or the discrepancy between predicted cognition and cognition extracted from the ground-truth future. Consequently, while the experimental results demonstrate the effectiveness of structured graph features, they fail to conclusively prove that world-model-based inference using predictions is essential.

  3. There is a mismatch between the paper's core claims and its primary evaluation metrics. The paper emphasizes structural plausibility, topological consistency, physical plausibility, and coherent 4D dynamics; yet, the primary 4D evaluations rely heavily on VBench and user study metrics (such as consistency, motion quality, aesthetics, and text alignment). While these metrics help assess the quality of rendered videos, they cannot directly measure 3D/4D geometric accuracy, multi-view geometric consistency, object-level trajectory accuracy, topological preservation, or physical correctness. 4. Its architectural contributions are inextricably intertwined with a massive and heterogeneous training pipeline. ** ST-Gen4D integrates a variety of 3D/4D datasets, proprietary and synthetic data, generated text prompts, powerful pre-trained encoders (such as Wan2.1 and 4D-VGGT), large-scale video pre-training, SDS supervision, and subsequent 3D/4D refinement steps. Since comparative methods were not retrained under identical data and pre-training conditions, it is difficult to determine whether the reported performance gains stem from the proposed cognitive/world model architecture or from more extensive training data and stronger pre-trained priors.

  4. The evaluation of long-horizon world model performance is insufficient. The paper acknowledges that autoregressive cognitive prediction can accumulate early topological errors, leading to structural degradation and semantic drift over long sequences. However, the study lacks a systematic evaluation across varying rollout horizons. Given that a primary motivation for introducing world models is to enable long-term state evolution, it is crucial to examine model performance across different prediction horizons.

Questions for authors
  1. How does the performance compare if the world model is removed and the current cognition graph is used directly to condition the 4D generator?

  2. Can the authors provide a quantitative evaluation comparing the predicted future cognition with the cognition extracted from ground-truth future data (e.g., by assessing the prediction accuracy of graph nodes or relationships)?

  3. Beyond VBench and user studies, can the authors provide evaluations regarding geometric or topological properties?

  4. To what extent is the reported performance gain maintained when using matched training data and a pre-trained backbone network?

  5. How do generation quality and structural consistency degrade as the rollout time span of the autoregressive world model increases?

10 comments

Q1 — Weakness 1

Q. What distinguishes ST-Gen4D from existing cognition-guided generation methods and 4D world models?

A. Section 3.2 combines geometry-informed appearance/dynamics graphs, logical relations, and semantic-query fusion. Action-conditioned prediction supplies future cognition; state decoding uses current-graph queries and predicted-state keys/values. The contribution is this organization.

Original Tables 6/8 support graphs/fusion; Table 1 (“Fusion controls across modalities and budgets”) compares fusion with ordinary attention, and Table 2 (“Generation controls across modalities and budgets”) compares predictive conditioning. The fusion comparison matches condition dimensions, attention depth/heads, and interfaces; ordinary attention retains a semantic residual. Sem-STF reuses C's generation results in Table 2. Q2 defines metrics/budgets.

The distinguishing design is the interface linking semantic-query fusion of global-appearance/local-dynamics graphs, action-conditioned prediction of future cognition, and CG-DiT conditioning. Comparing Table 1 with Table 2, text/EU Sem-STF raises SC/MS over ordinary attention from 0.9458/0.9637 to 0.9737/0.9678, quantifying the fusion benefit within the proposed generation pipeline.

Q2 — Weakness 2

Q. What direct evidence shows that predicted future cognition improves generation and accurately captures future nodes and relations?

A. For the A/B/S/C controls in Table 2 (“Generation controls across modalities and budgets”), A uses feature conditioning; B reuses the initial graph without prediction. S/C share predictor architecture and supervision; Table 3 reports 187.92 M parameters for each. They supply initial/predicted cognition, respectively. Controls adapt separately with shared data, corresponding initialization/checkpoints, and the input-encoding pipeline. Evaluation uses 100 public held-out scenes with source-grouped splits and three training × three generation seeds. EU fixes 20,000 updates; EC caps adaptation at 160 GPU-hours. For caption or single-image inputs, temporal/action representations are predicted from latent input cues using learned priors.

In Table 2, SC/BC/MS/DD/AQ/IQ denote subject/background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality. T-/I-CLIP is 100×mean normalized ViT-B/32 cosine similarity over 16 frames against prompt/initial image. Generation cells report mean ± sample standard deviation (SD) of three training-seed summaries; generation seeds are first averaged within scene/seed. Brackets in Table 4 denote 95% scene-bootstrap confidence intervals (CIs) for paired mean differences. X−Y means metric(X)−metric(Y); paired CIs resample unrounded scene differences averaged across seeds.

In Table 4, Text/EU C−S improves SC/MS by 0.0115/0.0220; C−S and C−B have positive paired 95% CIs for both metrics across both modalities and budgets. C−S demonstrates the benefit of evolving conditioning at fixed predictor capacity; C−B tests the complete predictive path. Separately, Table 5 and Table 6 assess Q5/Q7's observed-prefix forecasts against references encoded from actual future frames; diagnostic cells retain means ± SD across the same three training-seed summaries and scene-bootstrap CIs. At 16 seconds, Table 5 shows prediction versus persistence lowers state MSE from 0.2453 to 0.2194; Table 6 shows node feature error decreases from 0.3199 to 0.2681, while feature-affinity F1 increases from 0.6871 to 0.7063. The controls establish a generation benefit, while the diagnostics quantify prediction accuracy at state, node-feature, and feature-affinity relation levels.

Q3 — Weakness 3

Q. What evidence supports the claimed geometric, topological, and physical properties beyond rendered-video quality?

A. Geometry enters through embeddings, Gaussian deformations, and geometric/rendering losses. Original Table 2 reports static CD/F-Score/IoU-B=0.051/62.15/0.713; Original Table 9 uses IoU (intersection over union). IoU-B retains its distinct original label.

The EU geometric diagnostic in Table 7 uses 50 qualified Q5 scenes with shared observed prefixes/controls. Normalize the initial reference bounding-box diagonal to one and align once initially. Visible surfaces share extraction/sampling rules. Chamfer distance (CD) averages bidirectional unsquared nearest-neighbor distances; supplementary F-Score is the harmonic mean of surface precision/recall at distance 0.01, on a 0–1 scale. Average/final displacement error (ADE/FDE) measures corresponding trajectory errors over valid times/at 3.875 seconds, requiring persistent correspondence/common visibility. Contact precision (P), recall (R), and harmonic-mean F1 compare predicted/reference object-pair proximity within 0.01. Ratios pool counts within seed before averaging seeds; scene bootstrapping recomputes these ratios.

For geometric properties, Table 7 shows C versus B lowers CD from 0.0556 to 0.0528 and raises F-Score from 0.7496 to 0.7684, demonstrating closer surface agreement. For the topological concern, the separate feature-space relational diagnostic in Table 6 (Q7) improves affinity-edge F1 from 0.6871 to 0.7063 at 16 seconds versus persistence, providing a proxy for consistency of learned relational structure. For physical plausibility, Table 7 shows C versus B lowers ADE/FDE from 0.0352/0.0525 to 0.0317/0.0486 and raises contact F1 from 0.8533 to 0.8712. These kinematic and geometric-contact proxies show closer agreement with reference motion and object-pair proximity beyond rendered-video scores.

Q4 — Weakness 4

Q. How can architectural gains be separated from heterogeneous data, synthetic samples, pretrained encoders, SDS, and refinement?

A. ST-4D combines public assets and a Genie-3 subset. Data controls in Table 8 hold preparation, checkpoints, teachers, initialization, and test scenes fixed; public-only uses public adaptation groups while retaining shared preparation/pretraining. Full-data results reuse Table 2 exactly.

Foundation encoders, the pretrained diffusion model, and the Gaussian autoencoder remain frozen during supplementary adaptation; applicable conditioning modules are updated. SDS and spatiotemporal-refinement loss settings are shared; A/B omit prediction loss and S/C retain it. EC independently stops within 160 GPU-hours on eight A800 GPUs; preparation/pretraining costs are excluded. In Table 3, M denotes million parameters; GPU-hours sum device training hours, averaged across training seeds.

C's SC/MS gains over B persist across full-data and public-only adaptation under both budgets (Table 2; Table 8). The architecture controls measure prediction's contribution with common pretrained components and SDS/refinement loss settings; the public-only comparison shows that this benefit persists after removing self-built adaptation groups.

Q5 — Weakness 5

Q. How robust is autoregressive graph evolution across increasing rollout horizons, given the acknowledged accumulation of structural errors?

A. EU forecasting in Table 5 and Table 9 uses 100 held-out sequences with ≥16-second continuations. Within each paired seed, one frozen graph/state encoder and normalization define states and actual-future targets/references. Free-running input uses nine observed frames spanning −1…0 seconds, fixed action, and predicted history. One-step prediction uses actual history; persistence repeats the initial state.

Training uses 32 transitions at 1/8 second. Normalized mean squared error (MSE) compares predicted/reference states standardized by training means/SDs; cosine distance is 1−cosine similarity between original state vectors. Full/tail windows contain 8T/final-32 frames; state endpoints are T, last frames T−1/8.

At 4/8/16 seconds, free-running MSE rises to 0.0975/0.1295/0.2194 and remains below persistence's 0.1309/0.2007/0.2453 (Table 5). C also retains higher SC/MS than B in full-sequence and final-four-second windows (Table 9). These results show a sustained advantage in state forecasting and rendered temporal quality through the tested 16-second horizon.

Q6 — Question 1

Q. How does performance change when the current cognition graph directly conditions the generator without the world model?

A. In Table 2, B tests this condition. Table 4 reports EC C−B SC/MS gains of 0.0213/0.0257 for text and 0.0221/0.0256 for images.

The predictive path improves consistency and motion smoothness over static graph reuse in both modalities. C−S confirms these gains with predictor computation retained, supporting the value of evolving conditioning.

Q7 — Question 2

Q. How accurately does predicted future cognition match cognition extracted from actual future observations at the node and relation levels?

A. In Table 6, node error averages endpoint squared distances after Hungarian matching of L2-normalized features using Q5's frozen encoder. Consecutive matching propagates tracks; a switch occurs when a continuing predicted track changes its assigned reference track. Counts accumulate over 32/64/128 transitions and report mean totals per 100 sequences.

Affinity (1+cosine similarity)/2, threshold 0.5, defines undirected non-self edges. Matching permutes both predicted-adjacency axes; precision/recall/F1 compare predicted/reference edges, pooling counts over identical pairs/transitions within seed before averaging seeds. Scene bootstrapping recomputes these ratios. These evaluation-only readouts quantify learned-feature accuracy, association stability, and affinity agreement.

At 16 seconds, prediction reduces node error to 0.2681 versus persistence's 0.3199 and raises affinity F1 to 0.7063 versus 0.6871 (Table 6). The first result addresses node accuracy through matched feature distances; the second addresses relation prediction through thresholded feature-affinity edges. Both show closer agreement with cognition extracted from actual future observations.

Q8 — Question 3

Q. Can additional geometric or topological evaluations substantiate the claims beyond VBench and user studies?

A. Table 7 shows C versus B lowers CD 0.0556→0.0528 and raises contact F1 0.8533→0.8712. The separate feature-space diagnostic in Table 6 (Q7) raises 16-second affinity F1 from 0.6871 to 0.7063 versus persistence.

These evaluations address geometry through reference-surface accuracy and the topological concern through learned-relation proxies, with contact-proximity agreement adding interaction evidence. They complement VBench and user ratings with measurements against future reference observations.

Q9 — Question 4

Q. How much of the performance gain remains when training data and pretrained backbones are matched?

A. Original Table 7 varies backbones; Table 8 fixes them. Public-only text C−B SC/MS gains are 0.0169/0.0178 under EU and 0.0159/0.0174 under EC.

Relative to the full-data C−B gains in Table 2, public-only adaptation retains approximately 80% of the SC gain and 67% of the MS gain under EU, and 75%/68% under EC. This quantifies the substantial prediction benefit retained with matched public adaptation data and fixed pretrained backbones.

Q10 — Question 5

Q. How do generation quality and structural consistency degrade as the autoregressive rollout duration increases?

A. In Table 9, at 4/8/16 seconds, final-four-second SC falls 0.9503/0.9258/0.9056; in Table 6, node error rises 0.1648/0.1859/0.2681.

C retains higher SC/MS than the static control in both full-sequence and final-four-second windows (Table 9), demonstrating sustained relative benefits across the evaluated rollout durations.

Reviewer_WxTWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ST-Gen4D proposes a 4D generation framework centered on a "4D spatiotemporal cognition" (a global appearance graph plus a local dynamic graph, fused through semantic-bridged fusion). An autoregressive world model evolves this cognition, which then conditions latent diffusion for 4D Gaussian generation. The authors also aggregate public 3D/4D datasets together with a Genie-3-generated subset into ST-4D. Experiments cover text/image-to-3D and text/image-to-4D, using VBench and user studies, along with ablations on tokens, graph construction, fusion, and node count.

Critical strengths
  1. Breadth of tasks and a large aggregated dataset. Covering text-to-3D, image-to-3D, text-to-4D, and image-to-4D within one framework, plus ST-4D, is a potentially useful resource.
  2. Structured ablations. The token/graph/fusion/node-count ablations try to isolate components and support the design choices.
Critical weaknesses
  1. The main comparisons do not specify the generation task precisely enough. Tables 3 and 4 lack clear test-set definitions, sample counts, conditioning and rendering settings, baseline reproduction details, and user-study procedures. Without these, it is difficult to determine whether the margins reflect equivalent tasks or to interpret the reported metrics.
  2. The claimed world-model contribution is not directly isolated. The inference path from text or a single image to temporal/action tokens and an initial graph is unclear. The component ablations support graph-based conditioning, but they do not directly establish the benefit of evolving the graph autoregressively over reusing static conditioning. This distinction is central to the claimed contribution in 4D dynamics.
Questions for authors
  1. Inference path. For the text-only and single-image settings, how are the spatial/temporal/action tokens and the initial cognition graph obtained, including any generated intermediate sequence?
  2. Protocol for Tables 3 and 4. Please give a compact table of evaluation source, held-out samples, conditioning inputs, rendering, duration, and VBench dimensions, and state which baselines were reproduced under it.
  3. Autoregressive graph value. Can a control reuse the initial graph over time (same architecture, data, and budget) instead of predicted future graphs? Please report consistency and motion separately.
6 comments

Q1 — Weakness 1

Q. Are the generation tasks and evaluation procedures comparable and clear?

A. Original Tables 3/4 report text/image-to-4D generation. The supplementary evaluation in Table 1 (“Human ratings: EU, fixed training/generation seed 0”) compares internal controls under shared inputs, budgets, and metrics.

We compare A (features), B (static graph), S (static conditioning with predictor retained), and C (predicted cognition). Table 1 uses equal-update adaptation and one fixed training/generation seed. Thirty adults assess 30 scenes/modality: five distinct ratings/output, 240 outputs, 1,200 assignments. Blind randomized presentation shows the allowed condition. Four 1–10 scales assess input alignment, visual quality, scene/object consistency, and temporal smoothness. Cells report means [95% scene-bootstrap confidence intervals (CIs)], aggregating ratings within scenes.

Shared inputs, output settings, sampling, and blinded ratings make the supplementary internal comparisons interpretable within each modality, including the human study in Table 1.

Q2 — Weakness 2

Q. What isolates graph evolution from static conditioning?

A. For the controls in Table 2 (“Generation controls across modalities and budgets”), B retains graphs/Sem-STF without prediction. S retains predictor computation/supervision but supplies initial cognition; C supplies predicted cognition. S/C share architecture. For the paired contrasts in Table 3 (“Paired differences for the principal controls”), C−S tests evolving conditioning; C−B tests the complete predictive module. Variants share data, corresponding checkpoints, and the input representation pipeline, with separate adaptation.

With predictor architecture and computation retained in S, C−S directly tests evolving generator conditioning. The SC/MS gains in Table 2 and the positive paired differences in Table 3 support using predicted cognition.

EU/EC denote 20,000 adaptation updates/a 160 GPU-hour cap. SC/BC/MS/DD/AQ/IQ denote subject/background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality. CLIP is 100×mean normalized cosine similarity between generated frames and the text/initial image. Table 2 reports mean ± sample standard deviation (SD) across three training-seed means, averaging generations within scene/seed. Table 3 reports paired metric differences X−Y [95% scene-bootstrap CIs], resampling unrounded scene differences.

Q3 — Question 1

Q. How do text/image inputs produce initial graphs and temporal/action tokens?

A. Text or a single image is encoded directly, with temporal and action representations inferred from latent input cues and learned priors. No intermediate sequence is generated. Temporal tokens (T_T) and action tokens (T_A) are inferred in feature space from the input's latent cues together with learned temporal and action priors. Actual future observations are not inputs.

Semantic, spatial, and inferred temporal representations construct the initial graphs, with logical-token-guided relations and Sem-STF fusion. Wan2.1's role in Section 3.1 is semantic representation through its 3D Causal VAE. Action tokens condition the world model's state prediction; decoded future cognition then conditions CG-DiT through cross-attention, followed by Gaussian decoding (Sections 3.2–3.4).

Direct input features and learned priors establish the initial cognition and action condition, which the world model uses to predict future cognition.

Q4 — Question 2

Q. What are the data splits, inputs, rendering, metrics, and baseline comparison settings?

A. Original Tables 3 and 4 evaluate text-to-4D and single-image-to-4D generation, respectively. Their VBench columns report consistency, dynamics, and aesthetics, with text alignment additionally reported in Original Table 3. The separate user-study columns assess input alignment, visual quality, consistency, and smoothness.

For the original comparisons, baselines were initialized from publicly released pretrained checkpoints and fine-tuned on the same training split as ST-Gen4D. Their released implementations and method-specific inference or optimization procedures were retained. To make the original comparisons reproducible, the revised evaluation will use an 80/10/10 training/validation/test split by source scene or asset, keeping related views and temporal segments in the same partition. Within each modality, methods will receive the same held-out inputs and use matched evaluation cameras, rendering settings, clip durations, and metric implementations.

This shared protocol will align inputs and evaluation across methods within each task, while the complementary automated and human assessments cover both rendered quality and perceptual consistency.

Q5 — Question 3

Q. Does predicted cognition improve consistency and motion under matched controls?

A. Table 2 separates modalities/budgets; B improves SC/MS over A. Text/EU C/S SC is 0.9737/0.9622 and MS is 0.9678/0.9458. In Table 3, C−S intervals for SC and MS remain above zero in both modalities and budget settings. CLIP follows modality.

The positive C−S intervals support improvements in both subject consistency and motion smoothness across modalities and budgets under the matched controls.

Table 1 Human ratings: EU, fixed training/generation seed 0

TaskVariantAlignment ↑Quality ↑Consistency ↑Smoothness ↑
Text-to-4DA8.13<br>[7.93, 8.34]8.58<br>[8.39, 8.79]8.95<br>[8.72, 9.18]8.63<br>[8.43, 8.85]
Text-to-4DB9.40<br>[9.23, 9.57]8.93<br>[8.71, 9.14]9.17<br>[8.97, 9.37]8.95<br>[8.80, 9.11]
Text-to-4DS8.42<br>[8.27, 8.60]9.17<br>[9.02, 9.32]9.34<br>[9.18, 9.50]8.79<br>[8.61, 8.99]
Text-to-4DC9.11<br>[8.96, 9.27]9.31<br>[9.15, 9.46]9.46<br>[9.30, 9.61]9.50<br>[9.36, 9.63]
Single-image-to-4DA7.67<br>[7.41, 7.93]8.78<br>[8.57, 8.99]8.15<br>[7.93, 8.37]8.55<br>[8.35, 8.75]
Single-image-to-4DB8.79<br>[8.57, 9.01]9.01<br>[8.82, 9.20]8.58<br>[8.35, 8.81]9.06<br>[8.85, 9.26]
Single-image-to-4DS9.11<br>[8.89, 9.32]8.95<br>[8.75, 9.15]8.49<br>[8.23, 8.74]8.87<br>[8.67, 9.06]
Single-image-to-4DC9.23<br>[9.04, 9.41]9.34<br>[9.15, 9.51]8.81<br>[8.61, 9.02]9.08<br>[8.87, 9.28]

Table 2 Generation controls across modalities and budgets

SettingControlSC ↑BC ↑MS ↑DDAQ ↑IQ ↑CLIP ↑
Text/EUA0.9345 ± 0.00090.9415 ± 0.00190.9239 ± 0.00130.4744 ± 0.01840.6093 ± 0.00240.7167 ± 0.002232.14 ± 0.01
Text/EUB0.9526 ± 0.00160.9482 ± 0.00150.9413 ± 0.00280.5889 ± 0.01500.6228 ± 0.00130.7939 ± 0.002632.86 ± 0.03
Text/EUS0.9622 ± 0.00200.9519 ± 0.00160.9458 ± 0.00170.7378 ± 0.00510.6279 ± 0.00220.7865 ± 0.001432.94 ± 0.04
Text/EUC0.9737 ± 0.00110.9617 ± 0.00070.9678 ± 0.00090.7144 ± 0.00770.6513 ± 0.00070.8072 ± 0.002633.38 ± 0.04
Text/ECA0.9374 ± 0.00510.9443 ± 0.00280.9267 ± 0.00230.4733 ± 0.02330.6130 ± 0.00230.7193 ± 0.000332.27 ± 0.03
Text/ECB0.9561 ± 0.00210.9498 ± 0.00040.9448 ± 0.00150.6056 ± 0.01170.6239 ± 0.00020.7962 ± 0.001932.92 ± 0.08
Text/ECS0.9634 ± 0.00140.9531 ± 0.00310.9481 ± 0.00070.7467 ± 0.03060.6292 ± 0.00020.7879 ± 0.003733.15 ± 0.01
Text/ECC0.9774 ± 0.00090.9630 ± 0.00180.9704 ± 0.00130.7311 ± 0.02990.6551 ± 0.00150.8093 ± 0.002633.50 ± 0.08
Image/EUA0.9418 ± 0.00050.9466 ± 0.00250.9315 ± 0.00300.4911 ± 0.02040.6182 ± 0.00230.7413 ± 0.000881.72 ± 0.07
Image/EUB0.9583 ± 0.00140.9554 ± 0.00130.9491 ± 0.00290.5622 ± 0.02010.6367 ± 0.00090.7986 ± 0.003383.48 ± 0.05
Image/EUS0.9607 ± 0.00070.9538 ± 0.00160.9513 ± 0.00200.6033 ± 0.01200.6392 ± 0.00200.7934 ± 0.000583.76 ± 0.03
Image/EUC0.9786 ± 0.00050.9662 ± 0.00070.9741 ± 0.00200.7333 ± 0.01860.6584 ± 0.00020.8197 ± 0.002185.29 ± 0.15
Image/ECA0.9428 ± 0.00270.9501 ± 0.00460.9346 ± 0.00010.4978 ± 0.02270.6213 ± 0.00200.7453 ± 0.001481.94 ± 0.04
Image/ECB0.9601 ± 0.00150.9595 ± 0.00270.9517 ± 0.00180.5789 ± 0.02710.6384 ± 0.00070.8012 ± 0.000283.61 ± 0.06
Image/ECS0.9643 ± 0.00160.9574 ± 0.00160.9545 ± 0.00140.6178 ± 0.01350.6405 ± 0.00300.7960 ± 0.001783.95 ± 0.08
Image/ECC0.9822 ± 0.00060.9672 ± 0.00150.9773 ± 0.00090.7367 ± 0.01530.6609 ± 0.00140.8209 ± 0.003685.52 ± 0.06
Reviewer_bXztOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ST-Gen4D, a generative framework that embeds 4D spatiotemporal cognition—integrating global appearance structure and local dynamic topology—into a world model. This approach addresses key limitations of existing 4D generation methods, which rely heavily on external macro-scale constraints and fail to capture fine-grained local dynamics, resulting in oversmoothing and motion artifacts. ST-Gen4D integrates spatiotemporal representations extracted from multiple encoders into a structured cognitive graph. A causal Transformer-based world model then predicts future cognitive states, which condition a latent diffusion model to generate 4D Gaussians. Furthermore, the authors introduce the ST-4D dataset, created by combining public 3D/4D datasets with newly generated data. Extensive experiments on T3Bench, 3D-Front, and VBench demonstrate that ST-Gen4D outperforms state-of-the-art 3D/4D generation techniques, while ablation studies validate the contribution of each component.

Critical strengths
  1. Unlike conventional 4D generation relying on external macro constraints, this paper constructs a "4D cognition graph" integrating global appearance structure with local dynamic topology. Combining semantic-bridged fusion with a predictive world model effectively preserves spatial structural integrity and temporal coherence through state transitions.
  2. The method consistently sets new benchmarks across Text-to-3D, Image-to-3D, and Text-to-4D tasks (57.6 on T3Bench, 0.051 CD on 3D-Front, 97.43 temporal consistency on VBench). Thorough ablation studies systematically validate the explicit contribution of each proposed component.
Critical weaknesses
  1. ST-Gen4D integrates multiple existing foundation models and modules (encoders, world models, 4D-GS). While structurally coherent, the paper lacks a clear explanation of how "Semantic-Bridged Spatiotemporal Fusion" fundamentally differs from conventional fusion methods, and what unique capabilities are unlocked specifically through its integration with the world model.
  2. Heavy reliance on a massive dataset (ST-4D) and large pre-trained models makes it difficult to determine whether performance gains stem from the proposed 4D cognition mechanism or simply from data scale and model capacity. Controlled comparisons under unified training data/compute constraints are missing, and the trade-off between high computational overhead and actual performance gains remains insufficiently analyzed.
Questions for authors
  1. How do the proposed 4D Cognition Graph and "Semantic-Bridged Spatiotemporal Fusion" (Sem-STF) fundamentally differ from conventional spatiotemporal feature fusion mechanisms (e.g., cross-attention or direct concatenation)? Please provide explicit justification or ablation results showing the technical advantages of the structured graph representation over standard feature fusion baselines.
  2. Given the heavy reliance on pre-trained foundation models and the massive ST-4D dataset, how can the authors verify that the observed performance gains stem from the 4D spatiotemporal cognition mechanism rather than data scale and model capacity? Please provide quantitative comparisons against a baseline that omits the cognition graph and world model under identical training data and compute constraints.
  3. How do the authors verify that future state prediction via the world model specifically improves generation quality? Are there quantitative metrics assessing future state prediction errors and long-term error accumulation over extended sequences? Furthermore, please clarify the performance gap between ST-Gen4D and an ablation variant that bypasses the world model to directly condition the diffusion model on the current cognitive graph.
6 comments

Q1 — Weakness 1

Q. What distinguishes Sem-STF and its integration with the world model?

A. Semantic, global-appearance, and local-dynamics graphs combine positional/geometric embeddings with logical-token-guided message updates. Sem-STF aligns global/local streams through a common semantic query and adaptive fusion. An action-conditioned world model predicts compressed states and decodes future cognition for CG-DiT.

The graph ablation in Original Table 6 improves consistency from 86.24 to 97.83; Original Table 8 shows that Sem-STF improves over weighted fusion from 93.85 to 97.83.

The contribution is the coupling of fusion through a common semantic query with action-conditioned prediction of future graph conditioning. The ablations support the graph/fusion stages.

Q2 — Weakness 2

Q. How are data/prior effects controlled, and what is the computational trade-off?

A. The A/B/S/C controls in Table 1 (“Generation controls across modalities and budgets”) share grouped data splits, checkpoints, teachers, initialization, and the input-encoding pipeline. A removes graphs/prediction; B repeats initial graph conditioning; S retains predictor computation/supervision but supplies initial cognition; C supplies predictions. Each is adapted separately.

Text/image tasks share 100 held-out scenes, with three training and three generation seeds. Inputs are captions or one image; temporal/action representations are predicted from latent input cues using learned priors, without actual future observations. EU fixes 20,000 updates; EC caps additional adaptation at 160 GPU-hours on eight A800 GPUs. Foundational encoders, SDS teacher, and Gaussian autoencoder stay frozen; applicable graph/conditioning modules adapt. Public-only adaptation in Table 2 (“Public-only adaptation across modalities and budgets”) removes self-built groups but retains shared preparation and external priors.

In Table 1 and Table 2, SC/BC/MS/DD/AQ/IQ denote subject/background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality. T-/I-CLIP is 100×mean normalized CLIP cosine similarity between generated frames and the prompt/initial image. Generation results report mean ± sample standard deviation (SD) across three training-seed summaries, after averaging generation seeds within scene/seed.

In Table 2, public-only text/EU C−B retains SC/MS gains of 0.0169/0.0178. Under EC, Table 1 also shows C improves SC/MS over graph-free A and parameter-matched S in both modalities.

Combining Table 1 with Table 3 (“Trainable parameters and adaptation costs: Text/EU”), text/EU C improves SC/MS over B by 0.0211/0.0265 for an additional 0.94 adaptation GPU-hours (+0.68%). S/C share 187.92 M parameters (M=million). GPU-hours sum device training hours, averaged across training seeds. Adaptation excludes preparation/pretraining.

Together, the public-only, compute-capped, and parameter-matched controls in Table 2 and Table 1 support a contribution from predicted cognition under shared data and pretrained priors. In text/EU, the SC/MS gains require only 0.68% additional adaptation cost over static graph conditioning.

Q3 — Question 1

Q. How does structured fusion compare with concatenation and ordinary attention?

A. Fusion controls in Table 4 retain graphs, prediction, and generation with matched interfaces/attention dimensions. Ordinary attention receives a semantic residual, preserving semantic access without common-query alignment. The Weighted baseline in Original Table 8 is distinct from learned weighting here.

Comparing Table 4 with the corresponding C rows in Table 1, text/EU Sem-STF improves SC/MS from 0.9458/0.9637 to 0.9737/0.9678 relative to ordinary attention.

Sem-STF achieves higher SC/MS than both concatenation and ordinary attention across the modality/budget comparisons. This supports semantic-query alignment and adaptive fusion for subject consistency and motion smoothness with matched conditioning interfaces.

Q4 — Question 2

Q. What happens when both graphs and the world model are removed?

A. In Table 1, A implements joint removal with a trainable adapter and common generator interface. C−A tests the combined path; C−B tests prediction including action conditioning; C−S tests predicted conditioning with predictor architecture retained. Each control is adapted separately.

Table 1 separates modalities and budgets: text/EU C versus A improves SC 0.9345→0.9737, MS 0.9239→0.9678, and T-CLIP 32.14→33.38.

Jointly removing graphs and prediction therefore reduces consistency, motion smoothness, and input alignment relative to C. Retaining static graphs in B recovers part of this benefit, supporting the combined cognition pathway.

Q5 — Question 3

Q. Does prediction help, and how does accuracy degrade with rollout duration?

A. Table 5 shows text/EU C−S SC/MS gains of 0.0115/0.0220, with positive 95% confidence intervals (CIs), supporting evolving conditioning at matched predictor capacity.

Table 6 evaluates 100 held-out sequences using nine observed frames, prefix-derived fixed actions, and a shared frozen encoder within each seed. Free-running prediction uses its own history; one-step prediction uses actual history; persistence repeats the initial state. Training covers 32 transitions at 1/8 second. Normalized mean squared error (MSE) compares predicted/reference states standardized by training means/SDs; cosine distance is 1−cosine similarity. At 4/8/16 seconds, MSE rises 0.0975/0.1295/0.2194 but remains below persistence (0.1309/0.2007/0.2453).

Table 7 shows full-sequence SC declining 0.9503/0.9341/0.9167 across these horizons. C maintains higher SC/MS than B for both full sequences and final-four-second windows.

Table 8 reports mean squared endpoint node-feature error after Hungarian matching of L2-normalized features, feature-track assignment switches per 100 sequences, and edge precision/recall/F1 using affinity (1+cosine similarity)/2 thresholded at 0.5. Switches count changes in the reference assigned to a continuing predicted track; F1 is precision/recall's harmonic mean. These are learned-feature diagnostics. At 16 seconds, prediction reduces node error 0.3199→0.2681 and raises affinity F1 0.6871→0.7063 versus persistence.

State/feature cells report mean ± training-seed SD and [95% scene-bootstrap CI]; paired cells report mean differences [CIs]. X−Y means metric(X)−metric(Y); CLIP denotes text/image alignment according to input modality.

Prediction improves generation over static B and S while retaining lower forecasting error and better temporal quality through 16 seconds despite increasing rollout error.

Table 1 Generation controls across modalities and budgets

SettingControlSC ↑BC ↑MS ↑DDAQ ↑IQ ↑CLIP ↑
Text/EUA0.9345 ± 0.00090.9415 ± 0.00190.9239 ± 0.00130.4744 ± 0.01840.6093 ± 0.00240.7167 ± 0.002232.14 ± 0.01
Text/EUB0.9526 ± 0.00160.9482 ± 0.00150.9413 ± 0.00280.5889 ± 0.01500.6228 ± 0.00130.7939 ± 0.002632.86 ± 0.03
Text/EUS0.9622 ± 0.00200.9519 ± 0.00160.9458 ± 0.00170.7378 ± 0.00510.6279 ± 0.00220.7865 ± 0.001432.94 ± 0.04
Text/EUC0.9737 ± 0.00110.9617 ± 0.00070.9678 ± 0.00090.7144 ± 0.00770.6513 ± 0.00070.8072 ± 0.002633.38 ± 0.04
Text/ECA0.9374 ± 0.00510.9443 ± 0.00280.9267 ± 0.00230.4733 ± 0.02330.6130 ± 0.00230.7193 ± 0.000332.27 ± 0.03
Text/ECB0.9561 ± 0.00210.9498 ± 0.00040.9448 ± 0.00150.6056 ± 0.01170.6239 ± 0.00020.7962 ± 0.001932.92 ± 0.08
Text/ECS0.9634 ± 0.00140.9531 ± 0.00310.9481 ± 0.00070.7467 ± 0.03060.6292 ± 0.00020.7879 ± 0.003733.15 ± 0.01
Text/ECC0.9774 ± 0.00090.9630 ± 0.00180.9704 ± 0.00130.7311 ± 0.02990.6551 ± 0.00150.8093 ± 0.002633.50 ± 0.08
Image/EUA0.9418 ± 0.00050.9466 ± 0.00250.9315 ± 0.00300.4911 ± 0.02040.6182 ± 0.00230.7413 ± 0.000881.72 ± 0.07
Image/EUB0.9583 ± 0.00140.9554 ± 0.00130.9491 ± 0.00290.5622 ± 0.02010.6367 ± 0.00090.7986 ± 0.003383.48 ± 0.05
Image/EUS0.9607 ± 0.00070.9538 ± 0.00160.9513 ± 0.00200.6033 ± 0.01200.6392 ± 0.00200.7934 ± 0.000583.76 ± 0.03
Image/EUC0.9786 ± 0.00050.9662 ± 0.00070.9741 ± 0.00200.7333 ± 0.01860.6584 ± 0.00020.8197 ± 0.002185.29 ± 0.15
Image/ECA0.9428 ± 0.00270.9501 ± 0.00460.9346 ± 0.00010.4978 ± 0.02270.6213 ± 0.00200.7453 ± 0.001481.94 ± 0.04
Image/ECB0.9601 ± 0.00150.9595 ± 0.00270.9517 ± 0.00180.5789 ± 0.02710.6384 ± 0.00070.8012 ± 0.000283.61 ± 0.06
Image/ECS0.9643 ± 0.00160.9574 ± 0.00160.9545 ± 0.00140.6178 ± 0.01350.6405 ± 0.00300.7960 ± 0.001783.95 ± 0.08
Image/ECC0.9822 ± 0.00060.9672 ± 0.00150.9773 ± 0.00090.7367 ± 0.01530.6609 ± 0.00140.8209 ± 0.003685.52 ± 0.06

Table 2 Public-only adaptation across modalities and budgets

SettingControlSC ↑BC ↑MS ↑DDAQ ↑IQ ↑
Text/EUB0.8942 ± 0.00150.9304 ± 0.00090.9423 ± 0.00130.5422 ± 0.01170.6898 ± 0.00170.6962 ± 0.0008
Text/EUC0.9111 ± 0.00150.9436 ± 0.00200.9601 ± 0.00120.7144 ± 0.01840.6791 ± 0.00250.7086 ± 0.0034
Text/ECB0.8971 ± 0.00390.9315 ± 0.00360.9451 ± 0.00210.5444 ± 0.03100.6928 ± 0.00260.6994 ± 0.0026
Text/ECC0.9130 ± 0.00200.9449 ± 0.00230.9625 ± 0.00110.7156 ± 0.01170.6804 ± 0.00240.7094 ± 0.0035
Image/EUB0.8994 ± 0.00050.9347 ± 0.00150.9484 ± 0.00040.5611 ± 0.01020.6932 ± 0.00370.7090 ± 0.0007
Image/EUC0.9163 ± 0.00320.9479 ± 0.00220.9662 ± 0.00150.7322 ± 0.03290.6825 ± 0.00240.7214 ± 0.0014
Image/ECB0.9015 ± 0.00050.9378 ± 0.00230.9508 ± 0.00140.5644 ± 0.02140.6950 ± 0.00140.7097 ± 0.0020
Image/ECC0.9195 ± 0.00160.9493 ± 0.00260.9693 ± 0.00180.7344 ± 0.01020.6835 ± 0.00100.7243 ± 0.0018

Table 3 Trainable parameters and adaptation costs: Text/EU

VariantTrainable parametersAdaptation GPU-h, seed mean
A188.96 M143.43
B174.87 M137.91
S187.92 M138.27
C187.92 M138.85

Comments on the paper

1 comment
AuthorsOpenReview ↗

To all reviewers

We thank all reviewers for their detailed and constructive comments. We appreciate their recognition of the following strengths:

  • Well-motivated modeling of global appearance and local dynamics [bXzt, VEYy].

  • Strong performance across 3D and 4D generation tasks [bXzt].

  • Broad task coverage and the ST-4D dataset resource [WxTW].

  • Comprehensive component-level ablation studies [bXzt, WxTW, VEYy].

To address the reviewers’ key concerns, we have added blue-highlighted clarifications to the revised manuscript covering:

  • text-only and single-image inference, including missing-modality representations and the initialization of temporal/action features [WxTW].

  • The distinctive shared-query alignment and adaptive fusion in Sem-STF, supported by comparisons with weighted fusion and conventional cross-attention [bXzt, VEYy].

  • Shared training/test splits, evaluation conditions, and baseline implementation procedures [WxTW].

  • Comparisons of feature, static-graph, and predicted-graph conditioning under matched adaptation budgets (Table 9) [bXzt, WxTW, VEYy].

  • Direct evaluations of state and node prediction errors and learned-relation agreement across rollout horizons (Table 10) [bXzt, VEYy].

  • Surface, trajectory, and geometric-contact evaluations beyond rendered-video metrics (Table 11) [VEYy].

We once again thank all reviewers for their patience, thoughtful responses, and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.