acceptodds
Under review as a conference paper at ICLR 2027

4DGS-JEPA: Temporally Compositional Joint-Embedding Prediction for Dynamic Gaussian Splatting

Abstract

Dynamic Gaussian Splatting explicitly represents evolving 3D scenes, but is typically optimized for reconstruction, rendering, or future-state generation rather than reusable predictive dynamics. We propose 4DGS-JEPA, a Gaussian-native joint-embedding predictive architecture for causal multi-horizon prediction. It combines hierarchical scene, motion-group, and Gaussian-level representations with a horizon-conditioned transition supporting both direct prediction and recursive rollout. Its central principle is temporal composition: different chronological transition paths reaching the same future endpoint should produce compatible predictive states. Endpoint and multi-horizon path supervision anchor predictions to future target embeddings, while a selective geometry decoder and geometry-level composition ground the learned dynamics in coherent group motion and Gaussian geometry without reconstructing complete future appearance. We further introduce hybrid correspondence that preserves reliable persistent canonical identity and applies residual optimal transport when correspondence becomes ambiguous under reordering or topology change. We theoretically characterize zero-loss path agreement and finite-error rollout accumulation. In controlled dynamic Gaussian worlds, temporal composition reduces direct–composed latent discrepancy by about 61% while slightly improving predictive accuracy, geometry-level composition reduces group-motion path discrepancy by about 72%, hybrid correspondence remains robust under reordering and topology change, and predicted Gaussian geometry can be rendered into held-out future frames without pixel-level training. These results establish 4DGS-JEPA as a predictive, temporally compositional formulation of dynamic Gaussian worlds.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
4D Gaussian Splatting, JEPA, world models, temporal composition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4HnkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Gaussian-Native Joint Embedding Predictive Framework for predicting future states in dynamic scenarios. The method predicts future latent states using Hierarchical Gaussian Representations and a Horizon-Conditioned Transition Model. Additionally, a Temporal Composition Loss and a Geometric Composition Loss are designed to constrain the consistency between direct and recursive prediction results. Furthermore, the method introduces a Hybrid Correspondence mechanism to address uncertainties in the Gaussian primitive identity matching process. This method was experimentally evaluated on controlled synthetic scenes, with its performance validated primarily across four aspects: prediction consistency, geometric prediction accuracy, identity correspondence matching performance, and rendering quality of future frames.

Critical strengths

Compared to directly reconstructing the complete appearance of future scenes, predicting future Gaussian representations is a research direction with clear research motivation and practical significance. The Temporal Composition Objective and Geometric Composition Objective proposed in this paper provide a structured and theoretically grounded regularization method for prediction results across multiple temporal domains, helping to improve consistency among different prediction paths.

The ablation experiments clearly demonstrate that the proposed composition loss effectively reduces discrepancies in results across different prediction paths, thereby enhancing prediction consistency. Furthermore, the paper provides detailed experimental setups and supplementary experimental results, offering strong support for the method’s reproducibility.

Critical weaknesses

The paper relies entirely on synthetic data, with the main experiments involving only 18 Gaussian distributions and 3 predefined motion groups, lacking validation under real-world scenarios and reconstruction noise. Furthermore, the Hybrid mechanism and the prediction network were tested separately, and the effectiveness of the complete framework in actual prediction tasks has not yet been demonstrated.

In Table 1, although Path + Composition reduces CompErr from 0.00244 to 0.00058, Direct FDE actually increases from 0.03013 to 0.03206. In Table 4, simple constant-velocity extrapolation outperforms all learning models. The fundamental differences between this method and existing time-consistency constraints are not sufficiently clear; insufficient qualitative visualizations and overly verbose text also weaken the paper’s clarity and overall contribution.

Questions for authors

Q1. The current unseen horizons (3 and 6) effectively test interpolation rather than extrapolation beyond the training range. Could the authors evaluate prediction performance further beyond the training horizons, e.g., 12 or 16 steps ahead, and report both FDE and composition error for those settings? Comparisons with constant-velocity, constant-acceleration, and a simpler learned predictor would also help clarify the actual forecasting benefit of the proposed method.

Q2. Experiment 3 evaluates the matching mechanism in isolation, so its contribution to end-to-end forecasting accuracy remains unverified. Could the authors provide a small predictive experiment that involves Gaussian birth, death, split, or merge events, and compare the performance of fixed correspondence, OT-only, and the proposed Hybrid correspondence in that end-to-end setting?

Q3. What is the empirical benefit of the three-level Gaussian hierarchy (scene, motion-group, Gaussian) over a simpler capacity-matched encoder without the hierarchical structure? It would also be helpful if the authors report the model’s sensitivity to noisy Gaussian positions or incorrect motion-group assignments.

Q4. The current figures do not adequately illustrate the practical effect of the proposed consistency objectives. Could the authors provide multi-timestamp comparisons of predicted geometry against ground truth and simple baselines, preferably with visualizations of motion trajectories and corresponding error profiles?

3 comments

Thank you for recognizing the motivation for Gaussian-native prediction, and the temporal/geometric composition ablations. We have added Experiment 5 (Table 5; Appendix G), expanded the empirical comparison, and clarified the scope of the controlled results. We address the weaknesses and four questions below.

Regarding the main weaknesses. Experiment 5 extends evaluation beyond the original 18-Gaussian, three-group synthetic setting to reconstructed real scenes. The controlled ablations distinguish predictive accuracy from path consistency, while the correspondence tests remain separate from forecasting. We have also clarified the methodological novelty and improved the manuscript's organization and presentation.

Q1. Extrapolation, baselines, and accuracy versus consistency. Experiment 1 trains on horizons {1,2,4,8}; the unseen {3,6} horizons test generalization within that range, not extrapolation to 12 or 16 steps. We have not run the requested Δ=12/16 FDE and CompErr or constant-acceleration comparisons. The Endpoint-only variant is a simpler learned-loss baseline, though not a separate capacity-matched predictor.

To assess extrapolation beyond observed scene times, new Experiment 5 uses six real HyperNeRF scenes (final 10% withheld) and eight synthetic D-NeRF scenes (final 20% withheld). ODE-GS and 4DGS-JEPA share a prefix-fitted Deformable-GS front-end, use comparable Stage-2 capacities (3.88M/4.08M parameters), matched 90k-step training, three seeds, and the same rasterizer. Relative to ODE-GS, dynamic-region PSNR improves 12.86→13.19 dB on HyperNeRF and 12.14→13.61 dB on D-NeRF; full-image PSNR is 18.73 vs. 19.07 dB and 26.60 vs. 25.75 dB, respectively. Other full-image metrics are mixed (Table 5).

We agree with the Table 1 trade-off: Endpoint + Path + Composition lowers CompErr 0.00244→0.00058 but raises mean direct FDE 0.03013→0.03206. Endpoint + Composition, by contrast, lowers both FDE 0.03013→0.02905 and CompErr 0.00244→0.00096. Constant velocity is stronger on average for these smooth trajectories (0.02484 vs. 0.02905 FDE) and in one-step rendering (60.48 vs. 48.24 dB); at the longest trained horizon (Δ=8), Endpoint + Composition does better in direct FDE (0.05687 vs. 0.07120). Thus improved path consistency does not automatically imply better forecasting at every horizon (Table 1; Fig. 2).

Q2. End-to-end correspondence under topology change. Experiment 3 tests index, OT-only, and Hybrid matching under permutations, splits/merges, birth/death, and identity dropout. Hybrid improves split/merge lineage mass over OT 0.98100→0.98715 and reduces induced geometry discrepancy 0.05010→0.04906 (Table 3; Fig. 4; Appendix F). OT and Hybrid both attain birth/death F1 1.0 in this test. These are standalone correspondence results; forecasting FDE under birth/death or split/merge with fixed/OT/Hybrid matching has not been evaluated end-to-end. Experiment 5 likewise does not integrate learned residual-OT correspondence.

Q3. Hierarchy and robustness. The scene/group/Gaussian hierarchy and bottom-up pooling are specified in Sec. 3 and Appendix B, but the manuscript contains no capacity-matched flat-encoder ablation; hence we cannot attribute an isolated gain to the hierarchy. Controlled forecasting uses known motion groups. Experiment 3 includes positional noise (standard deviation 0.025), scale/semantic perturbations, and missing IDs for matching, not forecasting. Sensitivity of predicted FDE to noisy input positions or incorrect group assignments remains unmeasured.

Q4. Visualizations and distinction from temporal smoothing. The manuscript reports horizon-wise FDE and CompErr curves (Fig. 2), group-transform and Gaussian-discrepancy profiles (Fig. 3), correspondence stress-test curves (Fig. 4), and a held-out rendering alongside ground truth and analytic baselines (Fig. 5); Appendix F compares direct and composed rendering routes. These provide error profiles and visual evidence, although a multi-timestamp trajectory montage against ground truth has not been added. Methodologically, our composition loss compares different chronological decompositions reaching the same endpoint through the same closed, horizon-conditioned transition, anchored by future targets, and extends agreement to decoded SE(3) transforms and Gaussian geometry. This differs from simply enforcing adjacent-time smoothness; we do not claim temporal regularization itself is new.

We have also shortened the main exposition and consolidated the appendices. Together, we hope the revision strengthens the evidence for temporal and geometric consistency and extends the evaluation to extrapolation on reconstructed real and synthetic scenes.

Reviewer_4HnkOpenReview ↗

Post-Rebuttal Assessment

Thank you for the detailed response and additional experiments. I appreciate the extension to reconstructed real scenes. However, the improvements remain inconsistent across different settings. In particular, the full-image PSNR on D-NeRF decreases from 26.60 to 25.75 dB despite improvements in dynamic-region PSNR.

Moreover, the additional experiments do not sufficiently establish the independent contribution of temporal composition to forecasting accuracy. The end-to-end effectiveness of Hybrid correspondence and the necessity of the hierarchical architecture also remain unverified.

Overall, while the revision partially addresses my concerns, the evidence is still insufficient to demonstrate consistent predictive improvements and fully justify the proposed framework. Therefore, I maintain my original rating of 2 (Weak Rejection).

Thank you for your further assessment, and for recognizing the extension to reconstructed real scenes. We understand your concerns about inconsistent improvements and the independent contributions of the proposed components.

First, we would like to clarify one numerical point: the D-NeRF full-image PSNR comparison is 26.60 dB for 4DGS-JEPA versus 25.75 dB for ODE-GS, so our method improves PSNR by 0.85 dB, rather than decreasing it. It also improves dynamic-region PSNR by 1.47 dB and reduces dynamic LPIPS from 0.4397 to 0.3888. We agree that ODE-GS retains better full-image SSIM and LPIPS on D-NeRF, as well as stronger full-image results on HyperNeRF.

Regarding temporal composition, the controlled ablations distinguish prediction accuracy from route consistency. Adding composition to endpoint supervision lowers direct FDE from 0.03013 to 0.02905 while reducing latent path discrepancy from 0.00244 to 0.00096. Adding both path supervision and composition achieves the lowest path discrepancy (0.00058), although direct FDE is higher. Thus, the experiments establish a measurable consistency benefit and an accuracy improvement for a specific configuration, without assuming that lower path discrepancy always means lower forecasting error.

Regarding the other components, Experiment 2 isolates geometry-level composition, reducing group-transform path disagreement by 72.2%, while Experiment 3 evaluates hybrid correspondence under reordering and topology changes, improving split/merge lineage recovery from 0.98100 to 0.98715 relative to OT-only. These experiments establish the behavior of the individual mechanisms; they do not constitute a capacity-matched hierarchy ablation or an end-to-end hybrid-correspondence forecasting comparison.

We have also strengthened the qualitative evaluation. The new Fig. 6 compares held-out forecasts with ground truth, Figs. 8-9 show all fourteen reconstructed scenes, and Figs. 10-11 provide multi-timestamp real-scene renderings. These complement the horizon-wise prediction and consistency profiles already reported.

We believe the central contribution is the integration of Gaussian-native predictive states, temporally compositional transitions, geometry-grounded decoding, and identity-aware correspondence into a JEPA-style predictive framework. The controlled studies establish the roles of these mechanisms, while the new reconstructed-scene experiments demonstrate their relevance to future-time Gaussian forecasting. We believe this combined methodological and empirical contribution is useful, despite the remaining component-level validation gaps, and respectfully hope you will reconsider its significance.

Reviewer_VRmZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents 4DGS-JEPA, which predicts dynamic Gaussian scene states using latent prediction and a temporal-composition loss intended to improve consistency across prediction paths. It also introduces a geometry decoder and a hybrid correspondence mechanism. The components are evaluated mainly in controlled, procedurally generated settings.

Critical strengths
  1. The paper clearly distinguishes prediction accuracy from consistency across temporal paths.
  2. Synthetic experiments show reduced path discrepancy in latent states and geometric motion.
  3. The paper discusses its limitations in reasonable detail.
Critical weaknesses
  1. The method is not evaluated on real dynamic scenes or Gaussian representations reconstructed causally from observations, leaving its robustness to reconstruction, tracking, and occlusion errors unclear. Constant-velocity prediction outperforms the learned model on several reported metrics, limiting the evidence for improved predictive performance.
  2. Hybrid correspondence, geometry composition, and state prediction are largely evaluated separately; their combined end-to-end benefit is not established.
  3. The rendering experiment fixes appearance and camera parameters. It demonstrates renderability of predicted geometry, but not prediction of realistic appearance or future video.
  4. The theoretical analysis relies on idealized assumptions and is not convincingly connected to error behavior or long-horizon performance in trained models.
Questions for authors
  1. Can the method be evaluated on real dynamic scenes using Gaussian representations reconstructed causally from observations?
  2. Why does the constant-velocity baseline outperform the learned model in several experiments, and does temporal composition provide a consistent long-horizon prediction benefit?
  3. Can hybrid correspondence be integrated end to end with the predictive model and evaluated under topology changes?
  4. What conclusions about future-scene prediction are supported by the rendering experiment with fixed appearance and camera parameters?
3 comments

Thank you for recognizing the distinction between predictive accuracy and path consistency. We added Experiment 5 (Table 5; Appendix G), clarified the controlled experiments and theoretical claims, and streamlined the presentation.

1. Real scenes and reconstruction uncertainty (Q1). We evaluate six real HyperNeRF and eight synthetic D-NeRF scenes, withholding the final 10%/20% of chronological observations. All methods share a Deformable-GS representation fitted for 40k iterations on the observed prefix only. ODE-GS and 4DGS-JEPA use comparable Stage-2 modules (3.88M/4.08M parameters), matched 90k-step training, three seeds, and the same renderer; future frames are excluded from fitting and model selection.

Against ODE-GS, HyperNeRF dynamic PSNR rises 12.86→13.19 dB and dynamic LPIPS falls 0.4991→0.4832, although full-image PSNR is lower (18.73 vs. 19.07 dB). On D-NeRF, full PSNR rises 25.75→26.60 dB, dynamic PSNR 12.14→13.61 dB, and dynamic LPIPS falls 0.4397→0.3888; ODE-GS retains better full-image SSIM/LPIPS. These results demonstrate moving-region extrapolation from prefix-fitted reconstructed Gaussians, not streaming reconstruction or isolated robustness to tracking, occlusion, or group-estimation errors.

2. Constant velocity and long horizons (Q2). Constant velocity is strong because the procedural trajectories are smooth: its mean FDE is 0.02484 versus 0.02905 for Endpoint + Composition, and its one-step rendering PSNR is 60.48 versus 48.24 dB for the best learned variant. At the longest trained horizon, Δ=8, however, Endpoint + Composition has lower direct FDE (0.05687 vs. 0.07120). Relative to Endpoint alone, it reduces mean direct FDE 0.03013→0.02905 (3.6%) and CompErr 0.00244→0.00096 (60.6%). Adding both path and composition achieves the lowest CompErr (0.00058) and rollout gap (0.00029) but higher direct FDE (0.03206; Table 1, Fig. 2). Thus path consistency and accuracy are distinct, and composition does not guarantee long-horizon superiority. The controlled unseen horizons Δ=3/6 lie within the training range; Experiment 5 instead tests extrapolation beyond observed scene times, not a real-scene composition ablation.

3. Geometry and hybrid correspondence (Q3). Within the predictive model, explicit SE(3) composition reduces group-transform path discrepancy 0.00724→0.00202 (72.2%), with comparable direct FDE (0.06614 vs. 0.06675); full geometry composition lowers Gaussian discrepancy 0.00192→0.00130 (Table 2, Fig. 3). Separately, under reordering and topology perturbations, Hybrid improves split/merge lineage mass relative to OT-only (0.98100→0.98715) and lowers induced geometry discrepancy (0.05010→0.04906; Table 3, Fig. 4). This matching experiment is not end-to-end forecasting. Experiments 1–2 assume fixed identities, and Experiment 5 does not jointly evaluate learned prediction with residual OT. Hence the combined forecasting benefit under birth/death or split/merge remains unestablished.

4. Rendering, theory, and presentation (Q4; soundness). Experiment 4 uses fixed colors/opacity and camera to render predicted geometry without RGB training; it does not establish new appearance or future-video generation. Experiment 5 likewise transports Stage-1 appearance. Proposition 1 proves conditional zero-loss agreement on supervised matched states, whereas the rollout bound describes possible amplification of local errors by transition Lipschitz factors—not guaranteed long-horizon accuracy (Sec. 3.3; Appendix D). Experiment 1 reports target-relative FDE, rollout gap, and path discrepancy separately, relating empirical outcomes to the roles of anchoring, path supervision, and composition without claiming a numerical validation of the bound. We also clarified the contributions, condensed Discussion and Conclusion, and consolidated appendices.

The revised manuscript provides additional evidence for 4DGS-JEPA's predictive performance on real dynamic scenes, and complements the controlled experiments on temporal composition and geometric consistency.

Thank you for your response. I would retain my original score.

We appreciate your time, efforts and suggestions on improving this work, as well as your earlier recognition of the distinction between predictive accuracy and temporal-path consistency.

Since our previous response, we have further strengthened the presentation and qualitative evidence. The new comparison shows held-out forecasts against ground truth (Fig.6), while the appendix now includes all fourteen HyperNeRF/D-NeRF scene comparisons and multi-timestamp real-scene renderings (Fig.8-11). Together with the controlled ablations, these additions connect Gaussian-native predictive dynamics and path-consistency objectives to observable future-scene behavior.

We believe the main contribution is a shared predictive framework that learns explicit Gaussian scene dynamics through temporally compositional latent transitions, grounded geometric decoding, and hybrid correspondence. We respectfully hope the expanded evidence and clearer presentation will help establish the significance of this approach beyond the remaining experimental limitations.

Reviewer_G8VmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

4DGS-JEPA predicts dynamic Gaussians from past observations. It employs JEPA to predict future latent states and encourages direct and recursively composed predictions of the same future endpoint to agree in latent space and decoded geometry. Synthetic experiments demonstrate reduced latent and geometric path discrepancies and show future-frame rendering without pixel-level supervision.

Critical strengths
  1. Experiments demonstrate improved prediction accuracy and reduced path discrepancy in both the latent representations and the decoded geometry.
Critical weaknesses
  1. The paper’s practical benefit over prior future-state generation or reconstruction remains unclear.
    • While the paper distinguishes itself through joint-embedding prediction and explicit consistency across temporal prediction paths, the practical benefits of these differences remain insufficiently demonstrated.
    • Downstream evaluation or comparison with Gaussian world models would establishes the benefit of the paper.
  2. The evaluation is too restricted to establish applicability to reconstructed scenes.
    • The forecasting experiments use only 18 Gaussians with known motion groups and fixed persistent correspondence.
    • Held-out future-frame rendering uses fixed RGB colors, opacity, and a fixed camera.
    • Although the authors acknowledged it in the limitations, these restrictions leave effectiveness on real dynamic scenes uncertain.
Questions for authors
  1. What practical benefit does the proposed method offer over prior methods?
    • For example, could the authors provide a future prediction comparison to show whether reduced direct–composed discrepancy also improves prediction accuracy?
    • In particular, could the authors compare extrapolation PSNR with prior works [1,2]?
  2. Could the authors evaluate the method on real dynamic scenes?
    • Such an evaluation would help assess whether the benefits persist with reconstruction errors, estimated motion groups, and uncertain correspondence.

[1] Zhao et al. “GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesis.” SIGGRAPH 2024.

[2] Quan et al. "Particlegs: Learning neural gaussian particle dynamics from videos for prior-free physical motion extrapolation." CVPR 2026.

3 comments

Thank you for recognizing the improvements in predictive accuracy and latent/geometric path consistency. We agree that the submission's controlled experiments alone did not establish practical extrapolation on reconstructed scenes. We have therefore added Experiment 5 (Table 5; Appendix G) and expanded the related-work comparison.

1. Practical benefit and evaluation on real dynamic scenes (Questions 1–2). We now evaluate six real HyperNeRF and eight synthetic D-NeRF scenes, holding out the last 10% and 20% of chronological views, respectively. All methods share a scene-specific Deformable-GS representation fitted for 40k iterations on observed-prefix frames only. We compare (i) direct future-time queries of the frozen Stage-1 representation, (ii) ODE-GS, a continuous-time Gaussian-dynamics baseline, and (iii) 4DGS-JEPA. ODE-GS and ours use similar Stage-2 capacities (3.88M vs. 4.08M parameters), matched 90k-step optimization, three seeds, and the same rasterizer; both propagate Stage-1 appearance/opacity rather than predicting new appearance. Future frames are excluded from fitting, tuning, and checkpoint selection.

Held-out metricDeformable-GS queryODE-GS4DGS-JEPA
HyperNeRF full PSNR ↑18.5619.07±0.0618.73±0.24
HyperNeRF dynamic PSNR ↑12.7412.86±0.0613.19±0.19
HyperNeRF dynamic LPIPS ↓0.51080.49910.4832
D-NeRF full PSNR ↑24.0625.75±1.4926.60±0.09
D-NeRF dynamic PSNR ↑10.5312.14±0.6013.61±0.14
D-NeRF dynamic LPIPS ↓0.52190.43970.3888

Dynamic masks are identical across methods and use held-out targets only at evaluation. Relative to ODE-GS, 4DGS-JEPA improves dynamic PSNR by 0.33 dB on HyperNeRF and 1.47 dB on D-NeRF, with lower dynamic LPIPS on both. ODE-GS nevertheless leads on all HyperNeRF full-image metrics and on D-NeRF full-image SSIM/LPIPS. Thus the demonstrated practical benefit is improved rendered moving-region extrapolation under a shared prefix-fitted Gaussian front-end, not uniform image-quality superiority or a downstream planning result.

2. Does reduced path discrepancy improve forecasting accuracy? We evaluate both target-relative FDE and direct–composed latent discrepancy rather than assuming they are equivalent. In controlled Experiment 1, adding composition to endpoint supervision reduces mean direct FDE 0.03013→0.02905 (3.6%) and CompErr 0.00244→0.00096 (60.6%). At the longest trained horizon (Δ=8), its direct FDE is 0.05687, versus 0.07120 for constant velocity. However, constant velocity has better mean FDE across horizons (0.02484 vs. 0.02905). Adding path and composition together minimizes CompErr (0.00058) but raises mean direct FDE (0.03206). These ablations show that composition can improve accuracy in a particular configuration, but path agreement does not guarantee forecasting accuracy (Table 1; Fig. 2).

3. Suggested references and scope of the comparison. We thank the reviewer for pointing us to GaussianPrediction (Zhao et al., SIGGRAPH 2024) and ParticleGS (Quan et al., CVPR 2026). Both are relevant motion-extrapolation methods and are now included and discussed in Section 2. Their reported PSNR values use different scenes, views, and temporal protocols, so quoting them as a direct ranking would be misleading. We instead report a matched ODE-GS comparison, while acknowledging that a same-protocol evaluation against GaussianPrediction and ParticleGS remains unavailable.

The new HyperNeRF results address the original dependence on 18 procedurally specified Gaussians with known groups and fixed identities by testing extrapolation from representations reconstructed from real observed frames. However, the front-end is fitted to the observed prefix rather than updated online, and robustness to independently corrupted group assignments or uncertain correspondence is not isolated. Hybrid matching under reordering and topology perturbations is assessed separately in Experiment 3. Likewise, Experiment 4's fixed appearance/camera rendering demonstrates renderability of predicted geometry, not future appearance generation.

We hope this revision clarifies the benefit of temporal composition, while providing the requested real-scene extrapolation evidence.

Thank you for the response. However, my concerns remain unresolved.

  • (Minor) While ODE-GS is relevant, selecting it instead of the suggested works is not sufficiently convincing.
  • Emphasizing dynamic-region metrics over full-image metrics is unclear because 4DGS-JEPA explicitly penalizes drift in static regions.
    • ODE-GS outperforms the proposed method on all full-image metrics on HyperNeRF and on SSIM/LPIPS on D-NeRF. This makes the practical advantage of 4DGS-JEPA over prior works unclear.
    • The large standard deviation of ODE-GS on D-NeRF dataset needs further explanation.
  • The reported ODE-GS metrics differs from that in the original paper. Please clarify which differences in the experimental settings lead to this discrepancy.
  • No qualitative video results of future-frame extrapolation on real scenes are provided. Such results are important for evaluating temporal consistency and practical prediction quality.
  • The ablation results in Tables 1 and 2 should be extended to real scenes to validate the practical benefits of the proposed method.

Therefore, I will maintain my initial rating.

Thank you for the further assessment. We understand your remaining concerns about the choice of baseline, practical forecasting quality, seed variability, and the validation of temporal composition on real scenes. We have made additional revisions (see updated rebuttal version v3), particularly by incorporating qualitative future-frame comparisons and multi-timestamp real-scene renderings.

1. Baseline choice and experimental protocol. We selected ODE-GS because it directly models continuous-time dynamics of prefix-fitted Gaussian scenes, making it particularly suitable for comparing predictive transitions. Both methods use the same 40k-iteration Stage-1 reconstruction, comparable Stage-2 capacities (3.88M versus 4.08M parameters), 90k optimization steps, three seeds, and a shared renderer. The reported ODE-GS results are from our controlled rerun, not copied from its original paper. Differences in training and conditioning configurations and metric aggregation mean the numerical results should be interpreted within this matched protocol. We also discuss GaussianPrediction and ParticleGS in Related Work without treating results from different protocols as directly comparable.

2. Dynamic-region versus full-image performance. We agree that both are important. Static-region drift regularization acts on geometry and does not directly optimize full-frame photometric quality; dynamic-region metrics additionally isolate prediction errors in moving objects. On HyperNeRF, 4DGS-JEPA improves dynamic PSNR (13.19 versus 12.86 dB) and dynamic LPIPS (0.4832 versus 0.4991), while ODE-GS performs better on full-image metrics. On D-NeRF, ours improves both dynamic PSNR (13.61 versus 12.14 dB) and full-image PSNR (26.60 versus 25.75 dB), although ODE-GS retains better full-image SSIM and LPIPS. We present both metric families rather than claiming uniform superiority.

3. Seed variability. The D-NeRF results are averaged over three independently trained seeds, with full-image PSNR standard deviations of 1.49 dB for ODE-GS and 0.09 dB for 4DGS-JEPA. The complete qualitative grid includes an ODE-GS failure on bouncingballs in seed 0, illustrating one source of run-dependent behavior. We retain the measured variation rather than excluding difficult cases.

4. New qualitative video evidence. We have added Fig. 6, showing final held-out predictions against ground truth for real HyperNeRF and synthetic D-NeRF scenes, and Figs. 8-9, covering all fourteen evaluated scenes. Figs. 10-11 further show fixed-camera future-time sequences at four timestamps beyond the observed prefix on real scenes. These figures expose both successful forecasts and visible distortions, providing concrete visual evidence of future-scene evolution. The fixed-camera sweeps are qualitative and do not contain ground truth at every displayed timestamp.

5. Contribution of temporal composition. Controlled Experiments 1-2 isolate the objectives: adding latent composition reduces path discrepancy by 60.6% while improving direct FDE by 3.6% in the Endpoint + Composition configuration; explicit geometric composition reduces group-transform path discrepancy by 72.2%. The real-scene comparison evaluates the resulting predictive dynamics against ODE-GS, rather than independently ablating every component on HyperNeRF and D-NeRF.

We believe the contribution extends beyond improved image metrics: 4DGS-JEPA learns reusable Gaussian-native predictive dynamics, explicitly aligns alternative temporal prediction paths, and grounds those predictions in structured 3D geometry without a pixel-level forecasting objective. The controlled ablations and real-scene extrapolation results provide complementary evidence for this approach. We believe this methodological advance, and its demonstrated capabilities, outweigh the remaining evaluation limitations, and respectfully hope you will reconsider the paper's overall contribution.

Comments on the paper

2 comments
AuthorsOpenReview ↗

changes made and manuscript updated (rebuttal version)

We thank all reviewers for their constructive feedback. We have revised the manuscript, and added Experiment 5, a temporal-extrapolation evaluation on Gaussian scenes reconstructed from real and synthetic observations. The principal changes are summarized below; our individual replies address each review in more detail.

1. New Experiment 5: observed-prefix temporal extrapolation (Sec. 5, Table 5; Appendix G). We evaluate six real HyperNeRF scenes and eight synthetic D-NeRF scenes, withholding the final 10% and 20% of chronologically ordered views, respectively. All methods share a scene-specific Deformable-GS Stage-1 representation fitted for 40k iterations using only the observed prefix. We compare direct future-time queries of this frozen model with ODE-GS and 4DGS-JEPA, whose Stage-2 dynamics modules have comparable sizes (3.88M/4.08M parameters), matched 90k-step training budgets, and three random seeds. The methods share a renderer; the learned dynamics extrapolate Gaussian geometry while retaining Stage-1 appearance and opacity. Future frames are excluded from fitting and model selection, and dynamic masks are used only for evaluation. This tests prefix-fitted reconstructed scenes rather than a continuously updated online reconstructor.

2. New quantitative evidence for moving-region extrapolation (Table 5). Relative to ODE-GS, 4DGS-JEPA increases dynamic-region PSNR from 12.86 to 13.19 dB on HyperNeRF and from 12.14 to 13.61 dB on D-NeRF. Dynamic LPIPS also decreases (0.4991 to 0.4832 and 0.4397 to 0.3888, respectively). Full-image PSNR is mixed: 18.73 versus 19.07 dB on HyperNeRF and 26.60 versus 25.75 dB on D-NeRF (4DGS-JEPA versus ODE-GS); ODE-GS also retains better full-image SSIM and LPIPS on D-NeRF. These results extend the empirical evaluation beyond the original procedural Gaussian worlds without claiming uniform image-quality superiority.

3. Clearer distinction between prediction accuracy and temporal consistency (Experiment 1; Table 1, Fig. 2). Adding composition to endpoint supervision reduces CompErr by 60.6% (0.00244 to 0.00096) while improving mean direct FDE by 3.6% (0.03013 to 0.02905). Combining path supervision and composition yields the lowest CompErr (0.00058) and rollout gap (0.00029) but higher direct FDE (0.03206). Constant velocity remains stronger on average on the smooth controlled trajectories; Endpoint + Composition has lower direct FDE at the longest trained horizon, Δ=8 (0.05687 versus 0.07120). The unseen controlled horizons 3 and 6 fall inside the trained range, and Experiment 5 is not a separate real-scene composition ablation.

4. Clearer evidence for geometry, correspondence, and renderability (Experiments 2–4). Adding explicit SE(3) composition reduces direct–composed group-transform discrepancy by 72.2% (0.00724 to 0.00202); the full geometry-composition objective reduces Gaussian-level matching discrepancy from 0.00192 to 0.00130 (Table 2; Fig. 3). In the separate correspondence stress test, Hybrid improves split/merge lineage mass over OT-only (0.98100 to 0.98715) under identity and topology perturbations (Table 3; Fig. 4). This does not establish an end-to-end topology-changing forecasting benefit. The held-out-frame experiment demonstrates renderability of predicted Gaussian geometry without an RGB reconstruction loss, using fixed appearance and camera parameters (Table 4; Fig. 5; Appendix F).

5. Improved related work, theory, and presentation. We added and discussed ODE-GS, together with the reviewers' relevant suggestions GaussianPrediction and ParticleGS (Sec. 2), without treating PSNR numbers from unmatched protocols as direct comparisons. We clarified that temporal composition concerns agreement between different chronological transition paths to the same endpoint, rather than merely adjacent-time smoothing. We also distinguished the conditional zero-loss path-agreement result from the finite-error rollout bound, which characterizes possible error accumulation but does not guarantee long-horizon accuracy (Sec. 3.3; Appendix D). Finally, we refined the abstract and contributions, combined Discussion and Conclusion, consolidated the appendices, and documented the new experiment's protocols and evaluation scope in Appendix G.

We appreciate the reviewers' careful feedback and hope these revisions make the framework, evidence, and contribution clearer.

AuthorsOpenReview ↗

updated manuscript with rendered video evidence

We thank all reviewers for their detailed feedback and continued engagement.

Building on our earlier rebuttal, we have further revised and improved this work to strengthen its methodological explanation, to provide qualitative evidence of future-scene extrapolation, and to clarify the interpretation of the controlled and real-scene experiments. Revisions are highlighted in red.

1. Strengthened the motivation and central contribution (Abstract; Secs. 1-2). We revised the Abstract, Introduction, and Related Work to emphasize that 4DGS-JEPA learns reusable predictive dynamics over explicit Gaussian scene representations, rather than merely reconstructing observed frames or querying fitted deformation fields. The method combines a hierarchical Gaussian-native state, closed horizon-conditioned transitions, temporal composition, selective geometry decoding, and hybrid correspondence.

2. Added qualitative future-frame comparisons (Sec. 5; Fig. 6). We now show final held-out forecasts alongside ground truth and competing predictions for chickchicken (real HyperNeRF) and standup (synthetic D-NeRF). The displayed PSNR values are explicitly identified as single-frame dynamic-region measurements. These examples complement the aggregate results reported in Table 5.

3. Added comprehensive qualitative comparisons across all fourteen scenes (Appendix G; Figs. 8–9). The appendix now includes all six HyperNeRF and eight D-NeRF scenes, comparing direct Stage-1 querying, ODE-GS, 4DGS-JEPA, and the included analytic baselines. We retain difficult cases and visible forecasting failures alongside successful examples, allowing readers to assess scene-dependent performance.

4. Added multi-timestamp real-scene renderings (Appendix G; Figs. 10–11). Fixed-camera render sequences for chickchicken and split-cookie show predicted scene evolution at normalized times 0.91, 0.94, 0.97, and 1.00, all beyond the HyperNeRF training cutoff. These address the request for qualitative temporal extrapolation evidence and make differences in evolving geometry more directly observable.

5. Clarified the matched extrapolation protocol and quantitative results (Sec. 5; Appendix G). Experiment 5 uses a shared 40k-iteration Stage-1 Gaussian reconstruction and comparable Stage-2 dynamics modules trained for 90k steps over three seeds. The ODE-GS results are from our matched rerun rather than its originally published evaluation. Dynamic-region metrics use identical held-out evaluation masks and are reported alongside full-image metrics.

Relative to ODE-GS, 4DGS-JEPA improves dynamic-region PSNR on HyperNeRF (12.86 to 13.19 dB) and D-NeRF (12.14 to 13.61 dB), with lower dynamic LPIPS on both. On D-NeRF, it also improves full-image PSNR (25.75 to 26.60 dB); ODE-GS retains better full-image SSIM/LPIPS on D-NeRF and better full-image results on HyperNeRF. The complete qualitative grids include representative failure cases and illustrate run-dependent behavior.

6. Refined the theoretical and experimental interpretation (Secs. 3–5; Appendix D). We distinguish target-relative accuracy from direct–composed consistency. In controlled ablations, adding temporal composition reduces latent path discrepancy by 60.6% while improving direct FDE by 3.6% for Endpoint + Composition. Explicit geometry composition reduces group-transform disagreement by 72.2%. The theoretical statements distinguish conditional zero-loss path agreement from finite-error accumulation under Lipschitz assumptions. Hybrid matching is evaluated through a separate identity-perturbation stress test.

7. Improved presentation. We retained the ten-page main-paper format, clarified the scope of the qualitative examples, refined the wording of the core contributions, and double checked consistency.

Overall significance. We believe 4DGS-JEPA makes a distinctive contribution by bringing temporally compositional, geometry-grounded joint-embedding prediction to explicit dynamic Gaussian scenes. Unlike reconstruction-focused temporal modeling, the framework learns predictive latent dynamics, aligns alternative chronological routes to the same future state, and translates those predictions into structured Gaussian motion. The theoretical analysis, controlled ablations, matched real/synthetic extrapolation results, and new qualitative evidence together support this modeling perspective. We believe these contributions outweigh the remaining limitations of the experimental validation; we respectfully hope the reviewers/AC would consider the paper's significance in light of the final revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.