- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a Gaussian-Native Joint Embedding Predictive Framework for predicting future states in dynamic scenarios. The method predicts future latent states using Hierarchical Gaussian Representations and a Horizon-Conditioned Transition Model. Additionally, a Temporal Composition Loss and a Geometric Composition Loss are designed to constrain the consistency between direct and recursive prediction results. Furthermore, the method introduces a Hybrid Correspondence mechanism to address uncertainties in the Gaussian primitive identity matching process. This method was experimentally evaluated on controlled synthetic scenes, with its performance validated primarily across four aspects: prediction consistency, geometric prediction accuracy, identity correspondence matching performance, and rendering quality of future frames.
Critical strengths
Compared to directly reconstructing the complete appearance of future scenes, predicting future Gaussian representations is a research direction with clear research motivation and practical significance. The Temporal Composition Objective and Geometric Composition Objective proposed in this paper provide a structured and theoretically grounded regularization method for prediction results across multiple temporal domains, helping to improve consistency among different prediction paths.
The ablation experiments clearly demonstrate that the proposed composition loss effectively reduces discrepancies in results across different prediction paths, thereby enhancing prediction consistency. Furthermore, the paper provides detailed experimental setups and supplementary experimental results, offering strong support for the method’s reproducibility.
Critical weaknesses
The paper relies entirely on synthetic data, with the main experiments involving only 18 Gaussian distributions and 3 predefined motion groups, lacking validation under real-world scenarios and reconstruction noise. Furthermore, the Hybrid mechanism and the prediction network were tested separately, and the effectiveness of the complete framework in actual prediction tasks has not yet been demonstrated.
In Table 1, although Path + Composition reduces CompErr from 0.00244 to 0.00058, Direct FDE actually increases from 0.03013 to 0.03206. In Table 4, simple constant-velocity extrapolation outperforms all learning models. The fundamental differences between this method and existing time-consistency constraints are not sufficiently clear; insufficient qualitative visualizations and overly verbose text also weaken the paper’s clarity and overall contribution.
Questions for authors
Q1. The current unseen horizons (3 and 6) effectively test interpolation rather than extrapolation beyond the training range. Could the authors evaluate prediction performance further beyond the training horizons, e.g., 12 or 16 steps ahead, and report both FDE and composition error for those settings? Comparisons with constant-velocity, constant-acceleration, and a simpler learned predictor would also help clarify the actual forecasting benefit of the proposed method.
Q2. Experiment 3 evaluates the matching mechanism in isolation, so its contribution to end-to-end forecasting accuracy remains unverified. Could the authors provide a small predictive experiment that involves Gaussian birth, death, split, or merge events, and compare the performance of fixed correspondence, OT-only, and the proposed Hybrid correspondence in that end-to-end setting?
Q3. What is the empirical benefit of the three-level Gaussian hierarchy (scene, motion-group, Gaussian) over a simpler capacity-matched encoder without the hierarchical structure? It would also be helpful if the authors report the model’s sensitivity to noisy Gaussian positions or incorrect motion-group assignments.
Q4. The current figures do not adequately illustrate the practical effect of the proposed consistency objectives. Could the authors provide multi-timestamp comparisons of predicted geometry against ground truth and simple baselines, preferably with visualizations of motion trajectories and corresponding error profiles?
3 comments
Post-Rebuttal Assessment
Thank you for the detailed response and additional experiments. I appreciate the extension to reconstructed real scenes. However, the improvements remain inconsistent across different settings. In particular, the full-image PSNR on D-NeRF decreases from 26.60 to 25.75 dB despite improvements in dynamic-region PSNR.
Moreover, the additional experiments do not sufficiently establish the independent contribution of temporal composition to forecasting accuracy. The end-to-end effectiveness of Hybrid correspondence and the necessity of the hierarchical architecture also remain unverified.
Overall, while the revision partially addresses my concerns, the evidence is still insufficient to demonstrate consistent predictive improvements and fully justify the proposed framework. Therefore, I maintain my original rating of 2 (Weak Rejection).
Thank you for your further assessment, and for recognizing the extension to reconstructed real scenes. We understand your concerns about inconsistent improvements and the independent contributions of the proposed components.
First, we would like to clarify one numerical point: the D-NeRF full-image PSNR comparison is 26.60 dB for 4DGS-JEPA versus 25.75 dB for ODE-GS, so our method improves PSNR by 0.85 dB, rather than decreasing it. It also improves dynamic-region PSNR by 1.47 dB and reduces dynamic LPIPS from 0.4397 to 0.3888. We agree that ODE-GS retains better full-image SSIM and LPIPS on D-NeRF, as well as stronger full-image results on HyperNeRF.
Regarding temporal composition, the controlled ablations distinguish prediction accuracy from route consistency. Adding composition to endpoint supervision lowers direct FDE from 0.03013 to 0.02905 while reducing latent path discrepancy from 0.00244 to 0.00096. Adding both path supervision and composition achieves the lowest path discrepancy (0.00058), although direct FDE is higher. Thus, the experiments establish a measurable consistency benefit and an accuracy improvement for a specific configuration, without assuming that lower path discrepancy always means lower forecasting error.
Regarding the other components, Experiment 2 isolates geometry-level composition, reducing group-transform path disagreement by 72.2%, while Experiment 3 evaluates hybrid correspondence under reordering and topology changes, improving split/merge lineage recovery from 0.98100 to 0.98715 relative to OT-only. These experiments establish the behavior of the individual mechanisms; they do not constitute a capacity-matched hierarchy ablation or an end-to-end hybrid-correspondence forecasting comparison.
We have also strengthened the qualitative evaluation. The new Fig. 6 compares held-out forecasts with ground truth, Figs. 8-9 show all fourteen reconstructed scenes, and Figs. 10-11 provide multi-timestamp real-scene renderings. These complement the horizon-wise prediction and consistency profiles already reported.
We believe the central contribution is the integration of Gaussian-native predictive states, temporally compositional transitions, geometry-grounded decoding, and identity-aware correspondence into a JEPA-style predictive framework. The controlled studies establish the roles of these mechanisms, while the new reconstructed-scene experiments demonstrate their relevance to future-time Gaussian forecasting. We believe this combined methodological and empirical contribution is useful, despite the remaining component-level validation gaps, and respectfully hope you will reconsider its significance.
Thank you for recognizing the motivation for Gaussian-native prediction, and the temporal/geometric composition ablations. We have added Experiment 5 (Table 5; Appendix G), expanded the empirical comparison, and clarified the scope of the controlled results. We address the weaknesses and four questions below.
Regarding the main weaknesses. Experiment 5 extends evaluation beyond the original 18-Gaussian, three-group synthetic setting to reconstructed real scenes. The controlled ablations distinguish predictive accuracy from path consistency, while the correspondence tests remain separate from forecasting. We have also clarified the methodological novelty and improved the manuscript's organization and presentation.
Q1. Extrapolation, baselines, and accuracy versus consistency. Experiment 1 trains on horizons {1,2,4,8}; the unseen {3,6} horizons test generalization within that range, not extrapolation to 12 or 16 steps. We have not run the requested Δ=12/16 FDE and CompErr or constant-acceleration comparisons. The Endpoint-only variant is a simpler learned-loss baseline, though not a separate capacity-matched predictor.
To assess extrapolation beyond observed scene times, new Experiment 5 uses six real HyperNeRF scenes (final 10% withheld) and eight synthetic D-NeRF scenes (final 20% withheld). ODE-GS and 4DGS-JEPA share a prefix-fitted Deformable-GS front-end, use comparable Stage-2 capacities (3.88M/4.08M parameters), matched 90k-step training, three seeds, and the same rasterizer. Relative to ODE-GS, dynamic-region PSNR improves 12.86→13.19 dB on HyperNeRF and 12.14→13.61 dB on D-NeRF; full-image PSNR is 18.73 vs. 19.07 dB and 26.60 vs. 25.75 dB, respectively. Other full-image metrics are mixed (Table 5).
We agree with the Table 1 trade-off: Endpoint + Path + Composition lowers CompErr 0.00244→0.00058 but raises mean direct FDE 0.03013→0.03206. Endpoint + Composition, by contrast, lowers both FDE 0.03013→0.02905 and CompErr 0.00244→0.00096. Constant velocity is stronger on average for these smooth trajectories (0.02484 vs. 0.02905 FDE) and in one-step rendering (60.48 vs. 48.24 dB); at the longest trained horizon (Δ=8), Endpoint + Composition does better in direct FDE (0.05687 vs. 0.07120). Thus improved path consistency does not automatically imply better forecasting at every horizon (Table 1; Fig. 2).
Q2. End-to-end correspondence under topology change. Experiment 3 tests index, OT-only, and Hybrid matching under permutations, splits/merges, birth/death, and identity dropout. Hybrid improves split/merge lineage mass over OT 0.98100→0.98715 and reduces induced geometry discrepancy 0.05010→0.04906 (Table 3; Fig. 4; Appendix F). OT and Hybrid both attain birth/death F1 1.0 in this test. These are standalone correspondence results; forecasting FDE under birth/death or split/merge with fixed/OT/Hybrid matching has not been evaluated end-to-end. Experiment 5 likewise does not integrate learned residual-OT correspondence.
Q3. Hierarchy and robustness. The scene/group/Gaussian hierarchy and bottom-up pooling are specified in Sec. 3 and Appendix B, but the manuscript contains no capacity-matched flat-encoder ablation; hence we cannot attribute an isolated gain to the hierarchy. Controlled forecasting uses known motion groups. Experiment 3 includes positional noise (standard deviation 0.025), scale/semantic perturbations, and missing IDs for matching, not forecasting. Sensitivity of predicted FDE to noisy input positions or incorrect group assignments remains unmeasured.
Q4. Visualizations and distinction from temporal smoothing. The manuscript reports horizon-wise FDE and CompErr curves (Fig. 2), group-transform and Gaussian-discrepancy profiles (Fig. 3), correspondence stress-test curves (Fig. 4), and a held-out rendering alongside ground truth and analytic baselines (Fig. 5); Appendix F compares direct and composed rendering routes. These provide error profiles and visual evidence, although a multi-timestamp trajectory montage against ground truth has not been added. Methodologically, our composition loss compares different chronological decompositions reaching the same endpoint through the same closed, horizon-conditioned transition, anchored by future targets, and extends agreement to decoded SE(3) transforms and Gaussian geometry. This differs from simply enforcing adjacent-time smoothness; we do not claim temporal regularization itself is new.
We have also shortened the main exposition and consolidated the appendices. Together, we hope the revision strengthens the evidence for temporal and geometric consistency and extends the evaluation to extrapolation on reconstructed real and synthetic scenes.