- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MinkowskiPE, a positional encoding method for spatiotemporal modeling based on Minkowski geometry and Lorentz transformations. The proposed approach applies Lorentz transformations to query and key token features in the attention mechanism, where the transformation parameters are derived from learnable projections of joint spatiotemporal coordinates. With the algebraic properties of Lorentz transformations, the proposed MinkowskiPE enables attention to depend on relative position desplacements while preserving translation invariance. Experiments on microscopic molecular dynamics prediction and macroscopic video prediction demonstrate its effectiveness across different spatiotemporal modeling tasks.
Critical strengths
-
The proposed MinkowskiPE provides a new positional encoding mechanism with translation invariance. The proposed formulation is mathematically rigorous and supported by clear theoretical derivations.
-
This work conducts diverse experimental evaluations, including molecular dynamics in the domain of AI4Science, and video prediction on KTH video prediction benchmark.
Critical weaknesses
-
The work aims to combine physical inductive priors with the flexibility of learnable biases in spatiotemporal modeling, particularly in the positional encoding. However, the connection between Lorentz transformations and meaningful physical priors remains unclear. Specifically, Lorentz transformations are applied to learned query and key features (which is also semantics) rather than physical spacetime coordinates, while temporal and spatial coordinates are combined through learnable linear projections without explicitly preserving their distinct physical roles.
-
The technical contributions and performance improvements are not sufficiently validated or compared against relevant baselines. I summarize my concerns regarding the experimental evaluation in the following points according to the instructions:
-
Although Table 4 provides a controlled ablation against a model without positional encoding, the paper lacks comparisons with strong PE baselines, such as multidimensional RoPE and learnable RoPE, under the same architecture and training settings. Therefore, it remains unclear whether the improvements stem from the proposed Lorentz transformations or simply from introducing learnable relative positional encodings.
-
In Tables 1–3, the authors primarily compare against methods with substantially different architectures, such as GNNs and ODE/SDE-based models, making it difficult to determine whether the reported improvements stem from MinkowskiPE or the underlying Transformer architecture. Although Table 4 provides a No-PE ablation, the improvements are relatively modest, and comparisons with stronger positional encoding baselines are missing. Therefore, the experiments do not sufficiently isolate or validate the contribution of the proposed MinkowskiPE.
-
Questions for authors
-
Although the authors evaluate the proposed PE approach on two substantially different prediction tasks, both tasks essentially focus on temporal forecasting at a high level. It remains unclear why broader spatiotemporal tasks, such as 3D generation or temporal video understanding, are not considered to further validate the general applicability of the proposed method.
-
The other concerns are discussed in the Weakness Part.
Dear Reviewer FE7m,
Thank you for your follow-up. We respectfully disagree with your assessment that our PE baseline comparisons remain insufficient.
1. Regarding your original concern. Your initial review stated that
Your initial review mentioned only two broad PE categories without specific references. We have fully addressed both and gone beyond your original request. Our controlled comparisons cover six PE baselines, with MinkowskiPE achieving the best results across all eight metrics on two tasks. Our experiments include widely recognized methods such as M-RoPE and VideoRoPE, with over 5,000 combined citations. We acknowledge that other PE methods exist, but we do not believe that a comprehensive baseline comparison should include every existing method. Such a requirement is neither practical nor reasonable.
2. Regarding your second review. Our task is video prediction, which is distinct from video understanding and video generation. You question the relevance of M-RoPE and VideoRoPE because they were developed for video understanding, yet recommend RIFLEx and MultiShotMaster, which target video generation, and ViewRope, which focuses on camera-controlled video world modeling. These methods were not specifically designed for the video prediction task either.
The following table compares the original applications and academic impact of the PE methods discussed in your reviews and our rebuttal.
We maintain that our additional experiments have directly addressed your original concerns and provide sufficient evidence for the effectiveness of MinkowskiPE.
In the absence of specific references in your initial review, we prioritized six widely cited and representative PE baselines and conducted controlled comparisons against them. We believe these comparisons provide sufficient empirical evidence to demonstrate the effectiveness of MinkowskiPE.
While we acknowledge that further baselines could always be considered, the possibility of additional comparisons does not, in itself, render the existing evidence insufficient.
Moreover, introducing new baseline requirements just two days before the rebuttal deadline leaves limited time for rigorous implementation and evaluation. We will make every effort to accommodate these requests, but we respectfully disagree that our current comparisons are insufficient or that such late-stage requests should serve as grounds for that assessment.
Best regards,
The Authors