acceptodds
Under review as a conference paper at ICLR 2027

MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Coupling

Abstract

Modeling spatiotemporal coupling is a key challenge in building physical intelligence across scales, from microscopic to macroscopic. Existing models capture such structure broadly through physics-motivated dynamical formulations or learning-motivated architectures. The former provide stronger priors but may constrain flexibility, whereas the latter are more flexible but leave the spatiotemporal coupling largely implicit. We therefore seek an approach that combines flexible learning with an explicit geometric bias for jointly modeling time and space. To this end, we propose Minkowski Positional Encoding (MinkowskiPE), which uses joint temporal and spatial coordinates to parameterize Lorentz transformations applied to query and key features. With MinkowskiPE, the query–key attention score depends on position only through the relative spacetime displacement between the two tokens and is therefore invariant to global translation of the coordinates. This paradigm retains the standard dot-product attention interface and remains compatible with efficient attention implementations. We evaluate MinkowskiPE on microscopic molecular dynamics and macroscopic video prediction tasks, achieving the best results on all nine multi-trajectory molecular evaluations and reducing KTH video-prediction MSE by 9.9% relative to the best compared baseline while using roughly one-tenth as many parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
spatiotemporal modeling, Transformers, positional encoding, Minkowski geometry, Lorentz transformations, molecular dynamics, video prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FE7mOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MinkowskiPE, a positional encoding method for spatiotemporal modeling based on Minkowski geometry and Lorentz transformations. The proposed approach applies Lorentz transformations to query and key token features in the attention mechanism, where the transformation parameters are derived from learnable projections of joint spatiotemporal coordinates. With the algebraic properties of Lorentz transformations, the proposed MinkowskiPE enables attention to depend on relative position desplacements while preserving translation invariance. Experiments on microscopic molecular dynamics prediction and macroscopic video prediction demonstrate its effectiveness across different spatiotemporal modeling tasks.

Critical strengths
  • The proposed MinkowskiPE provides a new positional encoding mechanism with translation invariance. The proposed formulation is mathematically rigorous and supported by clear theoretical derivations.

  • This work conducts diverse experimental evaluations, including molecular dynamics in the domain of AI4Science, and video prediction on KTH video prediction benchmark.

Critical weaknesses
  • The work aims to combine physical inductive priors with the flexibility of learnable biases in spatiotemporal modeling, particularly in the positional encoding. However, the connection between Lorentz transformations and meaningful physical priors remains unclear. Specifically, Lorentz transformations are applied to learned query and key features (which is also semantics) rather than physical spacetime coordinates, while temporal and spatial coordinates are combined through learnable linear projections without explicitly preserving their distinct physical roles.

  • The technical contributions and performance improvements are not sufficiently validated or compared against relevant baselines. I summarize my concerns regarding the experimental evaluation in the following points according to the instructions:

    • Although Table 4 provides a controlled ablation against a model without positional encoding, the paper lacks comparisons with strong PE baselines, such as multidimensional RoPE and learnable RoPE, under the same architecture and training settings. Therefore, it remains unclear whether the improvements stem from the proposed Lorentz transformations or simply from introducing learnable relative positional encodings.

    • In Tables 1–3, the authors primarily compare against methods with substantially different architectures, such as GNNs and ODE/SDE-based models, making it difficult to determine whether the reported improvements stem from MinkowskiPE or the underlying Transformer architecture. Although Table 4 provides a No-PE ablation, the improvements are relatively modest, and comparisons with stronger positional encoding baselines are missing. Therefore, the experiments do not sufficiently isolate or validate the contribution of the proposed MinkowskiPE.

Questions for authors
  • Although the authors evaluate the proposed PE approach on two substantially different prediction tasks, both tasks essentially focus on temporal forecasting at a high level. It remains unclear why broader spatiotemporal tasks, such as 3D generation or temporal video understanding, are not considered to further validate the general applicability of the proposed method.

  • The other concerns are discussed in the Weakness Part.

Reviewer_2nktOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Minkowski Positional Encoding (MinkowskiPE) for joint spatiotemporal modeling in Transformers. The method projects spatiotemporal coordinates into scalar values to parameterize Lorentz transformations applied to queries and keys, and uses an indefinite metric to ensure that positional dependencies in the dot-product attention scores are determined solely by relative spacetime displacements. This construction is compatible with standard dot-product attention. The authors evaluate the method on protein–ligand trajectory prediction and video prediction, and demonstrate its benefits through controlled ablation studies that compare the same architecture with and without the proposed encoding.

Critical strengths
  1. The method is simple, and its key properties are clearly derived. Equations (7)–(12) explain how token-wise transformations achieve relative positional dependence while preserving the standard attention interface, making the method relatively easy to implement.

  2. The method achieves consistent gains across two substantially different domains: molecular trajectory prediction and video prediction. The controlled ablation in Table 4 further supports the effectiveness of the positional encoding itself, suggesting that the improvements are not solely attributable to differences between prediction architectures.

Critical weaknesses
  1. Potential test-set leakage in model selection. Appendix B.3 states that checkpoints are selected by test-set MSE. If accurate, this compromises the KTH results as an unbiased estimate of generalization. Please clarify the protocol and report results with validation-based checkpoint selection.
  2. Insufficient comparison with existing positional encodings. Table 4 compares MinkowskiPE only against No PE, without matched comparisons to RoPE, multidimensional RoPE, or video-specific encodings. Thus, it does not establish whether the Lorentzian construction offers an advantage over simpler positional mechanisms.
Questions for authors
  1. Appendix B.3 states that KTH checkpoints are selected based on the lowest test-set MSE. Could the authors instead select checkpoints using an independent validation set and report results on a held-out test set? Do the reported performance gains remain after this reevaluation?

  2. Could the authors directly compare MinkowskiPE with RoPE, multidimensional relative positional encodings, and other relevant geometric positional encodings under the same architecture and training protocol? These comparisons are important for determining whether the gains stem specifically from Minkowski geometry.

Reviewer_pUcMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MinkowskiPE, a positional encoding that uses joint spatial and temporal coordinates to parameterize Lorentz transformations of query and key features. The construction provides relative positional interactions within standard attention. Experiments are conducted on molecular dynamics and video prediction.

Critical strengths
  1. The formulation is compact, with a clear derivation of its relative-position property.

  2. Evaluation on two different domains supports its applicability to different spatiotemporal inputs.

  3. The controlled ablation shows improvements over NoPE in both domains.

Critical weaknesses
  1. Comparisons with other positional encodings are missing. The NoPE ablation alone is insufficient to establish the benefit of the Lorentz-based design.

  2. Checkpoints are selected using test-set MSE, which may introduce optimistic bias in the reported results.

  3. Evaluation is limited to predicting 20 future frames. Longer-horizon prediction or rollout experiments would help assess whether the gains persist.

Questions for authors

See above.

Comments on the paper

3 comments

Dear Reviewer FE7m,

Thank you for your follow-up. We respectfully disagree with your assessment that our PE baseline comparisons remain insufficient.

1. Regarding your original concern. Your initial review stated that

the paper lacks comparisons with strong PE baselines, such as multidimensional RoPE and learnable RoPE, under the same architecture and training settings.

Your initial review mentioned only two broad PE categories without specific references. We have fully addressed both and gone beyond your original request. Our controlled comparisons cover six PE baselines, with MinkowskiPE achieving the best results across all eight metrics on two tasks. Our experiments include widely recognized methods such as M-RoPE and VideoRoPE, with over 5,000 combined citations. We acknowledge that other PE methods exist, but we do not believe that a comprehensive baseline comparison should include every existing method. Such a requirement is neither practical nor reasonable.

2. Regarding your second review. Our task is video prediction, which is distinct from video understanding and video generation. You question the relevance of M-RoPE and VideoRoPE because they were developed for video understanding, yet recommend RIFLEx and MultiShotMaster, which target video generation, and ViewRope, which focuses on camera-controlled video world modeling. These methods were not specifically designed for the video prediction task either.

The following table compares the original applications and academic impact of the PE methods discussed in your reviews and our rebuttal.

StagePE MethodTaskCitations
FE7m's 1st ReviewMultidimensional RoPE---- (Reference Missing)
Learnable RoPE---- (Reference Missing)
Our 1st RebuttalStandard RoPEGeneral8456
Multidimensional RoPE (3D&4D)General8456 (Reimplemented from RoPE)
Learnable RoPEGeneral8456 (Reimplemented from RoPE)
VideoRoPEVideo Understanding82
M-RoPEVideo Understanding4967
FE7m's 2nd ReviewRIFLExVideo Generation72
MultiShotMasterMulti-shot Video Generation37
ViewRopeCamera-Controlled Video World Modeling17

We maintain that our additional experiments have directly addressed your original concerns and provide sufficient evidence for the effectiveness of MinkowskiPE.

In the absence of specific references in your initial review, we prioritized six widely cited and representative PE baselines and conducted controlled comparisons against them. We believe these comparisons provide sufficient empirical evidence to demonstrate the effectiveness of MinkowskiPE.

While we acknowledge that further baselines could always be considered, the possibility of additional comparisons does not, in itself, render the existing evidence insufficient.

Moreover, introducing new baseline requirements just two days before the rebuttal deadline leaves limited time for rigorous implementation and evaluation. We will make every effort to accommodate these requests, but we respectfully disagree that our current comparisons are insufficient or that such late-stage requests should serve as grounds for that assessment.

Best regards,
The Authors

Thank you for your clarification. I appreciate the authors' efforts in conducting additional controlled experiments and acknowledge that these comparisons strengthen the empirical evaluation.

To clarify, I was not requesting additional experiments or introducing new baseline requirements at this stage. The methods mentioned in my follow-up were intended as examples to illustrate that relevant PE designs for 3D spatial modeling and video generation/prediction have not been fully considered. As the authors also noted, M-RoPE in Qwen was primarily designed for multimodal understanding rather than generation or prediction, making it less directly relevant to the setting considered in this paper.

I do appreciate the controlled comparisons with RoPE and its variants for the authors' efforts. However, I would distinguish between controlled comparisons and comparisons with task-relevant methods. While these baselines are useful, several are straightforward RoPE extensions or methods primarily developed for video understanding, rather than recent PE approaches tailored to spatiotemporal generation or prediction.

Additionally, regarding your concern about the timing of my follow-up, I would like to clarify that I was simply expressing my reviews as a responsible reviewer, rather than requesting additional experiments at this late stage. My intention was to explain which concerns remain partially unresolved and to always welcome any further dicssuion along with other reviewers, not to impose new experimental requirements.

Dear Area Chair,

We would like to provide a brief summary of our rebuttal and the additional evidence presented during the discussion period.

The reviewers recognized the mathematical clarity and simplicity of our MinkowskiPE, as well as its effectiveness across molecular dynamics and video prediction. Their main concerns focused on experimental validation and the physical interpretation of our method. In response, we have made the following improvements:

  1. Controlled PE comparisons (Reviewers pUcM, 2nkt, and FE7m). We conducted controlled comparisons with seven PE baselines, including an adapted RIFLEx. Our method achieved the best performance across all eight evaluation metrics on two tasks.

  2. Validation-based evaluation (Reviewers pUcM and 2nkt). We conducted additional experiments with independent validation-based checkpoint selection, confirming that our method maintains strong performance under the revised evaluation protocol.

  3. Longer-horizon prediction (Reviewer pUcM). We extended molecular dynamics prediction from 20 to 60 future frames and retrained all compared methods under the same protocol. Our method achieved the best performance across all three metrics.

  4. Physical interpretation (Reviewer FE7m). We clarified the mathematical motivation for using Lorentz transformations as a representation-level inductive bias, including the derivation of relative positional interactions and the role of learnable coordinate projections.

These additional experiments and clarifications have been documented in our rebuttal and incorporated into the revised manuscript. We believe that the mathematically grounded formulation of MinkowskiPE, together with the strengthened experimental evidence across two distinct domains, constitutes a meaningful contribution to geometric representation learning and spatiotemporal modeling.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.