acceptodds
Withdrawn from ICLR 2027

Observation-Adaptive Spatiotemporal Positional Encoding: A State-Space Perspective

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning to perceive and predict the evolving physical world is central to physical intelligence, but developing intrinsic spatiotemporal perception remains a key challenge. Transformers represent position through positional encoding (PE). However, temporal coordinates in PE typically use raw time indices, which specify when observations are sampled but do not reflect how much the system state changes between them. To address this limitation, we propose SpatiotemporalPE, which combines spatial structure and temporal evolution. Using a state-space formulation, we relate changes in the underlying state to elapsed time. Specifically, we estimate local state variation from observation differences through averaging and noise correction to construct Observation-difference Coordinate. We instantiate SpatiotemporalPE for molecular dynamics simulation and video understanding. SpatiotemporalPE achieves the best MAE on 8 of 10 molecular dynamics tasks and reduces average MAE by 14.2% relative to the best baselines. SpatiotemporalPE improves overall accuracy from 43.85% to 44.63% in Video experiment. These results support observation-adaptive coordinates as a cross-domain inductive bias for spatiotemporal modeling.

Primary area:
learning on time series and dynamical systems
Keywords:
positional encoding, spatiotemporal representation learning, state-space modeling, molecular dynamics, video modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z6DmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

SpatiotemporalPE replaces raw time indices with a coordinate that reflects how much the observed system changes over time. The coordinate is estimated from observation differences using local averaging and noise correction, and then used in rotary positional encoding. The authors evaluate the method on molecular dynamics prediction and video understanding.

Critical strengths
  • The motivation is well founded: the same time interval can contain very different amounts of change, and the paper turns this observation into a concrete modification of positional encoding.
  • Evaluating the idea on both molecular dynamics and video is useful. The simple cumulative-coordinate baselines clarify the contribution of the local, noise-aware estimator, and the video comparison holds the backbone and training setup fixed.
Critical weaknesses
  • The molecular dynamics experiments lack the most direct control: the same model and training setup with ordinary time indices. Neither published comparisons nor SimpleV1/SimpleV2 fill this gap; the latter support the local estimator within the same architecture but, since they also use observation differences, say little about the coordinate itself. The video comparison is better controlled, but the 0.78-point gain is reported without variance across training runs and should be treated as suggestive.
  • The construction of f_k is underspecified for autoregressive use. The local window W_k is centered at t_k, and λ is estimated from the whole observation sequence, so the coordinate at a given history position depends on later positions in the input window. This does not mean the model uses future ground-truth observations; the coordinate can be recomputed from the available history and generated predictions. But the paper does not say how the window is truncated, how λ is updated, or whether f is recomputed after each generated snapshot. These choices affect both the autoregressive molecular dynamics rollout and streaming use; without them, the rollout is difficult to reproduce or audit.
Questions for authors
  1. Please compare against ordinary time indices with the same molecular dynamics backbone and training setup.
  2. Please specify how the window is truncated and λ updated during autoregressive rollout, and whether f is recomputed after each generated snapshot.
  3. Please fix the unresolved cross-reference in Section 3.2 (“as illustrated in ??”) and reconcile the 4K6W MAE, reported as 2.670 in Table 1 and 2.702 in Table 7.
Reviewer_JTAVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new positional encoding for spatiotemporal tasks, SpatiotemporalPE. The key contribution of SpatiotemporalPE is that instead of providing raw temporal indices as the positions used for embeddings, it constructs an observation-difference version, where they measure the difference between tokens and add noise, to measure how much variation has occurred between two tokens, and use this difference-weighted time as the new embedding. They validate this new positional embedding in two domains: molecular dynamics and video understanding.

Critical strengths

The topic of encoding spatiotemporal representations well is timely and important to the research community. Tackling both molecular dynamics and video understanding show diversity in the methods validation.

Critical weaknesses

Limited baselines and confidence intervals for evaluations. The evaluation spans two distinct domains, which is a strength, but provides limited evidence that the proposed encoding yields robust improvements across model configurations. Although the molecular experiments cover three dataset sizes and report using three random seeds, the tables omit statistical significance estimates. And the video improvement is only 0.78%, without confidence intervals. Consequently, it is difficult to assess whether the improvement is reproducible. Evaluation across additional backbones or model sizes, together with uncertainty estimates and controlled comparisons against raw-time encoding, would strengthen the claim of a general spatiotemporal inductive bias. Furthermore, there are missing baselines for the video examples. For example, no-positional encoding, standard PEs other than VideoROPE, etc should be included. At a minimum, these should include: (i) no positional embeddings (ii) spatial only PE (iii) Sinusoidal temporal PE and (iv) learned positional embeddings. Finally, to fairly compare results in the molecular setting, the missing comparison is the same molecular transformer with standard time PE and the proposed adaptive time PE, while keeping everything else identical.

The paper does not sufficiently establish why this particular prior should replace raw temporal information. It seems like intuitively we would want absolute or raw or relative coordinates to be presented to the model, and have the model need to learn what large or small variations look like in adjacent tokens, rather than inject inductive priors about what large variations look like a priori. The authors should test whether the statistic actually tracks physical change and thus show whether their assumption and implementation (squared difference and gaussian noise) are valid.

The figures are hard to follow and generally low quality. For example in figure one, it never explains to the reader what "Observational-Differences" are, and the two "spatial structure" boxes contain the exact same images with arrows inbetween without further explanation. In Figure 2, the multi-colored arrows are very difficult to follow and parse as they wrap around the image in two directions. alpha_k I think should be a function (i.e., a box that t goes into) as it "transforms the raw time interval into the coordinate increment", and the red arrows are never defined in the legend. There are also several writing mistakes (Line 23 "in Video experiment", Line 144 broken reference)

Questions for authors

How does SpatiotemporalPE perform when holding everything else constant on the molecular dynamics task?

How does SpatiotemporalPE perform compared against basic PE's like (i) no positional embeddings (ii) spatial only PE (iii) Sinusoidal temporal PE and (iv) learned positional embeddings when also accounting for statistical significance?

Does the PE proposed actually track physical changes in videos/molecules?

Reviewer_KA5cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduce SpatiotemporalPE, rotary positional encoding where temporal coordinate is replaced by an “Observation-difference Coordinate” f_k. Each observation is treated as a noisy measurement of a latent state (Ok=Sk+ϵkO_k = S_k + \epsilon_k), and the state increment is assumed to satisfy ΔSk∼N(0,λΔfkI)\Delta S_k \sim \mathcal{N}(0, \lambda \Delta f_k I). From this authors derive per-step scale αk=ReLU(v2‾Wk−2rk)/(λΔtk)\alpha_k = \mathrm{ReLU}(\overline{v^2}_{W_k} - 2r_k)/(\lambda \Delta t_k) using window-averaged squared observation differences. The noise variance rkr_k is estimated with a wavelet/MAD procedure and λ\lambda with a sequence-level median. The resulting coordinate is train-free and plugged into RoPE in two ways: as (x, y, z, f) for atom tokens in a protein–ligand MD Transformer, and in place of the frame index inside VideoRoPE’s layout for a video LLM.

Critical strengths
  1. Simple, training-free, and architecture-compatible : The coordinate is computed from the input sequence alone and changes only the position indices fed to RoPE. It can therefore be dropped into existing models; in the video experiment, only VideoRoPE’s temporal index is swapped. If the gains hold up, this is a cheap inductive bias with broad applicability, which is the main argument for the paper’s significance.
  2. Consistent MD gains at scale : The improvements hold on every metric across all three multi-trajectory splits, including MISATO-All (~13k training complexes), not only on the small single-trajectory tasks. This is the strongest empirical evidence in the paper and the main reason I do not rate it lower.
Critical weaknesses
  1. The experiments do not isolate the effect of the proposed coordinate.
  • MD has no same-architecture control. The method is compared only against baselines with different architectures (NeuralMD, BioMD, etc.); there is no run of the same Transformer with raw-time (x,y,z,t) RoPE. The only coordinate ablations are SimpleV1/V2 and noise correction. The reported gains could therefore come from the Transformer with 3D-coordinate RoPE rather than from ff.
  • High configuration sensitivity :Table 8 report MAE 3.893 / Stability 78.34 for the full method, versus 3.138 / 83.49 in Tables 1/3 for apparently the same method under a “different configuration”. A ~24% MAE swing across configurations is larger than most of the reported gains.
  • The video gain is within likely noise. +0.78 points on Video-MME is roughly 21 questions out of 2,700, assuming the full set was evaluated. It comes from a single fine-tuning run with no seeds or confidence intervals. The backbone is not named and L=2L=2 is undefined. Temporal Perception, the category the method most directly targets, drops by 3.64 points.
  • Numerical inconsistencies. 4K6W MAE is 2.670 in Table 1 but 2.702 in Table 7. By mean MAE in Table 1, the strongest baseline is NeuralMD-ODE (3.635), not BioMD (3.657), so the abstract’s “relative to the best baselines” should read about 13.8%.
  1. Limited novelty and missing positioning.
  • The construction reduces to a realized-variance clock. Substituting Eq. (12) into Eq. (8) gives Δfk=αkΔtk=ReLU(γk)/λ\Delta f_k = \alpha_k \Delta t_k = \mathrm{ReLU}(\gamma_k)/\lambda, so Δtk\Delta t_k cancels and raw time enters only through the global median λ\lambda. Under uniform sampling, as in both experiments, ff is just the normalized cumulative sum of window-averaged, noise-debiased squared increments. This is essentially a realized-variance (“intrinsic time”) clock with the classical microstructure-noise bias correction (cf. two-scales realized volatility, Zhang, Mykland & Aït-Sahalia, JASA 2005).
  • The theoretical framing adds little. The “maximum-likelihood” estimator in Appendix B is a single-sample moment match. The state-space/Kalman framing involves no filtering or state inference.
  • The difference from the simple baseline is small. Compared with SimpleV2 (Δfk=vk2/λ\Delta f_k = v_k^2/\lambda), the method differs only by windowing, subtracting 2rk2r_k, and the ReLU.
  • The closest ML work is neither discussed nor compared: a) data-dependent position encodings that accumulate input-dependent increments or transforms (CoPE; PaTH, NeurIPS 2025; Forgetting Transformer, ICLR 2025; TAPE, ICML 2025); b) temporal scaling and frequency design for video RoPE (HoPE, NeurIPS 2025; MHRoPE/MRoPE-I, ICLR 2026); c) content-aware temporal token lengths derived from frame differences (Run-Length Tokenization, NeurIPS 2024); d) continuous and multi-dimensional RoPE (STRING, ICML 2025; RoMAE, NeurIPS 2025).
  • Without these comparisons, it is hard to tell whether the hand-crafted estimator offers aything over a learned increment such as Δfk=softplus(MLP(ΔOk))\Delta f_k = \mathrm{softplus}(\mathrm{MLP}(\Delta O_k)).
Questions for authors
  1. What are the results of the same MD architecture with raw-time (x,y,z,t)(x,y,z,t) RoPE (and ideally with no temporal axis), under the Table 1/2 protocols?
  2. What exactly differs between the Table 8 configuration and that of Tables 1/3, and why does the full method’s MAE change from 3.138 to 3.893?
  3. In MD, is OkO_k the raw atomic coordinates or the learned embedding hkMDh_k^{\mathrm{MD}}? If it is learned, is ff recomputed during training, and do gradients flow through it?
  4. WkW_k is centered at tkt_k and the wavelet decomposition spans the sequence, so fkf_k depends on later observations. During autoregressive rollout, is ff recomputed each time a predicted snapshot is appended, which would change the coordinates of earlier tokens? How do prediction errors feed back into ff?
  5. How often does the ReLU yield Δfk=0\Delta f_k = 0, so that distinct frames share a temporal coordinate? Could this explain the Temporal Perception drop?
  6. For the video experiment, which backbone was used and what is LL? Please also report mean ± std over multiple seeds for the video results and for Tables 1–3 (three seeds are mentioned but no variance is shown), or a paired significance test.
  7. RoPE on absolute (x,y,z)(x,y,z) is translation-invariant but not rotation-invariant, whereas most baselines are SE(3)-equivariant. Is rotation augmentation used, and how sensitive are the results to the input frame?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.