acceptodds
Under review as a conference paper at ICLR 2027

Flow Equivariant Vision Transformer

Abstract

Many visual sequences exhibit a geometric flow structure, where time-parameterized Lie-group actions describe motion and elements of the associated Lie algebra generate different flows. Standard self-attention does not inherently respect this structure, potentially limiting generalization across moving reference frames with geometric symmetries. In this work, we illustrate how this problem manifests in vision transformers, causing them to process motion-related sequences in an unrelated fashion. We then introduce a provably flow-equivariant vision transformer that generalizes to motion regimes absent from training. By connecting the geometry of continuous motion to patch-based visual representations, this framework provides an inductive bias for video understanding and spatiotemporal forecasting. Empirically, we demonstrate improved performance on both simple visual motion forecasting as well as PDE forecasting for physical systems with natural flow symmetries.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Flow Equivariance, Vision Transformers, Sequence Modeling, Equivariance, Movement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UMFgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that standard self-attention is not flow equivariant and proposes a patch-based Flow Equivariant Vision Transformer. Building on FET, it lifts input sequences into multiple moving reference frames with shared weights, uses phase averaging to restore pixel-level translation equivariance, and combines the predictions via per-pixel velocity gating. Experiments on Flowing-MNIST and Navier–Stokes show improved generalization to unseen velocities within the velocity bank.

Critical strengths

Clear problem formulation and principled design. Figure 1 intuitively illustrates why self-attention is not flow equivariant, and the lift-based construction and equivariance argument are clean. The model achieves errors several orders of magnitude lower than ViT on unseen in-bank velocities, empirically confirming the design.

Critical weaknesses
  1. Incremental contribution over FET. The core mechanism (lift and weight sharing) is directly inherited from FET, and the new components are limited to replacing convolutions with patch attention, applying an existing phase-averaging technique, and a forecasting head. The potentially more interesting flow-mixing construction (Appendix B) is described but not evaluated. Moreover, the method essentially amounts to a brute-force search over 25 predefined velocity hypotheses selected via gating, and it does not generalize beyond the velocity bank.

  2. Narrow experimental scope and limited practical validation. Experiments are restricted to 32×32 synthetic data with global constant translations, and there are no real-video experiments to support the claims about video understanding in the abstract. The baselines are mainly a small two-block ViT, without comparisons to other equivariant or motion-aware models (e.g., FERNN). Although the model processes 25 velocity slices and 4 patch phases per input, no FLOPs or wall-clock comparison is reported. On Navier–Stokes, a non-learned mean-frame preprocessing combined with FNO clearly outperforms the proposed method, leaving its practical advantage unclear.

Questions for authors
  1. Is the performance gain due to equivariance itself or simply to velocity coverage? How would a ViT trained with data augmentation over the bank velocities ({−2,…,2}²), or a compute-matched ViT, compare?
  2. Does the method work beyond global constant translations? For example, how does the per-pixel gating behave when multiple objects move at different velocities (as in the independent-digit conditions in the appendix), with non-integer velocities, or with camera motion in real videos?
2 comments

Reviewer UMFg

W1: Incrementality, flow mixing, and velocity coverage

We thank the reviewer for raising this concern. Earlier FET Khaliliya & Keller, 2026 is a non-archival workshop paper; our public comment explains its relationship to this submission. Our FET inherits its moving-frame construction and shared weights. Our additions handle patch-grid origin shifts and align predictions and per-pixel weighting scores at forecast time before combining them.

Enlarging the fixed velocity bank adds coverage, not extrapolation beyond the new bank.

Supplementary Fourier phase correlation estimates four motions instead of evaluating 25 fixed ones, retaining our backbone and prediction head. Updates are 5.75× faster and inference 6.07× faster, but training memory falls only 2.7%. Four performed best among 1/2/4 on development validation; error is 2.63× higher than original FET for independently moving objects at unseen velocities within the bank. We credit this improvement to Hosseinzadeh et al., 2026.

W2: Practical evaluation and baselines

We appreciate the reviewer’s concern regarding experimental scope. We compared released FERNN Keller, 2025 with FET across five seeds, matching 10,000 updates, training scenes/order, optimizer schedule and checkpoint selection. Both observe and train to predict ten frames; evaluation predicts 30 on 128 validation scenes. MSE below is × 10⁻³; ± denotes seed standard deviation.

MotionFETFERNN
Independent, training distribution0.996±0.1640.671±0.103
Shared unseen velocity within bank0.0753±0.02561.438±0.469
Different unseen velocities within bank0.541±0.0391.083±0.304

“Shared” means digits move together; the last row uses two fixed velocity pairs. FET wins every seed on unseen motions; FERNN wins every seed on training-distribution motion. Outside the bank both errors are large (unscaled: FET 0.1524; FERNN 0.1547). FERNN has 739,584 parameters versus FET’s 134,472. A short timing comparison finds FERNN 21× faster per update, using 2.62 versus 0.83 GiB training memory. FERNN keeps its original rule of stopping gradients through predicted-frame inputs.

For weather, we forecast 69 atmospheric variables on a 128×256 grid from two historical ERA5 maps. FET and a matched Stormer-derived baseline share a small transformer; the baseline omits moving reference frames. Maps shift east/west; the main comparison trains both on −1/0/+1 cells per six hours. The score is three-day MSE divided by each variable’s variance, averaged over nine variables and shifts of ±2/±3/±4 cells per six hours.

Training comparisonSeedsFETBaselineError reduction
Same shifts, 1,000 updates50.4492270.69697535.55%
Baseline sees all nine shifts20.4519090.53233015.11%
Equal training-update time20.4519090.76126640.64%

FET wins every paired seed in each row. Reductions are relative to the baseline. All 36 evaluation dates (2019) were used in pilot work.

Mean FNO remains better on Navier–Stokes, however FET outperforms ViT.

Q1: Equivariance versus augmentation and compute

We give FET, ViT, and velocity-augmented ViT six hours of measured training-update time per seed (five seeds) on an A100-SXM4-80GB. They share the backbone and patch-phase averaging. ViT baselines complete about 74,000–76,000 updates versus 3,000 for FET. Models predict ten frames in training and 30 in evaluation. For independent training-distribution motion, validation MSE (× 10⁻³; mean ± seed standard deviation) is 2.097 ± 0.295 for FET, 3.971 ± 0.280 for ViT, and 9.019 ± 2.636 for augmented ViT: FET improves over ViT by 47.2%. For independent velocities absent from the original training distribution but within the bank, errors are 1.359 ± 0.168, 214.089 ± 43.940, and 35.332 ± 5.942. ViT and augmented ViT have 157.5× and 26.0× FET’s mean error; FET wins all five paired seeds against both in both conditions. Augmented ViT adds a common random velocity from {−1,0,1}² to both digits, extending each digit’s velocity support to the full bank {−2,…,2}².

Q2: Multiple velocities, fractional motion, and real camera motion

FET is trained and evaluated on independently moving objects; the scene need not become stationary in one frame. The shared transformer predicts temporal changes within each moving frame. In the five-seed, equal-training-time study, FET reduces mean 30-step validation MSE versus ViT by 47.2% on independent training-distribution motion, 39.8% with overlapping initial positions, and 30.7% for unequal-contrast digits. These results support forecasting under the tested motions. However, gates train through prediction loss and have not been validated as optical-flow estimates.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Reviewer_VwpTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies flow equivariance in visual sequence forecasting, namely whether a model maintains consistent predictions under time-dependent translating reference frames. Building on the existing Flow Equivariant Transformer, the authors propose a patch-based implementation: inputs are lifted into multiple moving reference frames and processed independently by a shared spatiotemporal Transformer; patch phase averaging restores pixel-translation equivariance; and candidate predictions are transformed back to the original reference frame at the prediction time and combined through pixel-wise gating. Experiments cover Flow-MNIST and two-dimensional Navier–Stokes forecasting with periodic boundary conditions.

Critical strengths
  1. Distinguishing static spatial equivariance from time-parameterized flow equivariance is relevant to video forecasting and physical system modeling. Applying the output transformation at the prediction time T, rather than the last observation time T-1, is also a carefully considered methodological detail.

  2. Rather than assuming that standard patch embeddings are equivariant to arbitrary pixel translations, the authors address this issue through phase averaging. The Navier–Stokes experiments also correctly distinguish image translations from Galilean transformations, which modify both spatial coordinates and velocity values.

  3. The main results include five random seeds, while the appendices report extended training, training and checkpoint-selection horizons, comparisons with the non-patch method, and velocity-bank expansion. The authors retain unstable seeds and explicitly report that Mean FNO outperforms the proposed model, helping readers objectively assess its scope and limitations.

Critical weaknesses
  1. The additional contributions beyond the existing FET have not been sufficiently isolated and validated. Section 3 explicitly states that inverse-flow lifting and weight sharing across moving reference frames are inherited from the existing FET. The main additions are the patch backbone, phase averaging, and a spatially resolved prediction aggregation head. These extensions are reasonable, but it remains unclear which new components produce the key gains and whether they provide insights beyond the existing flow-equivariant framework.

  2. The large performance differences remain confounded by computational cost and long-horizon forecasting stability. For each prediction, FET processes 25 velocity slices and evaluates multiple patch phases for each slice. Matching backbone width and update counts does not imply matching computational budgets. Appendix D explicitly acknowledges this distinction, but the paper still lacks parameter counts, training and inference costs, peak GPU memory usage, and at least one comparison under comparable computational budgets. This makes it difficult to assess the trade-off between predictive gains and computational cost.

  3. The Flow-MNIST velocity-generalization results should be more clearly distinguished from generalization to multi-object motion. Appendix D states that the two digits move independently in Mixed ID, whereas both digits share the same velocity in the fixed-velocity diagnostics. Thus, the Mixed ID and unseen-in-bank conditions in Table 1 differ in both velocity range and motion structure. When both objects undergo the same constant-velocity motion, there exists a reference frame in which the entire image sequence is stationary, substantially simplifying the task. This also helps explain why unseen-in-bank error is considerably lower than Mixed ID error. The experiment supports generalization to unseen global flows, but does not, by itself, establish generalization to more general multi-object motion.

Questions for authors

See weaknesses.

2 comments

W1: Isolating the added contributions

We thank the reviewer for raising this matter. We would like to highlight that the earlier FET work is a non-archival workshop workshop CFP (additional official comment visible to everyone explains the relationship to the workshop to our submission). Our FET inherits its moving reference frames and shared weights Khaliliya & Keller, 2026; our additions enable forecasts at every pixel with a patch-based transformer: average over patch-grid origins, then align predictions and the scores that weight their contributions at the forecast time before combining them.

Replacing learned combination weights with equal weights in trained checkpoints gives mean squared error (MSE) 0.155 on MNIST and 0.268 on Navier–Stokes.

W2: Computational cost and rollout stability

We value the concerns raised by the reviewer regarding computational cost. To address this, we completed six model variants × five seeds, each with six hours of training updates on an A100-SXM4-80GB. Models observe ten frames and train to predict ten; we evaluate 30 predicted frames on 128 separate validation scenes. The final test set is unused. Checkpoints are selected using ten-step error.

FET uses 25 fixed candidate velocities (the velocity bank). Fourier phase correlation estimates K candidate velocities from the input. Augmented ViT adds the same random velocity change to both digits during training. Primary MSE below measures independently moving digits drawn from the training distribution (ID); ± denotes sample SD across five seeds.

ModelMSE × 10−310^{-3}s/updateInference msTraining MiB
FET, 25 velocities2.097 ± 0.2957.137184.06848.1
ViT3.971 ± 0.2800.2867.50802.5
Augmented ViT9.019 ± 2.6360.2897.48802.5
Fourier K=12.901 ± 0.5160.3128.43807.9
Fourier K=23.357 ± 0.7440.62915.77813.7
Fourier K=41.309 ± 0.2301.24230.32825.3

FET has 47.2% lower mean error than ViT and wins all five seeds. Update times average full runs. Separate inference profiles report median time over 20 repetitions after five warmups, for one predicted frame per batch of four 32×32 images; memory is peak allocated GPU memory during ten-step training with activation checkpointing. All variants register 134,472 parameters; the confidence head is unused with one candidate. ViT also averages four patch origins.

The Fourier models are supplementary, following recent related motion-estimation work Hosseinzadeh et al., 2026. It retains the backbone and prediction head. K=4 was selected among three values on validation: updates are 5.75× faster and inference is 6.07× faster than FET, but training memory falls only 2.7%. It takes more updates within six hours and has 2.63× higher error on independent unseen velocities within FET’s bank.

Estimated backbone costs per example and predicted frame are 101.97 GFLOPs for FET and 16.32 for K=4.

W3: Global velocity generalization versus multiple objects

We appreciate the concern raised by the reviewer and to address directly we now evaluated them separately: “global” means both digits share a velocity; “independent” means their velocities are sampled separately. “Unseen in bank” means absent from training but included among FET’s candidate velocities. These are five-seed mean 30-step MSE × 10−310^{-3} from the equal-time study.

MotionFETViTAugmented ViT
Global ID0.2171.8073.908
Global unseen in bank0.219213.5888.889
Independent ID2.0973.9719.019
Independent unseen in bank1.359214.08935.332
Independent outside bank147.060222.999190.904

For independent unseen velocities in the bank, SDs are 0.168 for FET and 43.940 for ViT. FET also improves overlap and low-contrast tests.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Reviewer_TCDeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper identifies a lack of flow equivariance in standard self-attention and proposes a Flow Equivariant Vision Transformer that incorporates geometric motion symmetries into patch-based representations. This provides an inductive bias for spatiotemporal forecasting and generalization to unseen motion regimes. Experiments on visual motion and PDE forecasting show gains over matched ViTs.

Critical strengths
  1. Concrete architectural extension: The method adapts flow-equivariant modeling to patch-based forecasting through patch-phase averaging and forecast-time output alignment.
  2. Improved forecasting performance: Five-seed experiments show lower errors than the matched ViT on unseen in-bank velocities and extended rollouts.
Critical weaknesses
  1. Limited benchmark coverage: Evaluation uses custom periodic tasks closely matched to the assumed symmetry, leaving the benefit on established forecasting benchmarks unverified.
  2. Unclear incremental value: Non-patch FET is evaluated only at a short training budget, and no direct comparison with FERNN is provided.
Questions for authors
  1. What is the precise equivariance guarantee for the implemented finite velocity bank, particularly its stated interior domain?
  2. What are the training and inference costs, and do the gains over ViT persist at matched compute?
  3. How does the method perform when motion only approximately satisfies the assumed flow symmetry, and what adaptations are needed for broader applications involving acceleration, occlusion, or non-periodic boundaries?
2 comments

W1: Benchmark coverage

We appreciate the reviewer’s comment regarding benchmark coverage. To address this, we include an additional experiment in our rebuttal on an established weather forecasting dataset.

Specifically, using historical ERA5 weather maps, we forecast 69 atmospheric variables on a 128×256 grid from two maps six hours apart. We compare two models: a competitive vision transformer baseline, the ’Stormer’ model (Nguyen et al., 2024 NeurIPS), and a flow-equivariant augmentation of this model that shares the same base architecture but adds our proposed construction. We impose small east/west motion on the dataset, training both models on shifts of −1/0/+1 cells per six hours, and then test both models on more significant ’velocities’ (i.e., greater shifts per unit time). The primary evaluation metric is 72-hour mean squared error, divided by each variable’s variance and averaged over nine variables and shifts of ± 2/ ± 3/ ± 4 longitude cells per six hours.

As can be seen in the table below, the flow equivariant model has a 35% reduction in forecasting error at a small training budget (first row). In the second row, we show that even when the baseline uses an augmented training set which includes the additional shifts, the FET model still maintains a 15% error reduction.

Baseline trainingSeedsFETBaselineReduction
Same shifts, 1,000 updates50.4492270.69697535.55%
All shifts −4,…,+420.4519090.53233015.11%
Equal training time20.4519090.76126640.64%

Matching training time gives the baseline about 8,000 updates versus FET’s 1,000, using shifts −1/0/+1.

W2: Incremental value and predecessor comparisons

We thank the reviewer for raising this concern. We wish to highlight that the earlier FET work Khaliliya & Keller, 2026 is a non-archival workshop paper workshop CFP (the additional official comment visible to everyone explains the relationship to the workshop to our submission). Our model inherits its moving reference frames and shared weights, and our unique additions for this submission handle the patch-grid origin and align forecasts and spatial gating scores at the prediction time before pooling.

To address the reviewer’s concern, we directly compare the FET and FERNN Keller, 2025 on our Flow-MNIST task, i.e., forecasting images of two moving digits. As can be seen in the table below, the FET dramatically outperforms the FERNN on this task for held-out velocities.

In detail, both observe ten frames and train to predict ten. Five seeds use 10,000 updates each, matching scenes, optimizer schedule, and checkpoint selection. Evaluation predicts 30 frames on 128 separate validation scenes. MSE is × 10−310^{-3} (mean ± standard deviation across seeds)

MotionFETFERNN
Independent, training distribution0.996±0.1640.671±0.103
Shared unseen, in bank0.0753±0.02561.438±0.469
Paired unseen, in bank0.541±0.0391.083±0.304

Across all seeds, FET wins both unseen-motion tests and FERNN wins on training motion. Both have high outside-bank MSE (FET: 0.1524; FERNN: 0.1547, unscaled). FERNN stops gradients through forecast feedback.

Q1: Finite-bank guarantee

For integer periodic shifts, let Sag(x)=g(x−a)S_a g(x)=g(x-a) and (Γηf)t=Stηft(\Gamma_\eta f)_t=S_{t\eta}f_t. With observations t=0,…,T−1t=0,\ldots,T-1, shared branch weights and predictions and logits aligned at forecast time TT give

ΦV(Γηf)=STηΦV−η(f),V−η=v−η:v∈V.\Phi_V(\Gamma_\eta f)=S_{T\eta}\Phi_{V-\eta}(f),\quad V-\eta=\\{v-\eta:v\in V\\}.

This gives same-bank equivariance when boosts permute the full bank of periodic actions. Our bank V=−2,…,22V=\\{-2,\ldots,2\\}^2 is not closed under general boosts. We will correct §3.6’s interior claim.

Q2: Matched compute

Our five-seed study gives each model six hours of updates per seed, using the same forecast horizons. Independent-motion validation MSE is in 10−310^{-3} units.

MeasureFETViT
Training-velocity MSE2.097±0.2953.971±0.280
Unseen-in-bank MSE1.359±0.168214.089±43.940
One-step inference, batch 4 (ms)184.067.50
Training memory (MiB)848.1802.5

FET wins every seed, with mean errors 47.2% and 157.5-fold lower. With four candidates, the Fourier estimator (MELSTM) speeds up updates 5.75-fold but has 2.63× FET’s unseen-in-bank MSE.

Q3: Approximate symmetry

Our transformer models changes in moving frames, while pixelwise gates combine predictions. In that study, FET reduces MSE by 39.8% for overlap and 30.7% for unequal contrast versus ViT. Acceleration would require time-varying alignment (MELSTM) and future-motion prediction. Occlusion calls for visibility, memory and appearance modeling. For non-periodic boundaries, we would mask invalid samples and reassess equivariance.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarification of the Relationship to the Prior Non-Archival Workshop Paper

We would like to clarify the relationship between this submission and Flow Equivariant Transformers by Khaliliya and Keller. The earlier work appeared at ICML 2026 workshop on weight space symmetries, which is non-archival, with no formally published proceedings (workshop policy here: https://www.weightsymmetry.com/cfp). Additionally, the paper's evaluation was limited to a one-dimensional synthetic motion coherence classification task.

On the other hand, our submission develops that preliminary workshop paper approach into a patch-based architecture for two-dimensional spatiotemporal forecasting. This transition represents a significant extension to the existing FET, specifically when it comes to addressing the translation-equivariance issues introduced by patchification and aligning the predictions and aggregation weights at forecast time.

We kindly ask the reviewers and chairs to assess our submission's overall contribution in this context and the significant architectural changes required to transition from a toy classification task to a dense forecasting task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.