- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows that standard self-attention is not flow equivariant and proposes a patch-based Flow Equivariant Vision Transformer. Building on FET, it lifts input sequences into multiple moving reference frames with shared weights, uses phase averaging to restore pixel-level translation equivariance, and combines the predictions via per-pixel velocity gating. Experiments on Flowing-MNIST and Navier–Stokes show improved generalization to unseen velocities within the velocity bank.
Critical strengths
Clear problem formulation and principled design. Figure 1 intuitively illustrates why self-attention is not flow equivariant, and the lift-based construction and equivariance argument are clean. The model achieves errors several orders of magnitude lower than ViT on unseen in-bank velocities, empirically confirming the design.
Critical weaknesses
-
Incremental contribution over FET. The core mechanism (lift and weight sharing) is directly inherited from FET, and the new components are limited to replacing convolutions with patch attention, applying an existing phase-averaging technique, and a forecasting head. The potentially more interesting flow-mixing construction (Appendix B) is described but not evaluated. Moreover, the method essentially amounts to a brute-force search over 25 predefined velocity hypotheses selected via gating, and it does not generalize beyond the velocity bank.
-
Narrow experimental scope and limited practical validation. Experiments are restricted to 32×32 synthetic data with global constant translations, and there are no real-video experiments to support the claims about video understanding in the abstract. The baselines are mainly a small two-block ViT, without comparisons to other equivariant or motion-aware models (e.g., FERNN). Although the model processes 25 velocity slices and 4 patch phases per input, no FLOPs or wall-clock comparison is reported. On Navier–Stokes, a non-learned mean-frame preprocessing combined with FNO clearly outperforms the proposed method, leaving its practical advantage unclear.
Questions for authors
- Is the performance gain due to equivariance itself or simply to velocity coverage? How would a ViT trained with data augmentation over the bank velocities ({−2,…,2}²), or a compute-matched ViT, compare?
- Does the method work beyond global constant translations? For example, how does the per-pixel gating behave when multiple objects move at different velocities (as in the independent-digit conditions in the appendix), with non-integer velocities, or with camera motion in real videos?
2 comments
Dear Reviewers,
The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!
Best, AC
Reviewer UMFg
W1: Incrementality, flow mixing, and velocity coverage
We thank the reviewer for raising this concern. Earlier FET Khaliliya & Keller, 2026 is a non-archival workshop paper; our public comment explains its relationship to this submission. Our FET inherits its moving-frame construction and shared weights. Our additions handle patch-grid origin shifts and align predictions and per-pixel weighting scores at forecast time before combining them.
Enlarging the fixed velocity bank adds coverage, not extrapolation beyond the new bank.
Supplementary Fourier phase correlation estimates four motions instead of evaluating 25 fixed ones, retaining our backbone and prediction head. Updates are 5.75× faster and inference 6.07× faster, but training memory falls only 2.7%. Four performed best among 1/2/4 on development validation; error is 2.63× higher than original FET for independently moving objects at unseen velocities within the bank. We credit this improvement to Hosseinzadeh et al., 2026.
W2: Practical evaluation and baselines
We appreciate the reviewer’s concern regarding experimental scope. We compared released FERNN Keller, 2025 with FET across five seeds, matching 10,000 updates, training scenes/order, optimizer schedule and checkpoint selection. Both observe and train to predict ten frames; evaluation predicts 30 on 128 validation scenes. MSE below is × 10⁻³; ± denotes seed standard deviation.
“Shared” means digits move together; the last row uses two fixed velocity pairs. FET wins every seed on unseen motions; FERNN wins every seed on training-distribution motion. Outside the bank both errors are large (unscaled: FET 0.1524; FERNN 0.1547). FERNN has 739,584 parameters versus FET’s 134,472. A short timing comparison finds FERNN 21× faster per update, using 2.62 versus 0.83 GiB training memory. FERNN keeps its original rule of stopping gradients through predicted-frame inputs.
For weather, we forecast 69 atmospheric variables on a 128×256 grid from two historical ERA5 maps. FET and a matched Stormer-derived baseline share a small transformer; the baseline omits moving reference frames. Maps shift east/west; the main comparison trains both on −1/0/+1 cells per six hours. The score is three-day MSE divided by each variable’s variance, averaged over nine variables and shifts of ±2/±3/±4 cells per six hours.
FET wins every paired seed in each row. Reductions are relative to the baseline. All 36 evaluation dates (2019) were used in pilot work.
Mean FNO remains better on Navier–Stokes, however FET outperforms ViT.
Q1: Equivariance versus augmentation and compute
We give FET, ViT, and velocity-augmented ViT six hours of measured training-update time per seed (five seeds) on an A100-SXM4-80GB. They share the backbone and patch-phase averaging. ViT baselines complete about 74,000–76,000 updates versus 3,000 for FET. Models predict ten frames in training and 30 in evaluation. For independent training-distribution motion, validation MSE (× 10⁻³; mean ± seed standard deviation) is 2.097 ± 0.295 for FET, 3.971 ± 0.280 for ViT, and 9.019 ± 2.636 for augmented ViT: FET improves over ViT by 47.2%. For independent velocities absent from the original training distribution but within the bank, errors are 1.359 ± 0.168, 214.089 ± 43.940, and 35.332 ± 5.942. ViT and augmented ViT have 157.5× and 26.0× FET’s mean error; FET wins all five paired seeds against both in both conditions. Augmented ViT adds a common random velocity from {−1,0,1}² to both digits, extending each digit’s velocity support to the full bank {−2,…,2}².
Q2: Multiple velocities, fractional motion, and real camera motion
FET is trained and evaluated on independently moving objects; the scene need not become stationary in one frame. The shared transformer predicts temporal changes within each moving frame. In the five-seed, equal-training-time study, FET reduces mean 30-step validation MSE versus ViT by 47.2% on independent training-distribution motion, 39.8% with overlapping initial positions, and 30.7% for unequal-contrast digits. These results support forecasting under the tested motions. However, gates train through prediction loss and have not been validated as optical-flow estimates.