- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Fourier-World, a visual world model that predicts future VFM features by explicitly representing temporal dynamics in the Fourier domain. It applies a temporal FFT to four recent frozen DINOv3 feature maps, constructs a Fourier memory, and uses query-based Fourier cross-attention and spatial propagation to predict residual feature updates. The model is pretrained on Kinetics-700 and evaluated on five benchmarks covering segmentation, depth estimation, action prediction, and future-point forecasting, showing strong overall performance and good computational efficiency.
Critical strengths
- The paper presents a coherent and interpretable way to explicitly model temporal dynamics in VFM feature space, rather than relying entirely on implicit temporal modeling.
- The empirical evaluation is broad, covering five benchmarks and multiple downstream tasks, with competitive performance, efficiency analysis, and several diagnostic studies.
Critical weaknesses
- The role and robustness of the Fourier representation are not fully established. The DCT/Haar comparisons are mainly representation-level analyses rather than end-to-end trained predictors under the same setting, so it remains unclear how much of the gain comes specifically from the Fourier representation. This is further complicated by the use of non-uniform temporal sampling during pretraining while the FFT is still defined over sample indices, with timestamp conditioning applied only to the keys. Since the Fourier representation is the central contribution, stronger evidence that it is both necessary and robust would materially strengthen the paper.
- The quantitative evidence for long-horizon rollout stability is still limited. Most dataset-level evaluations focus on H1/H3, while longer-horizon results are mainly qualitative. Since reduced error accumulation is a central motivation of the method, stronger quantitative results at longer horizons are important to support the claimed stability advantage.
Questions for authors
- Could you provide end-to-end comparisons with DCT, Haar, or direct time-domain representations under the same architecture and training protocol? Also, how sensitive is the method to non-uniform temporal sampling, given that the FFT is computed over sample indices?
- Could you provide dataset-level quantitative results at longer rollout horizons, such as H5/H8, to verify whether the stability advantage persists beyond H3?
3 comments
Thank you for the detailed rebuttal and additional experiments.
The comparisons with alternative temporal representations and the irregular-sampling tests are helpful. The H5/H8 results also strengthen the evidence for rollout stability, although the advantage over Copy-last narrows at longer horizons.
My main concern remains how much of the improvement comes from Fourier modeling itself. Under a shared predictor, Fourier-real does not consistently outperform other representations, while the full model uses a different architecture. The activity gate also seems to have limited practical impact on the evaluated data.
Overall, the response addresses much of my feedback and strengthens the paper. I find the method promising, but its specific contributions are not yet fully isolated. I will maintain my original Weak Accept rating.
We sincerely appreciate the reviewer's thoughtful engagement throughout the discussion.
We are continuing to gather further evidence to strengthen our claims and better isolate the contributions of individual components, even after the rebuttal period. We will soon upload a PDF containing the experimental results from the rebuttal, along with additional supporting evidence and analyses.
Thank you once again for your valuable feedback and constructive discussion.
We sincerely thank the reviewer for the thoughtful and constructive review. Below, we address the raised weaknesses and questions with additional experiments.
Weakness 1 & Question 1: Alternative Temporal Representations and Irregular Sampling
We appreciate the reviewer's important observation regarding the role of Fourier representations. As stated in our Introduction, our goal is not to establish that Fourier is universally superior to other temporal bases, but rather to investigate whether established transforms and motion-analysis techniques can provide useful inductive biases for world models.
Our representation analyses in Tables 8 and 9 motivate the choice of Fourier coordinates, but do not establish their superiority in end-to-end prediction. We therefore conducted additional retraining experiments comparing Raw, DCT, Haar, and Fourier-real representations under a shared lightweight ViT predictor, following the same scratch-trained
-styleevaluation protocol.The first four representations use the shared lightweight predictor, while Fourier-World-style uses our original Fourier-specific architecture.
Fourier-real achieves the best performance on Cityscapes at both horizons and comparable performance on VSPW. Meanwhile, the full Fourier-World-style predictor achieves the highest performance across all four settings. These results support the utility of Fourier-based temporal modeling without claiming universal superiority over alternative bases. Further details are provided in our response to Reviewer 2Jp2.
We also thank the reviewer for questioning the sensitivity of sample-index FFT to non-uniform temporal sampling. To investigate this, we evaluated Fourier-World on Cityscapes using four observation schedules: Uniform [0,3,6,9], Mild [0,2,5,9], Strong [0,1,4,9], and Random [0,a,b,9], where a and b are distinct randomly selected indices from 1–8. The target index was fixed at 12, ensuring the same forecasting horizon. We compared predictions with and without timestamp-key conditioning.
Cityscapes — mIoU (↑)
Fourier-World maintains comparable performance across these sampling patterns despite computing the FFT over sample indices. Timestamp-key conditioning further improves performance in all four settings, particularly under random sampling (+2.01 mIoU).
These results provide evidence that Fourier-World remains effective under the evaluated non-uniform sampling patterns, while timestamp conditioning offers additional benefits in certain settings.
Weakness 2 & Question 2: Long-Horizon Rollout Stability
We agree that dataset-level quantitative evaluations beyond H3 are important for supporting our long-horizon stability claims. Although our original manuscript included qualitative examples of extended rollouts, it lacked corresponding dataset-level results.
To address this, we extended our evaluation to H5 and H8 on VSPW and KITTI using our Kinetics-700 pretrained Fourier-World checkpoint and the publicly released DeltaWorld checkpoint.
VSPW — mIoU (↑)
KITTI — RMSE (m, ↓)
Fourier-World achieves the best performance across all evaluated horizons on both datasets. Notably, DeltaWorld falls below Copy-last at H5 on VSPW and H3 on KITTI, whereas Fourier-World maintains better performance than Copy-last through H8.
Although the performance gap over Copy-last narrows at longer horizons, these results provide additional empirical evidence that Fourier-World mitigates error accumulation relative to DeltaWorld during extended autoregressive rollouts.
Further discussion is provided in our response to Reviewer 2Jp2.
We sincerely thank the reviewer again for the valuable suggestions. We will incorporate these additional experiments and clarify the corresponding claims and limitations in the revised manuscript.