acceptodds
Under review as a conference paper at ICLR 2027

Fourier-World: Temporal Fourier States for Visual World Models

Abstract

World models aim to learn the dynamics of the world by predicting the future from visual observations. Recent approaches have shifted from directly generating RGB frames to predicting future features of vision foundation models (VFMs), which can support diverse downstream tasks. However, small feature errors accumulate during autoregressive prediction, degrading long-term rollout performance, while how existing predictors internally represent and use temporal changes remains unclear. We propose Fourier-World, which mitigates error accumulation by explicitly structuring temporal context as a frequency-domain state. Fourier-World applies a Fourier transform to the four most recent context features. A query initialized from the latest feature reads the resulting Fourier coefficients to predict feature-space deltas, which are then accumulated to generate future dense VFM features. Pretrained on the large-scale Kinetics-700 dataset, Fourier-World demonstrates strong transfer performance in segmentation, depth estimation, action anticipation, and causal future-point forecasting across five benchmarks: Cityscapes, VSPW, KITTI, EK100, and TAP-Vid-RGB-Stacking. It also maintains stable predictions over long autoregressive rollouts, achieving high computational efficiency and competitive performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Model, Self-Supervised Learning, Latent Space Prediction, Visual Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pArrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Fourier-World, a visual world model that predicts future VFM features by explicitly representing temporal dynamics in the Fourier domain. It applies a temporal FFT to four recent frozen DINOv3 feature maps, constructs a Fourier memory, and uses query-based Fourier cross-attention and spatial propagation to predict residual feature updates. The model is pretrained on Kinetics-700 and evaluated on five benchmarks covering segmentation, depth estimation, action prediction, and future-point forecasting, showing strong overall performance and good computational efficiency.

Critical strengths
  1. The paper presents a coherent and interpretable way to explicitly model temporal dynamics in VFM feature space, rather than relying entirely on implicit temporal modeling.
  2. The empirical evaluation is broad, covering five benchmarks and multiple downstream tasks, with competitive performance, efficiency analysis, and several diagnostic studies.
Critical weaknesses
  1. The role and robustness of the Fourier representation are not fully established. The DCT/Haar comparisons are mainly representation-level analyses rather than end-to-end trained predictors under the same setting, so it remains unclear how much of the gain comes specifically from the Fourier representation. This is further complicated by the use of non-uniform temporal sampling during pretraining while the FFT is still defined over sample indices, with timestamp conditioning applied only to the keys. Since the Fourier representation is the central contribution, stronger evidence that it is both necessary and robust would materially strengthen the paper.
  2. The quantitative evidence for long-horizon rollout stability is still limited. Most dataset-level evaluations focus on H1/H3, while longer-horizon results are mainly qualitative. Since reduced error accumulation is a central motivation of the method, stronger quantitative results at longer horizons are important to support the claimed stability advantage.
Questions for authors
  1. Could you provide end-to-end comparisons with DCT, Haar, or direct time-domain representations under the same architecture and training protocol? Also, how sensitive is the method to non-uniform temporal sampling, given that the FFT is computed over sample indices?
  2. Could you provide dataset-level quantitative results at longer rollout horizons, such as H5/H8, to verify whether the stability advantage persists beyond H3?
3 comments

We sincerely thank the reviewer for the thoughtful and constructive review. Below, we address the raised weaknesses and questions with additional experiments.


Weakness 1 & Question 1: Alternative Temporal Representations and Irregular Sampling

We appreciate the reviewer's important observation regarding the role of Fourier representations. As stated in our Introduction, our goal is not to establish that Fourier is universally superior to other temporal bases, but rather to investigate whether established transforms and motion-analysis techniques can provide useful inductive biases for world models.

Our representation analyses in Tables 8 and 9 motivate the choice of Fourier coordinates, but do not establish their superiority in end-to-end prediction. We therefore conducted additional retraining experiments comparing Raw, DCT, Haar, and Fourier-real representations under a shared lightweight ViT predictor, following the same scratch-trained -style evaluation protocol.

RepresentationVSPW H1 ↑VSPW H3 ↑Cityscapes H1 ↑Cityscapes H3 ↑
Raw51.8644.8859.8248.70
DCT51.1744.8759.4848.02
Haar51.4545.0260.2948.22
Fourier-real51.2044.5860.4848.91
Fourier-World-style (Full)52.6045.9962.9350.22

The first four representations use the shared lightweight predictor, while Fourier-World-style uses our original Fourier-specific architecture.

Fourier-real achieves the best performance on Cityscapes at both horizons and comparable performance on VSPW. Meanwhile, the full Fourier-World-style predictor achieves the highest performance across all four settings. These results support the utility of Fourier-based temporal modeling without claiming universal superiority over alternative bases. Further details are provided in our response to Reviewer 2Jp2.

We also thank the reviewer for questioning the sensitivity of sample-index FFT to non-uniform temporal sampling. To investigate this, we evaluated Fourier-World on Cityscapes using four observation schedules: Uniform [0,3,6,9], Mild [0,2,5,9], Strong [0,1,4,9], and Random [0,a,b,9], where a and b are distinct randomly selected indices from 1–8. The target index was fixed at 12, ensuring the same forecasting horizon. We compared predictions with and without timestamp-key conditioning.

Cityscapes — mIoU (↑)

SamplingTimestamp OnTimestamp OffOn−Off (pp)
Uniform61.692761.3216+0.3711
Mild62.011561.6506+0.3609
Strong61.419261.4041+0.0151
Random61.855459.8461+2.0093

Fourier-World maintains comparable performance across these sampling patterns despite computing the FFT over sample indices. Timestamp-key conditioning further improves performance in all four settings, particularly under random sampling (+2.01 mIoU).

These results provide evidence that Fourier-World remains effective under the evaluated non-uniform sampling patterns, while timestamp conditioning offers additional benefits in certain settings.

Weakness 2 & Question 2: Long-Horizon Rollout Stability

We agree that dataset-level quantitative evaluations beyond H3 are important for supporting our long-horizon stability claims. Although our original manuscript included qualitative examples of extended rollouts, it lacked corresponding dataset-level results.

To address this, we extended our evaluation to H5 and H8 on VSPW and KITTI using our Kinetics-700 pretrained Fourier-World checkpoint and the publicly released DeltaWorld checkpoint.

VSPW — mIoU (↑)

MethodH1H3H5H8
Copy-last50.9444.2940.6837.18
DeltaWorld (mean-20)54.0446.1739.2432.62
Fourier-World54.2148.2344.2937.84

KITTI — RMSE (m, ↓)

MethodH1H3H5H8
Copy-last3.764.865.405.95
DeltaWorld (mean-20)3.375.196.768.35
Fourier-World3.154.0044.785.68

Fourier-World achieves the best performance across all evaluated horizons on both datasets. Notably, DeltaWorld falls below Copy-last at H5 on VSPW and H3 on KITTI, whereas Fourier-World maintains better performance than Copy-last through H8.

Although the performance gap over Copy-last narrows at longer horizons, these results provide additional empirical evidence that Fourier-World mitigates error accumulation relative to DeltaWorld during extended autoregressive rollouts.

Further discussion is provided in our response to Reviewer 2Jp2.


We sincerely thank the reviewer again for the valuable suggestions. We will incorporate these additional experiments and clarify the corresponding claims and limitations in the revised manuscript.

Thank you for the detailed rebuttal and additional experiments.

The comparisons with alternative temporal representations and the irregular-sampling tests are helpful. The H5/H8 results also strengthen the evidence for rollout stability, although the advantage over Copy-last narrows at longer horizons.

My main concern remains how much of the improvement comes from Fourier modeling itself. Under a shared predictor, Fourier-real does not consistently outperform other representations, while the full model uses a different architecture. The activity gate also seems to have limited practical impact on the evaluated data.

Overall, the response addresses much of my feedback and strengthens the paper. I find the method promising, but its specific contributions are not yet fully isolated. I will maintain my original Weak Accept rating.

We sincerely appreciate the reviewer's thoughtful engagement throughout the discussion.

We are continuing to gather further evidence to strengthen our claims and better isolate the contributions of individual components, even after the rebuttal period. We will soon upload a PDF containing the experimental results from the rebuttal, along with additional supporting evidence and analyses.

Thank you once again for your valuable feedback and constructive discussion.

Reviewer_TQGfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Fourier-World, a predictor that forecasts future frozen DINOv3 features from context frames. At each spatial token, a real DFT over the four-frame temporal window produces DC, complex AC and Nyquist coefficients, which are lifted into three memory tokens. Queries which are initialized from a shared learnable mask read this memory through temporal cross-attention and propagate the updates with spatial attention. The result is added as a residual to the latest feature, and a binary spectral-activity gate suppresses updates at tokens without non-DC energy. The model is pretrained on Kinetics-700 and evaluated without further training, through frozen heads, on Cityscapes, VSPW, KITTI, TAP-Vid-RGB-Stacking and an action-end-aligned EK100 protocol. It reports the best result on 10 of 11 metrics against COSMOS and DeltaWorld. The paper also includes scratch-trained capacity-matched “style” baselines, rollout geometry metrics, efficiency profiling, component ablations and a key/value attribution analysis.

Critical strengths
  • The design is simple and coherent. It separates state from dynamics: the latest VFM feature carries the scene through a residual path, while a parameter-free, invertible temporal DFT exposes how each token has changed. A content-free mask query, which cannot copy current content, reads only the non-DC modes through complex-linear projections that keep phase information intactt. Applying Fourier analysis to video is not new in itself, but using it as a per-token memory for residual prediction in a frozen VFM space is a clean idea.
  • Within DeltaWorld’s own protocol (frozen DINOv3 ViT-B, Kinetics-700 pretraining, shared task heads), this deterministic predictor matches or exceeds DeltaWorld’s mean-of-20 prediction on most metrics while its pretraining is cheaper. It also extends the protocol to point forecasting and action prediction.
Critical weaknesses
  • The paper does not ablate its components enough to show what each one contributes. Apart from query initialization (Table 11), the only ablations remove Fourier components from the already-trained model at test time (Table 6 and 10). This discards information, which would hurt any model. None of the main design choices is tested by retraining without it: the Fourier representation versus raw frames or a DCT, the residual output added to the last frame, the activity gate, the phase-preserving complex projections, or the exclusion of the DC mode from the values. The comparisons with other methods (Tables 2 and 3) change several of these at once. In addition, all “style” baselines perform worse than Copy Last on VSPW, which is unexpected.
Questions for authors
  • Could you retrain the model without each main component to show its contribution? This would cover raw frames or a DCT instead of Fourier modes, no residual output, no activity gate, unconstrained instead of complex projections, and DC included in the values. Could you also train the style baselines with the same residual output?
  • In Table 10, “Complete update off” shouldnt be identical to Copy Last, since a zero update returns the last observed feature? This holds at H3, but not at H1 (e.g., 51.44 vs 50.94 mIoU on VSPW; Tables 10 and 3). Could you clarify the source of this difference?
  • The activity gate switches off only when the four input features are exactly identical, which in actual video clips almost never happens because of noise, compression etc. Could you report how often the gate is zero on the benchmark clips and how results change without it?

As a suggestion for future work: Table 2 compares only against DeltaWorld’s mean-of-20 prediction, while DeltaWorld’s public release reports best-of-20 results that are higher on all six VSPW, Cityscapes and KITTI numbers. Although best-of-N uses the ground truth for selection, this gap illustrates the value of modeling uncertainty. Since real-world futures are often uncertain, could the framework be extended to stochastic prediction?

4 comments

We sincerely thank the reviewer for the constructive feedback. Below, we address the raised weaknesses and questions with additional experiments and analyses.


Weakness 1 & Question 1: Ablation and Retraining

We appreciate the reviewer's observation that our original ablations were primarily conducted at inference time without retraining. Although reproducing every experiment within the rebuttal period is challenging, we conducted additional retraining experiments following the matched scratch-trained -style protocol.

First, regarding the comparison across temporal bases, this concern overlaps with Reviewer 2Jp2's comments, where we provide further details. The results are summarized below.

RepresentationVSPW H1 ↑VSPW H3 ↑City H1 ↑City H3 ↑
Raw51.8644.8859.8248.70
DCT51.1744.8759.4848.02
Haar51.4545.0260.2948.22
Fourier-real51.2044.5860.4848.91
Fourier-World-style52.6045.9962.9350.22

The first four representations use a shared lightweight predictor, while Fourier-World-style uses our original Fourier-specific architecture. Fourier-real achieves the best performance on Cityscapes under the shared predictor, supporting the potential utility of Fourier coordinates.

We also retrained Fourier-World with its main architectural components modified or removed, including residual prediction, the activity gate, complex-linear projection, and DC exclusion from values.

ModelCity H1City H3VSPW H1VSPW H3
FW Full62.9350.2252.6245.98
No-residual56.4744.4250.2343.92
No-gate62.9249.8352.6145.91
Unconstrained projection63.4049.8152.6845.99
DC-values63.7251.7252.3845.68

Removing residual prediction causes the largest performance degradation, confirming its importance in our architecture. Removing the gate has only a modest effect on benchmark performance, consistent with its primary role as a structural safeguard for static contexts (see Question 3).

Including DC in values yields mixed results, improving Cityscapes while slightly degrading VSPW. This suggests that the benefit of DC exclusion may depend on the dataset. We will further discuss this observation in the revision.

The unconstrained projection increases the parameter count from 85.653M to 91.551M, so this comparison should be interpreted with that limitation in mind.

Following the reviewer's suggestion, we additionally retrained the style baselines with the same residual output formulation.

ModelCity H1City H3VSPW H1VSPW H3
FW Full62.9350.2252.6245.98
No-residual56.4744.4250.2343.92
DINO-World direct56.4343.1950.6743.73
DINO-World + residual55.6541.4250.9744.24
DINO-Foresight direct58.8047.3649.1141.22
DINO-Foresight + residual61.0947.5250.0041.42
V-JEPA-style direct56.7442.8247.7437.49
V-JEPA-style + residual53.5639.5850.9444.22
Copy-last53.5639.6050.9444.30

The effectiveness of residual prediction varies across architectures. Importantly, Fourier-World maintains the best performance in all four settings even when competing predictors use the same residual formulation. Thus, our gains cannot be explained by residual prediction alone.

Under the same limited 2,500-update training budget, several style baselines fall below Copy-last on VSPW H3, whereas Fourier-World consistently outperforms it.

This suggests that these architectures may benefit from larger training budgets, whereas Fourier-World already achieves strong performance under limited training, consistent with the benefits of its explicit inductive bias.

Question 2: Complete Update Off vs. Copy-last

We thank the reviewer for identifying this inconsistency. The reviewer is correct that disabling all prediction updates should exactly recover Copy-last.

Upon rechecking our evaluation records, we found that the H1 ablation results in Appendix Table 10 had inadvertently been taken from the 20K checkpoint instead of the intended 60K checkpoint.

We corrected the evaluation using the 60K checkpoint. The updated results are shown below, where ablation rows report changes relative to the full model.

VariantVSPW H1VSPW H3City H1City H3
Fourier-World54.2148.2363.1951.75
Imaginary quadrature off-2.510-3.134-7.016-8.366
AC off-2.140-2.990-8.238-7.778
Nyquist off-1.822-2.410-6.715-7.565
AC phase spatial shuffle-0.862-1.411-3.748-4.216
Complete update off-3.266-3.931-9.633-12.150

The corrected Complete update off results now match Copy-last across all four settings, up to rounding precision.

For completeness, we also report the performance of the available checkpoints at different pretraining stages.

UpdatesVSPW H1VSPW H3City H1City H3
20K53.7146.997661.68546.7281
40K53.9847.455662.171747.4329
60K54.2148.2363.1951.75

The original H1 inconsistency is consistent with the accidental use of the 20K checkpoint. The corrected ablations preserve the overall trends and do not change our main conclusions. We will correct Table 10 and include the additional information in the revised manuscript.

Question 3: Spectral-Activity Gate

We thank the reviewer for this insightful observation. We measured the frequency of zero-valued gates at every individual spatial token across all evaluated crops on Cityscapes, VSPW, and KITTI (H1–H3), using our 60K pretrained checkpoint.

Interestingly, we observed zero instances of gate == 0. This is consistent with the exact-zero criterion, since real-world VFM features generally contain nonzero temporal variation.

Therefore, the gate did not suppress prediction updates in these evaluations, and our benchmark improvements cannot be attributed to gate-induced copying of static regions. The no-gate retraining results also show only modest differences in performance.

Rather than serving as a primary mechanism for improving ordinary benchmark performance, the gate provides a structural safeguard that guarantees an exact static fixed point when all observed features are identical, as demonstrated in Appendix B.6. We will clarify this role in the revised manuscript.

Future Work: Stochastic Prediction

We agree that modeling uncertainty is an important direction. Fourier-World could potentially be extended to stochastic prediction by introducing stochastic query initialization, analogous to the multiple-hypothesis approach used in DeltaWorld. We leave this extension for future work.


We sincerely thank the reviewer again for the valuable suggestions. We will incorporate the additional experiments, correct the identified inconsistency, and clarify the corresponding claims in the revised manuscript.

I thank the authors for their effort in addressing my concerns with additional experiments and clarifications.

The explanation for the old Table 10 (now Table 12) is convincing. One small note: the revised Table 12 seems to contain a new mistake. The VSPW H1 entries for “Complete update off” (−0.862) and “AC phase spatial shuffle” (−0.679) do not match the rebuttal (−3.266 and −0.862).

Regarding the retraining results, residual prediction appears to be the key component, while the gate, the complex projection and the DC exclusion have small or mixed effects. Table 15 also shows that without the residual, Fourier-World performs comparably to the other methods, while adding it to some baselines can bring clear gains (e.g., +2.29 mIoU for DINO-Foresight-style on Cityscapes H1). Is there a deeper explanation or intuition for why this component plays such a crucial role in this framework? I would also encourage including the remaining component ablations and the clarified role of the gate in the manuscript, as they are currently missing.

As a minor suggestion, it would help to mark the changes made during the rebuttal in a different color e.g. blue, so that they are easier to spot in the revised manuscript.

Overall, I will maintain my score.

Thank you for following up and for pointing out the two incorrect entries in Table 12.

1. Correction to Table 12

Both incorrect VSPW H1 entries were simple typos. We have corrected “Complete update off” to −3.266 and “AC phase spatial shuffle” to −0.862, matching the values reported in our rebuttal. Thank you for catching these typos.

2. Why residual prediction helps Fourier-World

Direct and residual prediction use the same future-feature loss, but they place different demands on the predictor. A direct predictor must reconstruct the entire next feature map, including scene content that persists across frames. In Fourier-World, the residual path carries the last observed feature map into the prediction. The reader can therefore use the non-DC Fourier values, which encode observed temporal variation, to estimate where and by how much that feature map should change. We believe this alignment between the information available to the reader and the update it must predict helps explain why residual prediction works particularly well in Fourier-World.

As you noted, DINO-Foresight-style also benefits from a residual output, gaining 2.29 mIoU points on Cityscapes at H1. For DINO-World-style, the same change yields modest gains on VSPW and lowers mIoU at both horizons on Cityscapes. Fourier-World gains 6.46 points on Cityscapes at H1, improves in all four settings, and achieves the highest mIoU in each setting when all predictors use a residual output. We discuss these results in Appendix B.7 and Table 16 of the revised manuscript.

3. Component ablations and the activity gate

Appendix B.3 now reports the remaining retraining ablations in Table 13 and clarifies the role of the activity gate. When the observed features at a location are identical across all four frames, the gate suppresses both the local update and any incoming spatial update. As demonstrated in Appendix B.6, this ensures that a fully static context remains unchanged throughout rollout. Across the evaluated Cityscapes, VSPW, and KITTI clips, we observed zero instances of the gate suppressing an update.

We have uploaded the revised manuscript. We would appreciate it if you could consult the automatically generated PDF diff to see the changes once it becomes available. Thank you again for your careful reading and constructive suggestions.

Reviewer_2Jp2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Fourier-World predicts future dense features of a frozen DINOv3 backbone from four observed frames. It transforms the temporal context into Fourier coefficients and uses temporal cross-attention, spatial attention, and residual feature updates to generate future features autoregressively. The design includes amplitude-preserving normalization and activity gating, with an exact copy-last fixed point for static contexts. The predictor is pretrained on Kinetics-700 and evaluated through segmentation, depth, future-point forecasting, and a stop-aligned action-classification protocol across five benchmarks. The paper also studies feature geometry, computational cost, and properties of the temporal representation.

Critical strengths
  1. The method introduces a coherent and relatively lightweight inductive bias for feature prediction. The separation of temporal components, phase-aware processing, and residual updates is motivated explicitly, and the static-context fixed-point property is a useful, concrete guarantee. The appendix's representation diagnostics help explain the motivation for choosing Fourier coordinates.

  2. The evaluation covers several downstream tasks with a frozen visual backbone, including transfer from Kinetics-700 and comparisons with controlled predictor variants. The gains over copy-last and competitive feature-prediction baselines, particularly at H3, make this more than a purely representational proposal. Reporting feature-geometry diagnostics and computational cost alongside task metrics is useful. The lower reported FLOPs are encouraging, although the model is not the fastest method in every latency comparison.

Critical weaknesses
  1. The central Fourier-specific contribution is not sufficiently isolated from the rest of the predictor. A complete four-frame Fourier transform is an invertible change of coordinates; the important empirical question is whether those coordinates improve learning or rollout behavior relative to simpler temporal representations under comparable conditions. Appendix B.2 does compare Fourier, DCT, and Haar, but it evaluates representation statistics and decomposes predictions from a shared lightweight autoregressive rule, rather than comparing separately trained predictors. The authors explicitly acknowledge this limitation. Likewise, removing quadratures or frequency components at inference demonstrates dependence on information used during training, but does not establish superiority over an alternative representation trained from the outset. The query-initialization ablation is helpful but does not resolve this attribution issue. Without matched trained controls, it remains unclear how much of the gain comes from Fourier memory rather than residual prediction, normalization, gating, or the attention design.

  2. The evidence for stable long-term dynamics is narrower than the framing suggests. The main quantitative downstream comparisons emphasize H1 and H3, corresponding to short horizons, while the longer qualitative examples do not establish dataset-level long-horizon reliability. Preserving feature geometry and having a static fixed point are useful properties, but neither establishes accurate evolution of moving content over extended rollouts. The work provides evidence for short-horizon visual feature forecasting; stronger claims about long-term world modeling require more direct validation. I do not regard the absence of robot control or action conditioning as a flaw by itself, nor consider a large robot WAM a necessary baseline. The concern is the gap between the paper's own rollout-stability claims and the quantitative evidence presented.

Questions for authors
  1. Can you provide a matched training comparison against raw temporal features or temporal differences, and a DCT or Haar representation, retaining comparable model capacity, training budget, normalization, residual prediction, and gating? Which gains remain specifically attributable to the Fourier representation? This would most directly affect my assessment of the contribution.

  2. Do the improvements persist in dataset-level evaluations at H5, H8, or beyond, relative to copy-last and the strongest feature-prediction baseline? A breakdown on moving regions or higher-motion examples would help distinguish accurate dynamics from preservation of mostly static content. If such evidence is unavailable, please clarify the supported scope of the long-term stability claim.

  3. For the stop-aligned EK100 protocol, how much of the gain is due to useful predicted future features rather than information already available in the observed context? An observed-context-only or repeated-last-feature control with an otherwise matched classification setup would help. Please also keep this protocol distinct from standard pre-onset action anticipation when interpreting the results.

2 comments

We thank the reviewer for the thoughtful and constructive feedback. We conducted additional experiments to address the concerns raised.


Weakness 1 & Question 1: Comparison with Alternative Temporal Representations

Fourier-World's predictor explicitly leverages Fourier-specific properties, including DC/AC/Nyquist decomposition, complex-valued phase information, and amplitude-preserving values. Therefore, directly replacing Fourier coefficients with other temporal bases within the original architecture may not provide a fair comparison.

To address this, we designed a separate, lightweight ViT-based predictor and retrained Raw, DCT, Haar, and Fourier-real representations under matched conditions, following the same scratch-trained -style protocol used in our paper.

RepresentationVSPW H1 ↑VSPW H3 ↑Cityscapes H1 ↑Cityscapes H3 ↑
Raw51.8644.8859.8248.70
DCT51.1744.8759.4848.02
Haar51.4545.0260.2948.22
Fourier-real51.2044.5860.4848.91
Fourier-World-style (Full)52.6045.9962.9350.22

Fourier-World-style uses our original Fourier-specific predictor and is reported separately from the matched lightweight-predictor comparison.

Under the shared lightweight predictor, Fourier-real achieves the best performance on Cityscapes at both H1 and H3, while showing comparable performance to alternative bases on VSPW. These results suggest that Fourier representations can provide a useful inductive bias for temporal prediction.

Our goal is not to claim that Fourier is universally superior to other temporal bases. Rather, as stated in our Introduction, our central question is:

"Can established transforms and analysis techniques for understanding motion benefit world models?"

Weakness 2 & Question 2: Dataset-Level Long-Horizon Rollout Stability

We agree that dataset-level quantitative evaluations beyond H3 are important for supporting our rollout stability claims.

We therefore extended our evaluation to H5 and H8 on VSPW and KITTI, using our Kinetics-700 pretrained Fourier-World checkpoint and the publicly released DeltaWorld checkpoint.

VSPW — mIoU (↑)

MethodH1H3H5H8
Copy-last50.9444.2940.6837.18
DeltaWorld (mean-20)54.0446.1739.2432.62
Fourier-World54.2148.2344.2937.84

KITTI — RMSE (m, ↓)

MethodH1H3H5H8
Copy-last3.764.865.405.95
DeltaWorld (mean-20)3.375.196.768.35
Fourier-World3.154.0044.785.68

Notably, DeltaWorld falls below Copy-last at H5 on VSPW and H3 on KITTI, whereas Fourier-World maintains better performance than Copy-last even at H8.

Although the performance gap between Fourier-World and Copy-last narrows at H8, these results provide additional empirical evidence that Fourier-World mitigates error accumulation relative to DeltaWorld during extended autoregressive rollouts.

We will incorporate these quantitative results and clarify the supported scope of our long-horizon stability claims in the revised manuscript.

Question 3: EK100 Evaluation and the Utility of Predicted Future Features

We thank the reviewer for suggesting additional controls to determine whether predicted future features provide useful information beyond the observed context.

We conducted two additional evaluations:

  • Observed-only: A separate linear classifier trained using only the four observed frame features, following the same classifier training protocol.
  • Repeat-last: The final observed feature is repeated three times to replace the three predicted future features, preserving the original seven-frame classifier input.

EK100 — MCR@5 (↑)

MethodVerbNounAction
Observed-only41.8936.9120.38
Repeat-last45.2638.4722.64
DeltaWorld31.4330.3120.24
Fourier-World47.5540.8125.54
Oracle future59.6042.4930.78

Interestingly, DeltaWorld performs worse than both Observed-only and Repeat-last across all three metrics.

In contrast, Fourier-World consistently outperforms Repeat-last by 2.29 Verb, 2.34 Noun, and 2.90 Action MCR@5 points.

These results demonstrate that Fourier-World's predicted future features provide useful information for action classification beyond simply repeating the last observed representation.

We also agree with the reviewer that our EK100 evaluation should be clearly distinguished from standard pre-onset action anticipation. We will explicitly clarify that our evaluation follows an action-stop-aligned prediction protocol.


We thank the reviewer again for the constructive suggestions. We will incorporate these findings and clarify the corresponding claims in the revised manuscript.

Thank you again for your thoughtful review. Your questions guided three revisions to the manuscript. We now report a matched training comparison of four temporal representations, dataset-level rollout results through H8 on VSPW and KITTI, and a repeat-last control for the stop-aligned EK100 evaluation. We also clarify the scope of the rollout evidence and distinguish the EK100 protocol from standard pre-onset action anticipation.

As the discussion period draws to a close, we would be grateful if you could take a final look at the revised manuscript and reconsider your assessment in light of these results and our responses to your questions. Thank you for your time and consideration.

Comments on the paper

3 comments

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

Dear reviewers,

Thank you for your time and thoughtful feedback. With less than a day remaining in the rebuttal period, we would be grateful if you could review the revisions summarized below as you finalize your assessments.

Summary of Manuscript Revisions

  • Controlled comparison of transform bases. We evaluated Raw-centered, DCT, Haar, and Fourier-real using the same predictor. Fourier-real achieved the best embedding geometry at H3H_3 on both Cityscapes and VSPW, while differences in mIoU across the four bases were modest.
  • Long-horizon rollout evaluation. We extended the dataset-level evaluation to H5H_5 and H8H_8 on VSPW and KITTI. Fourier-World outperformed both DeltaWorld and Copy-last through H8H_8 on both benchmarks. The revised manuscript also points to the qualitative KITTI rollout in the appendix.
  • Copy-last baseline on EK100. We now report the EK100 scores for Copy-last using the repeat-last baseline and have removed the earlier statement that this baseline was omitted.
  • Residual prediction ablation. We added an ablation in the appendix in which the baselines were retrained using the same residual formulation. The effect of residual prediction varied across architectures, and Fourier-World achieved the highest mIoU in all four settings.
  • Stochastic prediction. We added a paragraph to the appendix outlining stochastic prediction as a direction for future work.

In the camera-ready version, we plan to document the retraining protocol and the experimental controls for the ablations in greater detail.

Sincerely,
The authors

Dear reviewers,

We have uploaded a further revision of the manuscript. We corrected two typos in Table 12 and added the remaining component ablations in Appendix B.3 (Table 13), along with a clarification of how the activity gate preserves static contexts. Appendix B.7 now explains why residual prediction is particularly effective in Fourier-World and relates this interpretation to the baseline comparisons in Table 16.

We would appreciate it if you could review the changes using the automatically generated PDF diff once it becomes available. Thank you again for your time and feedback.

Sincerely,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.