acceptodds
Under review as a conference paper at ICLR 2027

Latent Prediction Helps Physical Representations Only When It Predicts the Future

Abstract

Joint-embedding predictive architectures (JEPAs), which predict in a learned latent space, are reported to learn better representations of physical systems than masked autoencoders, which reconstruct pixels. Previous studies, however, change the space of the prediction target together with its time horizon, the architecture and the training pipeline making it difficult to determine which factor contributes to improved representations. With this motivation, we separate prediction space from prediction horizon. We train ViT-S encoders on three physical simulations from The Well. These encoders are identical except for their objectives. We train on four distinct objectives, varying whether they predict (a) pixels or latent representations and (b) the current eight-frame clip or the next eight-frame clip. Across all encoder layers, we probe representations for regime parameters and for pooled and token-level physical quantities at current and future times. We find that regime probes and future-time probes favour representations trained on the Latent-Future objective, suggesting that latent prediction helps physical representations only when paired with future prediction. Probes for same-time physical quantities provide similar results for all models. We further analyse representations after adding noise to the inputs and find there is not one objective that produces representations more robust to noising. This paper analyses how prediction space and prediction horizon affect learned representations while holding the other training components fixed.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, JEPA, latent prediction, vMAE, pixel reconstruction, physical simulation, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rtxgOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work compares learned representations of joint-embedding predictive architectures (JEPAs) and masked autoencoders (MAEs). Specifically, the learned representations of physical simulations (i.e., video data) are compared using the Well dataset. Learned representations are probed for physical regime parameters to assess their quality. In contrast to previous work, this evaluation cleanly separates whether current clip prediction/ reconstruction or future clip prediction/ reconstruction leads to better representations. The results confirm that latent prediction with JEPA leads to better representations than masked autoencoding, while highlighting that JEPA yields significantly better representations when predicting future rather than current or past representations.

Critical strengths
  1. The work includes a controlled comparison of objectives that cleanly separates the general learning objective (JEPA vs. MAE) from using future or current/ past targets. The result that JEPA-like objectives are only superior when predicting the future is significant for the research community and challenges related frame-wise objectives like DINO (Caron et al., 2021).
  2. The result that probing results can even improve for JEPA representations when the target offset increases (see Figure 1) is an interesting finding that could lead to further research.

Caron et al., 2021: Emerging properties in self-supervised vision transformers. In ICCV

Critical weaknesses
  1. As mentioned in the limitations, the comparison only includes one model architecture (a ViT-S). Additionally, a vanilla ViT is primarily designed for image data, while the comparison focus on video data. Thus, evaluating more models like ViViT (Arnab et al., 2021) that are specifically designed for video understanding would improve the work.
  2. The evaluated JEPA objective is a bit dated. Using the recent LeJEPA (Balestriero & LeCun, 2025) objective would improve the work.

Arnab et al., 2021: ViViT: A Video Vision Transformer. In ICCV

Balestriero & LeCun, 2025: LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. In arXiv

Questions for authors

These questions are clarification questions and suggestions for the rebuttal:

  1. Why does the plot in the center of Figure 3 not include results for the pixel-current target or are they identical to the results for the pixel-future target?

  2. Do the results change when increasing the encoder capacity? I.e., using a ViT-B, ViT-L, ViT-H, etc.

  3. Can both objectives be combined? I.e. adding both a predictor for the JEPA objective and a decoder for pixel reconstruction and training them jointly.

3 comments

Thanks for the review!

Pixel-Current in the centre panel of the depth figure

You’re right, they overlap, and we’ve adjusted this to be clearer!

Larger encoders and alternative architectures

We agree that evaluating larger encoders and alternative video architectures would be valuable. We did not run ViT-B/L/H, ViViT or LeJEPA experiments. These would require substantially more compute and time. We appreciate it’s a limitation and scale our claims with this in mind.

Combining latent prediction with pixel reconstruction

We additionally trained an encoder using the Latent-Future objective together with Pixel-Current's masked-reconstruction loss, with equal loss weights and no tuning (Appendix F.6, Table 14). This is a one seed experiment so far but will be brought to 3 seeds by the CRC.

The combined objective preserves Latent-Future's long-horizon performance:

  • Rayleigh–Bénard, offset 40: error relative to persistence 0.076, compared with 0.074 for Latent-Future.
  • Active matter: 0.174, compared with 0.198 for Latent-Future.
  • Shear flow: 0.015, compared with 0.021 for Latent-Future.

It also retains some complementary strengths of pixel reconstruction. In active matter, it matches Pixel-Current on the current state; in shear flow, it gives the best encoder result at each future horizon. This suggests that adding a pixel reconstruction loss does not necessarily displace the information learned by latent-future prediction. Rather, the combined objective can preserve long-horizon information while adding complementary detail, although this finding requires confirmation across further seeds. A very cool idea worth exploring further!

Thank you for your response. The new caption of Figure 3 improves the clarity and the new results for using both objectives are interesting. More seeded runs and different model architectures would further improve the work. Overall, I’m leaning toward increasing my rating to 4.

Thanks for this! If you're still leaning towards a four we would really appreciate you changing it to that before the deadline?

Reviewer_mMnVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares pixel and latent prediction of current and future clips on three physical simulation datasets. Evaluations show that predicting future latents generally improves physical parameter estimation and long-horizon prediction of global quantities, but the improvements are less consistent for local predictions and noise robustness.

Critical strengths

The paper examines when latent prediction helps representation learning on physical simulation data by separating prediction space from prediction horizon. The results suggest that predicting future latents helps capture system parameters and future physical quantities. The findings in this paper provide useful insights about when and how to use latent for PDE and scientific forecasting.

Critical weaknesses

The paper evaluates learned representations through probes of physical parameters and selected quantities, but it is unclear whether better probe scores indicate better modeling of physical dynamics. The improvements are less consistent for local quantities, and no full-field forecasting comparison is provided.

Questions for authors

Are there results comparing latent- and pixel-trained models on full-field forecasting or multi-step rollouts? More analysis connecting the representation probe results to actual forecasting performance would better clarify their value for physical dynamics modeling. What explains why combining latent prediction with future prediction helps more than either change alone? To what extent do the probe results show that the representations capture physical dynamics, rather than just information useful for predicting the selected quantities?

2 comments

Thanks for the review!

Do the probes capture information beyond the original selected quantities?

We added probes of future energy at large, medium and small spatial scales, which are new targets not included in the original set of 13 physical quantities (Appendix, Spatial-scale correlations and future energy). At offset 40, Latent-Future has the lowest error in seven of the nine system-by-scale comparisons. In Rayleigh–Bénard, its largest improvements over Pixel-Current are for medium- and small-scale energy (80% and 81% lower error). In shear flow, it also improves medium- and small-scale energy prediction (44% and 37% lower error). These results show that the long-horizon advantage is not specific to the original hand-selected physical quantities.

We also added the control-plus-encoder analysis. In shear flow, where the regime-and-time control explains very little of the target variation, adding encoder features substantially reduces prediction error. This supports the interpretation that the representations contain information about the particular trajectory, rather than only its regime parameters.

Does this establish full-field forecasting or multi-step rollouts?

We added a direct full-field test (Appendix F.5, Table 13). We used a linear probe on each frozen token to predict its future patch at offsets 8, 16 and 40. All four encoders beat persistence and a linear probe on the raw pixels. The differences between objectives are smaller here than on the pooled physical quantities. Each offset is predicted directly from the input clip.

In drafting our experimental method at the start of this paper, we thought a lot about multi-step rollouts, and decided against it for the reason that doing true autoregressive rollouts using these frozen encoders requires, in the latent case, decoding the output back into pixel space after each time step. Therefore, measuring rollout error both encodes the information we want (how much information on the physical dynamics is there?) and a certain decodability bias (how easy is it for the latent representation to be decoded into pixel space?), the second of which might be favoured by the pixel objectives.

We therefore attempted to capture physical dynamics by probing variables in the future: this is a form of physical ‘forecasting’, and in order to predict this accurately with respect to persistence and our regime+time control, a representation must encode information on the dynamics of the system.

We do, however, agree that probes of global physical quantities do not, by themselves, establish accurate predictions of the complete physical field but this is also not what they are trained to do so. We have therefore kept the claims focused on the information accessible from frozen representations and simple probes, rather than claiming that the encoders are complete physical simulators.

Thanks for adding the experiment. It addresses my concern, though I’m more interested to see clearer evidence of Latent-Future’s practical value for full-field prediciton. I’m keeping my rating at 3.

Reviewer_XLESOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies empirically the effects of prediction space (pixel vs. latent) and prediction horizon (current vs. future) on physical representation learning. The authors compare four objectives on three physical simulation datasets. They find that future latent prediction performs best on regime probing and long-horizon physical-quantity probing.

Critical strengths
  • The paper tries to separate the effects of prediction space and prediction horizon. This is a useful question, as previous comparisons between JEPA and MAE often involve several differences in their training setups.

  • The four objectives use the same encoder and datasets. The comparison is easy to understand and helps identify how different training objectives affect the learned representations.

Critical weaknesses
  • Limited contribution beyond the empirical comparison. The controlled study of prediction space and horizon is useful, but the paper provides little explanation for why their combination works better. The proposed explanation about slowly varying physical structures is not directly tested.

  • Active matter shows quite different trends from the other two systems. The authors do not provide a convincing explanation for these differences, despite making a broad claim about physical representations.

  • The best encoder layer, probe type, and hyperparameters are selected independently for each setting. It is unclear whether the main findings hold with a fixed layer and probe architecture.

  • The conclusion claims that the results suggest that prediction horizon can be as important than latent embedding in JEPA. However, the results do no evaluate models trained at various prediction horizons.

Questions for authors
  • Why does active matter behave differently from the other two systems?

  • Could the authors provide evidence for the proposed explanation that Latent-Future favors slowly varying physical structures?

  • Please evaluate the model trained for various prediction horizons.

2 comments

Thank you for the review! Find the main points below, and a discussion on active matter in the main comment.

Dependence on probe selection and representation layer

We re-evaluated the stored probe results under alternative protocols:

  • Ridge-only and MLP-only probes.
  • Fixed representation layers (6, 11 and 12) shared across encoders and targets.
  • A common layer chosen per system, rather than separately for each encoder.

At offset 40, Latent-Future remains the best objective under all 11 protocols in Rayleigh–Bénard and shear flow, and under 10 of 11 protocols in active matter (Table 8). We additionally evaluated Ridge probes on the submission checkpoints with both layer and regularisation penalty fixed in advance, so that neither is selected on validation. At the last two layers, Latent-Future is best in all 36 combinations of system, layer and penalty from 10⁻⁵ to 1. The effect is not equally strong under every protocol: it is largest in the final encoder blocks, and in shear flow it is substantially larger with MLP than Ridge probes. We report these qualifications rather than claiming that every probe family or layer gives the same ranking. The active-matter result is also qualified under the common final-checkpoint rule: Pixel-Future ties Latent-Future under MLP probes at layers 11 and 12.

Dependence on the training prediction horizon

We trained additional Latent-Future and Pixel-Future models with targets 16 and 32 frames ahead, keeping the other training components fixed. These additional experiments use one seed, so we present them as exploratory evidence. By the CRC, 3 seeds will be trained.

For latent prediction, longer training horizons do not consistently improve long-horizon probing:

  • In Rayleigh–Bénard, offset-40 error relative to persistence is 0.074, 0.075 and 0.078 for training gaps of 8, 16 and 32 frames.
  • In active matter, the corresponding values are 0.198, 0.183 and 0.192.
  • In shear flow, longer gaps are less effective (0.021, 0.028 and 0.040), although all remain substantially better than the current-clip objectives.

Pixel prediction behaves differently:

  • In Rayleigh–Bénard, a 16-frame target does not help, but a 32-frame target closes most of the long-horizon gap to Latent-Future.
  • In shear flow, the long-horizon error falls from 0.049 to 0.025 and 0.018 as the pixel training gap increases from 8 to 16 and 32 frames.
  • In active matter, the final-checkpoint pixel models also come closer to Latent-Future at long horizons.

Thus, predicting the future helps across the horizons tested, while the latent-versus-pixel contrast is clearest at the next-clip horizon used in the main paper. We have avoided claiming that latent prediction is superior at every possible training horizon.

These results are reported in Appendix, Training prediction horizon.

Summary

The additional analyses support the main conclusion while making its scope more precise: the advantage of Latent-Future is robust to many changes in probe protocol at the paper's training horizon, but its magnitude depends on system, layer, probe family and training horizon. Let us know if you need any clarifications, and if not, we’d appreciate if you’d consider raising the score to an accept.

Hi, we hope our comment has addressed your points? If so, we'd appreciate if you'd consider improving your score.

Comments on the paper

1 comment

We thank all the reviewers for their responses in what has been an insightful process! Each review has received an individual reply, but a summary of some key points (which are referred to at points in the individual responses) is listed below.

Active matter

We have expanded the discussion of why active matter behaves differently from Rayleigh–Bénard convection and shear flow (Results, Controls show where the test is informative; Appendix, Active-matter controls and trajectory-specific information). The important distinction is not that active matter has no trajectory-specific variation at long horizons. At offset 40, the fraction of variation remaining after accounting for regime and clip time is 0.12, comparable to 0.17 in Rayleigh–Bénard. Rather, the remaining trajectory-specific variation is not usefully predictable from the input representation at this horizon:

  • At offset 8, adding encoder features reduces control error from 0.405 to 0.157–0.265.
  • At offset 16, it reduces error from 0.406 to 0.225–0.357, with Latent-Future giving the largest improvement.
  • At offset 40, the control error is 0.359, and adding encoder features gives 0.359–0.378. No objective meaningfully improves on the control.

This contrasts with Rayleigh–Bénard and shear flow, where encoder features continue to improve substantially on the regime-and-time control at offset 40. We therefore interpret the active-matter result as a limit on the trajectory-specific information recoverable at this horizon, rather than as evidence that the encoders have failed to learn useful physical representations.

Evidence about slowly varying physical statistics

We added an analysis of temporal correlations at different spatial scales and probes of future energy at large, medium and small scales (Appendix, Spatial-scale correlations and future energy). Fine-scale patterns decorrelate rapidly in all three systems. Large-scale structure persists to 40 frames in Rayleigh–Bénard and shear flow, but not in active matter (see above). The new energy probes show that Latent-Future's advantage is not limited to the original physical targets: it has the lowest error in seven of the nine system-by-scale comparisons at offset 40. The results therefore refine our explanation. The representations capture slowly varying statistics of the flow, including energy at different spatial scales, rather than necessarily predicting the exact fine-scale pattern. We have also qualified the claim that pixel prediction is dominated by unpredictable detail: this is clearly supported for active matter, but is too strong as a general explanation for all three systems.

A more consistent checkpoint protocol

We now evaluate every encoder in the main comparison at its final checkpoint after 100,000 training steps (Setup, Checkpoints). In the submission, latent encoders were evaluated at their final checkpoint, whereas pixel encoders were evaluated at their lowest pretraining-validation loss. This selected the active-matter Pixel-Future models unusually early, at approximately 18,000–24,000 steps. We now use a common training budget and final-checkpoint evaluation for all objectives. This is the fairer comparison because it removes an objective-dependent checkpoint-selection rule and gives every encoder the same training budget. The protocol follows the fixed-budget principle used in V-JEPA (Bardes et al., 2024), whose models are trained for 90,000 iterations, although our architecture and training schedule are not identical. This change does not alter the overall conclusion: the combined Latent-Future objective remains strongest at long horizons across the three systems. It does, however, make the active-matter result more conservative: final-checkpoint Pixel-Future approaches Latent-Future at offset 40 (error relative to persistence 0.222 versus 0.200, within its seed variation) and ties it on individual-clip regime prediction (MSE 0.0156 versus 0.0159). We have revised the active-matter claims accordingly rather than presenting the latent advantage as uniform across systems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.