acceptodds
Under review as a conference paper at ICLR 2027

Flow-JEPA: Robust Latent Dynamics for JEPA World Models via Flow Matching

10 views

Abstract

Joint-Embedding Predictive Architectures (JEPAs) provide a powerful framework for latent world modeling and planning in a reconstruction-free manner. Although numerous JEPA-based approaches have been proposed to mitigate representation collapse, our experiments on localized, out-of-distribution visual noise reveal that performance degradation remains pronounced and unresolved. We propose Flow-JEPA (F-JEPA), a flow-based latent dynamics model that jointly generates a sequence of future latent states conditioned on the current observation and actions. A Gaussian distribution serves as the flow source, exposing the vector field to perturbed latent trajectories as it learns to transport them toward clean future representations. This formulation retains the reconstruction-free JEPA framework while switching from pointwise transition regression to stochastic trajectory-level prediction. F-JEPA raises mean success from % to % under clean observations and from % to % under noisy conditions. Further evaluations over varying perturbation severity and inference settings show that the performance advantage persists across a broad range of conditions. These results suggest that conditional flow matching provides a promising alternative to deterministic autoregressive prediction as a dynamics formulation in JEPA world models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Joint-Embedding Predictive Architectures, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VD1MOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Flow-JEPA. Instead of using the one-forward-step autoregressive next-latent prediction as in LeWM, this paper introduces flow-matching over a full trajectory with multiple timesteps. Experimental results report the mean success from 86% to 92% under clean observations and from 67% to 86% under noisy conditions.

Critical strengths
  1. The presentation is sound and clear. Notions are easy to follow.
  2. The design choice is well motivated: stable representations do not imply robust dynamics and autoregressive latent prediction is vulnerable.
  3. The improvements, especially on the noisy-condition benchmark, are promising and consistent across various environments.
Critical weaknesses
  1. The proposed components are not well ablated: Although the authors admit it in the limitations, I would like to point out here that all the modifications, including joint trajectory prediction, predictor architecture, stochastic training, etc. are not ablated individually, which weakens the contributions of this paper. We do not know which components contribute the most and make the method work.
Questions for authors
  1. Table 1 includes DINO-WM, but the noisy-observation comparison in Table 2 includes only LeWM. Could the authors evaluate DINO-WM under the same perturbation protocol? This would help establish whether the reported robustness gains extend to a baseline using frozen pretrained representations. If this evaluation is infeasible, please explain the practical limitation.
2 comments
AuthorsOpenReview ↗

Response to Reviewer VD1M

We thank the reviewer for recognizing that the motivation is well grounded, the presentation is clear, and the noisy-condition improvements are consistent across environments.

W1: “The proposed components are not well ablated… joint trajectory prediction, predictor architecture, stochastic training, etc. are not ablated individually.”

We added Fast-LeWM [1] as an additional baseline. Fast-LeWM predicts multiple future latent states in parallel across the planning horizon, avoiding recursive autoregressive rollout, while using deterministic MSE supervision.

Under localized Gaussian noise:

MethodTwo-RoomReacherPush-TOGBench-CubeMean
DINO-WM7836205848
LeWM7674585867
Fast-LeWM2644664846
Flow-JEPA9690906686

Fast-LeWM also performs substantially below Flow-JEPA under blur, brightness, and saturation perturbations. The Fast-LeWM comparison is reported in Table 2 for localized Gaussian noise and in Tables 7–9 for the additional perturbation types.

These results support our proposed flow-matching formulation beyond joint multi-horizon prediction alone.


Q1: “Could the authors evaluate DINO-WM under the same perturbation protocol?”

We have now evaluated DINO-WM [2] under all clean and perturbed conditions, including localized Gaussian noise, blur, brightness, saturation, and the corresponding perturbation-severity sweeps.

Under localized Gaussian noise, DINO-WM achieves 48% mean success, compared with 67% for LeWM, 46% for Fast-LeWM, and 86% for Flow-JEPA.

DINO-WM is strongest in some individual settings under the additional perturbations, particularly on Two-Room under blur and under some stronger appearance shifts. Because DINO-WM uses a frozen, externally pretrained DINOv2 ViT-S encoder, whereas Flow-JEPA, LeWM, and Fast-LeWM use ViT-Tiny encoders trained only on environment-specific data, we treat DINO-WM as a complementary pretrained-feature reference and distinguish this comparison from the controlled comparison among the three ViT-Tiny methods. For these evaluations, we use the officially released LeWM and Fast-LeWM checkpoints. The DINO-WM checkpoints are taken from Oberweger et al [3].

References
[1] Gao, Y. and Xu, X. Fast LeWorldModel. 2026.
[2] Zhou, G., et al. DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning. 2024.
[3] Oberweger, F. F., et al. Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations and Goals. 2026.

AuthorsOpenReview ↗

Additional experiment addressing W1

W1: The proposed components are not well ablated… joint trajectory prediction, predictor architecture, stochastic training, etc. are not ablated individually.

We now added an architecture-controlled multi-horizon MSE variant in Two-Room.

The predictor receives empty tokens as inputs and predicts all future latents in a single forward pass, supervised by the same MSE loss used in Fast-LeWM. Because flow time is absent, flow-time-conditioned AdaLN is replaced with RMSNorm. Apart from these changes, no other model component or training setting is modified.

MethodCleanGaussianBlurBrightnessSaturation
MSE variant9464328690
Flow-JEPA10096829898

Fast-LeWM controls for joint multi-horizon prediction, while the additional MSE variant further reduces the architectural confound. Together, these comparisons provide stronger evidence that the observed robustness is associated with the flow-matching formulation rather than either of these factors alone. The architecture-controlled results are reported in Table 11 of Appendix B.3.

Reviewer_ZkpEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Flow-JEPA, a reconstruction-free latent world model that replaces deterministic autoregressive prediction with trajectory-level conditional flow matching. The model jointly predicts future latent states conditioned on the current observation and action sequence, aiming to improve robustness against visual perturbations and accumulated prediction errors. Experiments across four control environments demonstrate improved planning success under both clean and noisy conditions compared with existing JEPA-based world models.

Critical strengths
  1. The paper identifies an important limitation of autoregressive JEPA world models, namely their sensitivity to task-irrelevant visual perturbations and accumulated prediction errors.
  2. The proposed approach achieves substantial improvements under visual perturbations across four control environments. Additional experiments investigate sensitivity to perturbation severity, flow integration steps, and planning budgets.
Critical weaknesses
  1. Applying flow matching to latent world modeling is not fundamentally new, as explored in prior work such as FlowWM. The main contribution is adapting this formulation to JEPA-based planning, but the methodological novelty remains incremental.
  2. The flow matching objective perturbs future latent trajectories during training, but the current observation used for conditioning remains clean. It is therefore unclear why this formulation should inherently improve robustness to noisy conditioning observations. Direct analyses of latent prediction errors and representation sensitivity are missing.
  3. Robustness is primarily evaluated using localized Gaussian noise in four relatively simple control environments. More realistic distribution shifts, longer-horizon tasks, and stronger baselines are needed to demonstrate general applicability.
Questions for authors

See the "Weaknesses"

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZkpE (1/2)

We thank the reviewer for recognizing the importance of the robustness failure mode in autoregressive JEPA world models and the consistent gains of Flow-JEPA across the evaluated environments.

W1: “Applying flow matching to latent world modeling is not fundamentally new, as explored in prior work such as FlowWM… the methodological novelty remains incremental.”

Our contribution is not the generic use of latent-space flow matching, but the exploration of flow matching as an action-conditioned latent-dynamics formulation for JEPA-based world models, together with an empirical investigation of its effect on robustness.

FlowWM [1] and Flow-JEPA differ in both setting and objective. FlowWM studies stochastic future forecasting in a fixed pretrained feature space without action conditioning and is primarily motivated by multimodal future prediction. In contrast, Flow-JEPA learns an action-conditioned latent world model jointly with a compact JEPA representation and uses the predicted future latent trajectory for goal-conditioned planning.

More importantly, we ask a different research question: whether replacing recursive, deterministic autoregressive latent prediction with trajectory-level conditional vector-field prediction improves robustness to perturbation-induced deviations in the latent state.


W2: “The flow matching objective perturbs future latent trajectories during training, but the current observation used for conditioning remains clean. It is therefore unclear why this formulation should inherently improve robustness to noisy conditioning observations.”

The current latent state conditions future prediction through cross-attention, where the noisy future latents act as queries and the current state and action embeddings provide the conditioning keys and values. During training, the predictor therefore repeatedly learns to process perturbed latent queries throughout the network. This differs from pointwise next-latent regression, which is supervised only at clean latent transitions.

Our hypothesis is that training the dynamics model over stochastic regions of latent space yields a vector field that is more tolerant to latent deviations at inference.

To better separate the effect of flow matching formulation, we added comparisons with Fast-LeWM [2]. Fast-LeWM also predicts multiple future latent states across the full horizon, but uses deterministic MSE supervision.

Under localized Gaussian noise:

MethodTwo-RoomReacherPush-TOGBench-CubeMean
DINO-WM7836205848
LeWM7674585867
Fast-LeWM2644664846
Flow-JEPA9690906686

The substantial gap to Fast-LeWM is consistent with our hypothesis. The comparison with Fast-LeWM under localized Gaussian noise is reported in Table 2.

We agree that direct latent-sensitivity or prediction-error analyses would further strengthen the mechanistic explanation and identify these as useful future analyses.

AuthorsOpenReview ↗

Response to Reviewer ZkpE (2/2)

W3: “Robustness is primarily evaluated using localized Gaussian noise… More realistic distribution shifts, longer-horizon tasks, and stronger baselines are needed.”

We have expanded the evaluation in two directions.

First, we add Fast-LeWM [2] and DINO-WM [3] as baselines under all clean and perturbed conditions. Second, we add three additional perturbation types: blur, brightness, and saturation.

At representative perturbation levels:

PerturbationDINO-WMLeWMFast-LeWMFlow-JEPA
Blur mean71666487
Brightness mean74625582
Saturation mean76747289

The representative results under blur, brightness, and saturation are reported in Tables 7–9, we also provide full severity sweeps in Appendix B.4.

DINO-WM is stronger in some individual settings. Because DINO-WM uses a larger ViT-S encoder with external DINOv2 pretraining, we distinguish absolute performance against DINO-WM from the controlled comparison among Flow-JEPA, LeWM, and Fast-LeWM, which use the same ViT-Tiny encoder family and environment-specific training data. For these evaluations, we use the officially released LeWM and Fast-LeWM checkpoints. The DINO-WM checkpoints are taken from Oberweger et al [4].

We agree that the current tasks have relatively short planning horizons. We retain the LeWM/Fast-LeWM benchmark setting for controlled comparison and state evaluation on longer-horizon tasks as an important future direction.

References
[1] Porcher, F., et al. Flow Matching in Feature Space for Stochastic World Modeling. 2026.
[2] Gao, Y. and Xu, X. Fast LeWorldModel. 2026.
[3] Zhou, G., et al. DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning. 2024.
[4] Oberweger, F. F., et al. Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations and Goals. 2026.

AuthorsOpenReview ↗

Additional experiment addressing W2

W2: “It is therefore unclear why this formulation should inherently improve robustness to noisy conditioning observations. Direct analyses of latent prediction errors and representation sensitivity are missing.”

To directly test whether the observed planning robustness is associated with improved dynamics prediction under perturbed observations, we added an open-loop terminal latent prediction-error analysis in Two-Room. For each candidate action sequence, we execute the 25-step sequence in the simulator and compare the predicted terminal latent with the real encoded terminal observation under clean or perturbed condition. We report normalized MSE using the clean terminal latent as the normalization reference.

MethodCleanGaussianBlurBrightnessSaturation
LeWM1.2271.352 (+0.125)2.747 (+1.520)2.597 (+1.370)1.293 (+0.066)
Flow-JEPA0.9791.114 (+0.135)1.277 (+0.298)1.261 (+0.282)0.970 (-0.009)

Values in round brackets denote the change relative to the clean condition. Flow-JEPA yields lower normalized terminal prediction error in all five conditions. These results indicate that the robustness advantage of Flow-JEPA is accompanied by more stable latent-dynamics prediction under appearance perturbations.

The full protocol and results are provided in Table 15 of Appendix B.4.

Reviewer_A5EbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Flow-JEPA, which proposes to formulate the predictor in JEPA-based world models using conditional flow matching (CFM) rather than autoregressive prediction. The motivation stems from how latent spaces, while intended to remove nuisance details from raw visual inputs, do not necessarily translate to similarly robust latent dynamics. Thus, CFM, which naturally incorporates stochasticity, may be intuitively more robust to perturbations. Flow-JEPA is implemented on top of LeWorldModel, an existing latent world model that uses SIGReg, and evaluated on Two-Room, Reacher, Push-T, and OGBench-Cube. The experiments show that Flow-JEPA performs well with normal, clean observations, as well as perturbed observations that have Gaussian noise injected. Further ablations show that the advantage of Flow-JEPA grows with increased noise effects.

Critical strengths
  1. The introduction and motivation for the proposed method is well-written and easy to follow. Similarly, the diagram (Figure 3) for the method itself.
  2. The method shows clear gains over the LeWM baseline, particularly in the noisy setting.
  3. The method is robust to different noise scales and sizes.
Critical weaknesses
  1. Only one form of noise is used. Furthermore, the noise seems to be injected into the observation itself rather than affecting the dynamics, so this seems like it is more so testing the encoder, which JEPA was designed to do. Also, it is unclear if the constraints for the noise to be far away from the agent and task-relevant objects have a large impact and if this is a realistic setting. The noise form (Gaussian) is well-aligned with the flow matching objective, so it is unclear how the method may do with other noise forms.
  2. Flow-JEPA is primarily only tested on LeWM in the noisy conditions.
  3. There are important ablations missing. Several baselines such as an action chunking / multi-action prediction are immediate ones to run. While this is stated in the limitations, it seems to me like the most straightforward ablations to run to disentangle their effects.
Questions for authors
  1. How much compute / how much slower is Flow-JEPA compared to the autoregressive baseline?
  2. What if you cannot predict the whole trajectory at once because it is too long?
  3. What if you somehow sampled correlated latents for the future positions? Would this potentially do better?
  4. Why did you use 10 CEM iterations instead of the usual 30?
5 comments
AuthorsOpenReview ↗

Response to Reviewer A5Eb (1/2)

We thank the reviewer for recognizing that the motivation is clear, the method is easy to follow, and that Flow-JEPA shows consistent gains over LeWM under noisy observations and across different perturbation scales.

W1: “Only one form of noise is used… this seems like it is more so testing the encoder… The noise form (Gaussian) is well-aligned with the flow matching objective.”

Injecting perturbations into the observation is deliberate. Our central question is whether a JEPA world model can maintain reliable planning when visual appearance changes while the underlying physical state and environment dynamics remain unchanged.

Although JEPA-style representation learning can encourage the encoder to suppress task-irrelevant visual details, it does not guarantee complete invariance to observation perturbations. A perturbation can shift the encoded state, and the downstream effect of that shift also depends on how the learned dynamics predictor behaves. Therefore, our motivation is that robustness requires a dynamics predictor that tolerate representation deviations alongside useful representations. Flow-JEPA addresses this through conditional flow matching over future latent trajectories. We have clarified this motivation in Section 1 and Section 3.1.

To address the concern that the original evaluation used only localized Gaussian noise, we have added three additional perturbation types: blur, brightness, and saturation. Unlike the background-localized noise, these transformations modify the full observation. Their definitions are provided in Appendix A.2.

At representative perturbation levels, the mean success rates are:

PerturbationDINO-WMLeWMFast-LeWMFlow-JEPA
Blur71666487
Brightness74625582
Saturation76747289

The representative results are reported in Tables 7–9, with full perturbation-factor sweeps provided in Appendix B.4. Across these sweeps, Flow-JEPA remains generally stronger than LeWM and Fast-LeWM [1]. DINO-WM [2] is strongest in some individual settings, but it also benefits from a larger ViT-S encoder and large-scale external DINOv2 pretraining. Among Flow-JEPA, LeWM, and Fast-LeWM, which use the ViT-Tiny encoder and environment-specific training data, Flow-JEPA shows substantially stronger robustness across the tested perturbations.


W2, W3: “Flow-JEPA is primarily only tested on LeWM in the noisy conditions.” “There are important ablations missing. Several baselines such as an action chunking / multi-action prediction are immediate ones to run.”

We have expanded the noisy-condition evaluation to include both Fast-LeWM [1] and DINO-WM [2] under the same perturbation protocols. Fast-LeWM builds on LeWM but avoids recursive one-step rollout by predicting future latent states in parallel. These predictions are supervised with an MSE objective. This comparison tests whether the robustness gain can be explained by predicting multiple future horizons in parallel.

Under localized Gaussian noise:

MethodTwo-RoomReacherPush-TOGBench-CubeMean
DINO-WM7836205848
LeWM7674585867
Fast-LeWM2644664846
Flow-JEPA9690906686

Fast-LeWM achieves 46% mean success compared with 86% for Flow-JEPA. The same gap is observed under blur, brightness, and saturation. These results support our hypothesis that flow matching formulation imporves the robustness to visual perturbations.

We also evaluate all four methods under blur, brightness, and saturation. The localized-Gaussian comparison is reported in Table 2, with factor sweeps in Table 3, 4, 11, 12. The additional perturbation results are reported in Tables 7–9, with full sweeps in Tables 14–16.

For these evaluations, we use the officially released LeWM and Fast-LeWM checkpoints. The DINO-WM checkpoints are taken from Oberweger et al [3].

AuthorsOpenReview ↗

Response to Reviewer A5Eb (2/2)

Q1: “How much compute / how much slower is Flow-JEPA compared to the autoregressive baseline?”

We did a direct inference-time comparison between Flow-JEPA and LeWM under the same 10-iteration CEM budget on Two-Room:

MethodDynamics latencyPlanning time50 rollouts
LeWM2.1 ms / prediction105.0 ms21.7 s
Flow-JEPA5.0 ms / flow step400.0 ms34.4 s

Flow-JEPA is approximately 3.8×3.8\times slower per planning cycle because it performs multiple flow integration steps. However, the end-to-end gap over 50 rollouts is smaller, approximately 1.6×1.6\times, because Flow-JEPA typically completes the task with fewer replanning cycles. We report this analysis in Appendix B.2.


Q2: “What if you cannot predict the whole trajectory at once because it is too long?”

The current implementation matches the planning horizon used by LeWM, Fast-LeWM, and DINO-WM. The formulation itself is not restricted to a specific horizon and could be extended to longer prediction windows. For substantially longer trajectories, another option is to apply the predictor recursively over multiple chunks.

However, we agree that maintaining accurate predictions over substantially longer horizons remains an important open problem in world modeling. We identify evaluation on longer-horizon tasks as a future direction.


Q3: “What if you somehow sampled correlated latents for the future positions? Would this potentially do better?”

Thank you for the insightful question. A correlated source distribution could encode additional temporal structure and may simplify the transport problem. However, it would not remove the need to model the action-conditioned dynamics: at inference time, the predictor still does not observe the ground-truth future trajectory and must determine how the sampled latent trajectory should evolve under the given action sequence.

We therefore view temporally correlated or otherwise structured source distributions as a promising extension of the current framework.


Q4: “Why did you use 10 CEM iterations instead of the usual 30?”

LeWM uses 10 CEM iterations in Two-Room, Reacher, and OGBench-Cube, and 30 iterations in Push-T.

For Flow-JEPA on Push-T, we found that 10 iterations already gives the strongest performance under both clean observations and localized Gaussian noise. We therefore use 10 iterations for the main Flow-JEPA Push-T result and explicitly evaluate 10, 20, and 30 iterations.

For the additional blur, brightness, and saturation experiments, all four methods follow the LeWM planning settings.

We have clarified this in Sections 5.1 and 5.4.

References
[1] Gao, Y. and Xu, X. Fast LeWorldModel. 2026.
[2] Zhou, G., et al. DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning. 2024.
[3] Oberweger, F. F., et al. Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations and Goals. 2026.

AuthorsOpenReview ↗

Additional experiment addressing W3

W3: “There are important ablations missing. Several baselines such as an action chunking / multi-action prediction are immediate ones to run.”

Our comparison with Fast-LeWM tests whether the robustness gain can be explained by multi-horizon prediction. Since Fast-LeWM uses a different predictor architecture, we added an architecture-controlled multi-horizon MSE variant in Two-Room to further reduce this architectural confound.

The predictor receives empty tokens as inputs and predicts all future latents in a single forward pass, supervised by the same MSE loss used in Fast-LeWM. Because flow time is absent, flow-time-conditioned AdaLN is replaced with RMSNorm. Apart from these changes, no other model component or training setting is modified.

MethodCleanGaussianBlurBrightnessSaturation
MSE variant9464328690
Flow-JEPA10096829898

The MSE variant remains reasonably strong under clean observations, but degrades substantially under perturbations.

Together with the Fast-LeWM comparison, this stricter architecture-controlled ablation provides additional evidence that Flow-JEPA's robustness advantage is not explained simply by parallel multi-horizon prediction or by the predictor architecture. The complete ablation is reported in Table 11 of Appendix B.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.