acceptodds
Under review as a conference paper at ICLR 2027

Invariant Imagination: Learning What Should Not Change the Future in JEPA World Models

Abstract

Joint-Embedding Predictive Architectures (JEPAs) support latent prediction and visual planning, yet strong clean performance does not guarantee reliable predictions under degraded observations. We characterize this vulnerability through the predictive brittleness gap, defined as the increase in prediction error when clean and degraded contexts are evaluated against the same clean target. We introduce **FAPI** (Future-Anchored Predictive Invariance), an adaptation framework that transforms observation-derived contexts while preserving clean target representations as shared predictive anchors. This asymmetric design encourages predictive stability without an explicit pairwise consistency loss or changes to the prediction architecture. We evaluate **FAPI** on LeWorldModel and Sub-JEPA across PushT, OGB-Cube, TwoRoom, and DMC under Gaussian noise, patch masking, and incomplete history. Under Gaussian noise, FAPI+LeWM improves mean planning success from 49.5% to 61.5% and reduces the mean brittleness gap by 24.6%; **FAPI+Sub-JEPA** improves success from 40.0% to 57.0% with a 43.1% gap reduction. Both models also improve mean clean success and planning under patch and history masking. Extensions to V-JEPA on Something-Something-v2 and I-JEPA on ImageNet yield lower prediction error and stronger clean–corrupt consistency, supporting the applicability of future anchoring across action-conditioned, video, and image prediction settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-Embedding Predictive Architecture, World Models, Predictive Robustness, Future-Anchored Learning, Latent Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1QAUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to make JEPA world models more reliable when observations contain noise, occlusion, or missing history.

  • Research problem: Strong prediction accuracy under clean observations does not guarantee robustness to corrupted inputs. The authors introduce the predictive brittleness gap, which measures the additional future-prediction error caused by degraded observations.
  • Core method: The proposed FAPI (Future-Anchored Predictive Invariance) adds noise, occlusion, or history masking to the observation context while keeping the action inputs and clean prediction target unchanged. Different degraded contexts are trained to predict the same future representation. FAPI fine-tunes existing JEPA models without changing their architecture or adding an explicit consistency loss.
  • Experimental results: Across four control benchmarks, FAPI improves average planning success under Gaussian noise from 49.5% to 61.5% for LeWM and from 40% to 57% for Sub-JEPA. Experiments with V-JEPA on video prediction and I-JEPA on image representation prediction also show lower prediction errors and greater stability.
  • Contribution and limitations: The paper provides an approach to evaluating and improving JEPA robustness by learning which observation changes should leave predictions unchanged. However, the degraded input must retain enough predictive information; severe information loss caused by occlusion remains difficult to address.
Critical strengths
  1. The paper presents a clear problem formulation and a well-focused motivation. The brittleness gap is simple and intuitive, and Figure 1 supports the conclusion that “clean predictive accuracy and predictive stability capture distinct properties of a learned world model.”

  2. The method is simple and broadly applicable, supported by substantial implementation and experimental effort. The authors demonstrate its effectiveness across multiple architectures and benchmarks.

Critical weaknesses
  1. The core method is essentially data augmentation that perturbs the input while keeping the target clean—a well-established approach closely related to denoising autoencoders. However, the paper lacks an adequate discussion of related work and does not cite relevant prior studies.
  2. The theoretical framing of Proposition 3.1 is overstated, while the result and its implications are not explained clearly enough.
  3. The proposition assumes a fixed target encoder, whereas LeWM is trained end to end. During FAPI training, the anchor itself can therefore change, yet the paper does not adequately discuss this discrepancy or its implications for the theoretical claim.
  4. The evaluation perturbations are similar in nature to the training perturbations: blur, cutout, and repetition versus noise, patch masking, and history masking. Improvements under these conditions are expected outcomes of data augmentation. Moreover, the explicit OOD evaluation is limited to TwoRoom. The method and experiments themselves are sound overall. I would raise my score if the authors could convincingly establish the novelty of their method relative to existing augmentation approaches.
Questions for authors

see weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 1QAU (1/3): Related Work and Contribution

Dear Reviewer 1QAU,

Thank you for your constructive assessment. We have completed matched clean continued-training controls for all three held-out operators on TwoRoom. Below, we clarify the relationship to prior methods, the theoretical scope, and the extent of the empirical evidence.

Weakness 1: Related work and the contribution

The core method is essentially data augmentation that perturbs the input while keeping the target clean—a well-established approach closely related to denoising autoencoders. However, the paper lacks an adequate discussion of related work and does not cite relevant prior studies.

Reply: Thank you for identifying this omission. Corrupted-input/clean-target learning is established in denoising autoencoders. We will cite Vincent et al. (ICML 2008) and expand the discussion of related self-distillation methods, including BYOL and DINO.

In our action-conditioned world-model experiments, FAPI trains degraded visual or historical contexts to predict a clean future representation while retaining the recorded action inputs. The objective concerns stability of future predictions rather than reconstruction of the corrupted observation; context representations themselves need not be identical.

Our contribution is the diagnosis of predictive brittleness and the controlled evaluation of context-degradation adaptation for prediction and planning. The completed matched controls below demonstrate planning benefits beyond additional clean training. Clean-CT denotes continued self-supervised training using the backbone's native training recipe, with the additional FAPI context corruptions disabled.

Table: Mean planning success (%) across PushT, OGB-Cube, TwoRoom, and DMC.

ModelCleanGaussianPatchHistory
Frozen LeWM86.049.538.580.0
Clean-CT LeWM85.546.535.575.0
FAPI+LeWM90.061.541.585.5
Frozen Sub-JEPA80.540.026.076.0
Clean-CT Sub-JEPA81.040.526.078.5
FAPI+Sub-JEPA89.557.034.586.0

Gaussian noise uses σo=0.10\sigma_o=0.10; patch and history masking ratios are 0.300.30. Each per-task result uses 50 episodes. Fine-tuned denotes five-epoch clean continued self-supervised training with the same settings and budget as FAPI.

We have clarified the contribution and its relationship to prior denoising approaches.

AuthorsOpenReview ↗

Response to Reviewer 1QAU (2/3): Theory and the Moving Encoder

Weakness 2: The role and implications of Proposition 3.1

The theoretical framing of Proposition 3.1 is overstated, while the result and its implications are not explained clearly enough.

Reply: Thank you for highlighting this ambiguity. Proposition 3.1 explains how regression to a shared future anchor controls disagreement between predictions from transformed contexts. At a fixed training snapshot, small regression risk bounds average predictive disagreement. This does not guarantee monotonic improvement during training, prevent collapse, or establish planning benefits.

We also distinguish predictive drift from the excess-risk brittleness gap. For clean prediction pp, corrupted prediction qq, and their shared target yy,

\frac{1}{d}\mathbb{E}\left\lVert q-p\right\rVert_2^2 +\frac{2}{d}\mathbb{E}\left\langle q-p,p-y\right\rangle.$$ The cross term means that a smaller brittleness gap alone does not imply smaller predictive drift. We will clarify these distinctions in Section 3.1 and Appendix K, and remove the claim that the bound establishes the empirical redundancy of pairwise consistency regularization. ### Weakness 3: A changing encoder during end-to-end training > The proposition assumes a fixed target encoder, whereas LeWM is trained end to end. During FAPI training, the anchor itself can therefore change, yet the paper does not adequately discuss this discrepancy or its implications for the theoretical claim. **Reply:** Thank you for highlighting the ambiguity in the fixed-encoder assumption. A forward prediction or multi-step rollout uses a single parameter setting, while end-to-end training updates the encoder across optimization iterations. We have revised Proposition 3.1 to state the comparison at each training snapshot $s$. For the same recorded trajectory $q$, all context transformations share the clean target $z_s^\star(q)$ computed with the current target encoder. Define $$\mathcal{L}_s := \frac{1}{d}\mathbb{E}_{q,\xi} \left\lVert \hat z_s^\xi(q)-z_s^\star(q)\right\rVert_2^2,$$ and $$D_s := \frac{1}{d}\mathbb{E}_{q,\xi,\xi'} \left\lVert \hat z_s^\xi(q)-\hat z_s^{\xi'}(q) \right\rVert_2^2.$$ For independent transformations drawn from the same distribution, the conditional variance identity yields $$D_s\leq 2\mathcal{L}_s.$$ This requires a shared target within the current snapshot and allows the encoder to change between snapshots. After an update, both the anchor and the conditional-uncertainty term can change. The proposition therefore does not control anchor drift across iterations or guarantee convergence during joint training. We will clarify this scope in Section 3.1 and Appendix K and document encoder updates and gradient flow in Appendix B.3.
AuthorsOpenReview ↗

Response to Reviewer 1QAU (3/3): Held-Out Evaluation and Ablations

Weakness 4: Held-out operators and the scope of generalization

The evaluation perturbations are similar in nature to the training perturbations: blur, cutout, and repetition versus noise, patch masking, and history masking. Improvements under these conditions are expected outcomes of data augmentation. Moreover, the explicit OOD evaluation is limited to TwoRoom. The method and experiments themselves are sound overall. I would raise my score if the authors could convincingly establish the novelty of their method relative to existing augmentation approaches.

Reply: Thank you for this observation. We have completed matched clean continued-training controls for Gaussian blur, block cutout, and context repetition. These operators were excluded from adaptation of the evaluated checkpoints. Clean-CT denotes continued self-supervised training using the backbone's native training recipe, with the additional FAPI context corruptions disabled.

FAPI exceeds the Clean-CT control in all twelve settings, with average gains of 13.0, 9.5, and 5.5 percentage points under blur, cutout, and repetition, respectively. These transfer benefits persist when the additional training budget is matched.

Table: Held-out-operator planning success (%) for LeWM on TwoRoom.

OperatorLevelFrozenClean-CTFAPI
Gaussian blurL1L_186.082.094.0
Gaussian blurL2L_284.078.092.0
Gaussian blurL3L_378.076.088.0
Gaussian blurL4L_472.070.084.0
Block cutoutL1L_188.084.092.0
Block cutoutL2L_276.076.088.0
Block cutoutL3L_378.074.084.0
Block cutoutL4L_466.064.072.0
Context repetitionL1L_188.088.094.0
Context repetitionL2L_288.086.094.0
Context repetitionL3L_386.086.090.0
Context repetitionL4L_482.082.086.0

Blur settings (k,σb)(k,\sigma_b) are (3,0.5)(3,0.5), (5,1.0)(5,1.0), (9,2.0)(9,2.0), and (13,3.0)(13,3.0). Cutout sizes are 50×5050\times50, 71×7171\times71, 100×100100\times100, and 123×123123\times123 on 224×224224\times224 observations. Context repetition probabilities are pr=0.05p_r=0.05, 0.100.10, 0.200.20, and 0.300.30 for L1L_1 through L4L_4, respectively.

The component-removal study uses fixed evaluation conditions across variants. FAPI+LeWM improves clean and combined-degradation success over Clean-CT LeWM by 8.0 and 14.0 percentage points. Removing noise, patch masking, or history masking during adaptation lowers observed combined-degradation success by 2.0, 6.0, and 4.0 points, respectively.

Table: TwoRoom component-removal planning success (%).

Model / variantCleanCombined degradation
Frozen LeWM88.082.0
Clean-CT LeWM88.078.0
FAPI+LeWM96.092.0
FAPI+LeWM w/o noise92.090.0
FAPI+LeWM w/o patch92.086.0
FAPI+LeWM w/o history90.088.0

Combined degradation uses Gaussian noise σo=0.05\sigma_o=0.05, patch masking ratio 0.100.10, and history masking ratio 0.050.05. Components are removed only during adaptation; evaluation conditions are fixed.

These results support benefits beyond additional clean training and transfer to held-out operators within related corruption families on TwoRoom. We will limit the generalization claim to this evidence. The contribution in predictive-brittleness diagnosis and future-anchored adaptation for planning will be distinguished from established augmentation principles.

Thank you again for your time and constructive feedback. We welcome any further questions.

Sincerely,
Authors

Reviewer_o9H8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the robustness of Joint-Embedding Predictive Architecture (JEPA) world models when visual observations or historical frames are degraded at test time. The authors show that strong planning performance under clean observations does not guarantee stability under perturbations. For example, on PushT, LeWorldModel (LeWM) can collapse in planning under mild (Gaussian) observation noise. To quantify this sensitivity, the authors define a "predictive brittleness gap", which is measured as the excess latent MSE incurred when predicting a clean future target from a perturbed context rather than a clean context. They propose Future-Anchored Predictive Invariance (FAPI), a short fine-tuning stage that applies visual corruptions, patch masking, and history masking to the conditioning context while keeping the action sequence and the clean future target fixed as a shared anchor. Applying a few epochs of FAPI adaptation on top of pre-trained LeWM, Sub-JEPA, V-JEPA, and I-JEPA checkpoints improves degraded planning success on continuous control tasks and reduces prediction drift on Something-Something-v2 and ImageNet.

Critical strengths

The main strength of this paper is the novel perspective on the robustness of JEPA-style models, and an empirical evaluation of their predictive "brittleness" (Section 2 and Figure 1). Showing on PushT that the model with the highest clean success rate (LeWM) collapses much more under mild observation noise than models with lower clean accuracy (Sub-JEPA and JEPA-WM) is a useful and practical observation. The proposed future-anchored adaptation recipe is also simple, requires no architectural changes, and is evaluated across continuous control, video prediction, and image prediction.

Critical weaknesses

I have a few general concerns:

  1. Omission of stronger 2025–2026 baselines (especially JEPA-WM from Figure 1) and missing clean fine-tuning control in Tables 1 to 3:

    • In Figure 1a, the authors evaluate four world models on PushT (Sub-JEPA, DINO-WM, JEPA-WM, and LeWM) and show that under sigma_o = 0.10 noise, unadapted JEPA-WM (Terver et al., 2026) is by far the most robust model, achieving more than double the planning success rate of the adapted FAPI+LeWM and FAPI+Sub-JEPA models reported later in Tables 1 and 3. However, both JEPA-WM and DINO-WM seem to be excluded from all subsequent experiments (Tables 1 to 3). Similarly, for video prediction in Table 4, the paper cites V-JEPA 2 (2025) in the introduction to motivate modern video world models, but only evaluates the older V-JEPA (2024) checkpoint.
  2. On the methodology side, perturbing or masking the online context while anchoring regression to a clean, unperturbed target representation is already standard in standard self-distillation and JEPA frameworks. So the proposed "Future-Anchored Predictive Invariance" (FAPI) essentially amounts to applying standard asymmetric data augmentation during world-model fine-tuning.

  3. In Section 4, the authors note the importance of comparing against a baseline fine-tuned without context perturbations to separate the effect of FAPI from additional adaptation epochs. This control is included for V-JEPA and I-JEPA in Table 4. However, it is missing from all world-model planning tables (Tables 1 to 3), where 5-epoch adapted FAPI models are compared directly against frozen LeWM and Sub-JEPA. Since FAPI also improves clean planning success, it is unclear how much of the gain comes from the extra fine-tuning epochs rather than the context perturbations.

Questions for authors
  1. Why were JEPA-WM (which outperforms your adapted models by more than two-fold under observation noise in Figure 1a) and DINO-WM omitted from Tables 1 to 3, and how does FAPI perform when applied to JEPA-WM and V-JEPA 2?

  2. Clean fine-tuning control: How do LeWM and Sub-JEPA perform in Tables 1 and 3 when fine-tuned for 5 epochs with the same optimizer schedule on clean data alone?

  3. I also wonder: how does the analysis performed in this paper relate to the standard bias/variance tradeoff in statistical learning?This seems fundamentally closely related to the paper's framework, yet does not seem to be mentioned in the text.

6 comments
AuthorsOpenReview ↗

Response to Reviewer o9H8 (1/3): Stronger Backbones

Dear Reviewer o9H8,

Thank you for your constructive comments and for recognizing the robustness diagnosis. We have completed matched clean continued-training controls, extended the PushT comparisons to JEPA-WM and DINO-WM, and added V-JEPA 2 classification results.

Weakness 1: Stronger backbones and missing controls

Omission of stronger 2025–2026 baselines (especially JEPA-WM from Figure 1) and missing clean fine-tuning control in Tables 1 to 3:

In Figure 1a, the authors evaluate four world models on PushT (Sub-JEPA, DINO-WM, JEPA-WM, and LeWM) and show that under σo=0.10\sigma_o=0.10 noise, unadapted JEPA-WM (Terver et al., 2026) is by far the most robust model, achieving more than double the planning success rate of the adapted FAPI+LeWM and FAPI+Sub-JEPA models reported later in Tables 1 and 3. However, both JEPA-WM and DINO-WM seem to be excluded from all subsequent experiments (Tables 1 to 3). Similarly, for video prediction in Table 4, the paper cites V-JEPA 2 (2025) in the introduction to motivate modern video world models, but only evaluates the older V-JEPA (2024) checkpoint.

Reply: Thank you for highlighting these comparisons. We have completed matched Fine-tuned controls for LeWM and Sub-JEPA on all four benchmarks. They use the same training budget and optimization settings as FAPI, with only the additional FAPI context corruptions disabled. FAPI retains improvements over these controls. Clean-CT denotes continued self-supervised training using the backbone's native training recipe, with the additional FAPI context corruptions disabled.

We have also completed Frozen/Clean-CT/FAPI comparisons for JEPA-WM and DINO-WM on PushT.

Table: PushT planning success (%). Gaussian noise uses σo=0.10\sigma_o=0.10.

ModelCleanGaussian
Frozen JEPA-WM88.034.0
Clean-CT JEPA-WM84.028.0
FAPI+JEPA-WM96.040.0
Frozen DINO-WM92.06.0
Clean-CT DINO-WM90.06.0
FAPI+DINO-WM96.010.0

Relative to the Clean-CT controls, FAPI improves clean and Gaussian success by 12.0 and 12.0 percentage points for JEPA-WM, and 6.0 and 4.0 points for DINO-WM, respectively. FAPI+JEPA-WM achieves 40.0% Gaussian success, compared with 34.0% for Frozen JEPA-WM, the strongest unadapted backbone highlighted in Figure 1 of the submission.

We have additionally completed a separate V-JEPA 2 evaluation on SSv2 using frozen-encoder linear probing.

Table: V-JEPA 2 linear-probe Top-1 accuracy (%) on SSv2.

ModelCleanGaussianHistoryCombined
Frozen V-JEPA 251.548.150.147.8
Clean-CT V-JEPA 250.646.950.146.3
FAPI+V-JEPA 253.952.352.351.5

Gaussian noise uses σo=0.02\sigma_o=0.02, history masking uses pt=0.05p_t=0.05, and Combined applies both. These are our measured V-JEPA 2 results, separate from the first-generation V-JEPA evaluation.

Compared with Clean-CT V-JEPA 2, FAPI improves Top-1 accuracy by 3.3, 5.4, 2.2, and 5.2 percentage points under clean, Gaussian, history-masked, and combined conditions, respectively. It also exceeds Frozen V-JEPA 2 under all four conditions.

Question 1: Adaptation of JEPA-WM, DINO-WM, and V-JEPA 2

Why were JEPA-WM (which outperforms your adapted models by more than two-fold under observation noise in Figure 1a) and DINO-WM omitted from Tables 1 to 3, and how does FAPI perform when applied to JEPA-WM and V-JEPA 2?

Reply: Thank you for raising this question. The initial adaptation study focused on LeWM and Sub-JEPA. Including JEPA-WM and DINO-WM in the robustness diagnosis without evaluating their adaptation left a gap, which we have now addressed.

The tables above report the completed comparisons. Under Gaussian noise on PushT, FAPI+JEPA-WM achieves 40.0%, compared with 34.0% for Frozen and 28.0% for Fine-tuned JEPA-WM. FAPI+DINO-WM achieves 10.0%, compared with 6.0% for both controls.

For V-JEPA 2, FAPI achieves 53.9%, 52.3%, 52.3%, and 51.5% SSv2 Top-1 accuracy under clean, Gaussian, history-masked, and combined conditions. These results extend the empirical support to the additional evaluated backbones.

AuthorsOpenReview ↗

Response to Reviewer o9H8 (2/3): Methodology and Bias-Variance

Weakness 2: Relationship to asymmetric augmentation

On the methodology side, perturbing or masking the online context while anchoring regression to a clean, unperturbed target representation is already standard in standard self-distillation and JEPA frameworks. So the proposed "Future-Anchored Predictive Invariance" (FAPI) essentially amounts to applying standard asymmetric data augmentation during world-model fine-tuning.

Reply: Thank you for raising this connection. We acknowledge the precedents in asymmetric augmentation and clean-target prediction. In our world-model experiments, FAPI addresses stability of action-conditioned future predictions under degraded observation contexts.

For each recorded trajectory, FAPI perturbs visual evidence or masks preceding observations while retaining the recorded actions and corresponding clean future target. The desired invariance concerns what the model predicts about the future; degraded contexts may contain different information and need not have identical representations. We inherit JEPA's prediction architecture and introduce additional context degradations to address the predictive brittleness identified in Section 2.

This formulation also constrains the transformations: they must retain predictive information about the recorded future. Keeping the target clean alone is insufficient when a corruption destroys that information.

The completed matched controls show planning benefits beyond clean continued training with the same budget. For example, under Gaussian noise, mean success increases from 46.5% to 61.5% for LeWM and from 40.5% to 57.0% for Sub-JEPA. We will clarify this contribution and distinguish the additional transformations from each backbone's native training recipe.

Question 3: Connection to the bias-variance tradeoff

I also wonder: how does the analysis performed in this paper relate to the standard bias/variance tradeoff in statistical learning? This seems fundamentally closely related to the paper's framework, yet does not seem to be mentioned in the text.

Reply: Thank you for suggesting this connection. FAPI's context transformations provide regularization by encouraging stable future predictions. However, overly strong corruptions can remove predictive information and, for a fixed representation, increase irreducible conditional uncertainty. Stronger invariance therefore need not improve prediction or planning.

This connects our framework to regularization and bias-variance considerations, while also accounting for information loss. Our predictive drift measures variation across context transformations for a fixed model, whereas classical estimator variance measures variation across sampled training datasets. Our analysis complements the classical decomposition rather than estimating its terms. We have made this connection explicit in Section 3.1.

AuthorsOpenReview ↗

Response to Reviewer o9H8 (3/3): Clean-Training Controls

Weakness 3: Separating context corruption from additional training

In Section 4, the authors note the importance of comparing against a baseline fine-tuned without context perturbations to separate the effect of FAPI from additional adaptation epochs. This control is included for V-JEPA and I-JEPA in Table 4. However, it is missing from all world-model planning tables (Tables 1 to 3), where 5-epoch adapted FAPI models are compared directly against frozen LeWM and Sub-JEPA. Since FAPI also improves clean planning success, it is unclear how much of the gain comes from the extra fine-tuning epochs rather than the context perturbations.

Reply: Thank you for identifying this missing control. We have completed five-epoch clean continued-training controls for LeWM and Sub-JEPA on all four benchmarks. Clean-CT denotes continued self-supervised training using the backbone's native training recipe, with the additional FAPI context corruptions disabled.

These controls match FAPI in initialization, training data, optimization schedule, update count, trainable modules, and native regularization. Only the additional FAPI context corruptions are disabled. Planning configurations and evaluation initializations are matched.

Table: LeWM planning success (%).

ConditionModelPushTOGB-CubeTwoRoomDMCMean
CleanFrozen96.074.088.086.086.0
CleanClean-CT94.074.088.086.085.5
CleanFAPI98.078.096.088.090.0
GaussianFrozen4.046.080.068.049.5
GaussianClean-CT4.042.076.064.046.5
GaussianFAPI16.052.092.086.061.5
PatchFrozen2.042.080.030.038.5
PatchClean-CT2.040.074.026.035.5
PatchFAPI6.046.084.030.041.5
HistoryFrozen90.066.082.082.080.0
HistoryClean-CT86.058.080.076.075.0
HistoryFAPI96.072.092.082.085.5

Table: Sub-JEPA planning success (%).

ConditionModelPushTOGB-CubeTwoRoomDMCMean
CleanFrozen92.072.096.062.080.5
CleanClean-CT90.074.094.066.081.0
CleanFAPI100.080.098.080.089.5
GaussianFrozen8.044.056.052.040.0
GaussianClean-CT8.050.054.050.040.5
GaussianFAPI14.062.074.078.057.0
PatchFrozen6.042.038.018.026.0
PatchClean-CT8.044.034.018.026.0
PatchFAPI10.050.048.030.034.5
HistoryFrozen90.064.090.060.076.0
HistoryClean-CT94.064.088.068.078.5
HistoryFAPI100.076.096.072.086.0

Gaussian noise uses σo=0.10\sigma_o=0.10; patch and history masking ratios are 0.300.30. Each success rate uses 50 episodes per task and condition. Mean averages the four tasks.

Relative to the Clean-CT controls, FAPI improves mean success by 4.5, 15.0, 6.0, and 10.5 percentage points for LeWM, and 8.5, 16.5, 8.5, and 7.5 points for Sub-JEPA, under clean, Gaussian, patch-masking, and history-masking conditions, respectively. The clean and corrupted gains persist when the additional training budget is matched.

Question 2: Five-epoch clean-training results

Clean fine-tuning control: How do LeWM and Sub-JEPA perform in Tables 1 and 3 when fine-tuned for 5 epochs with the same optimizer schedule on clean data alone?

Reply: Thank you for requesting these results. Averaged across the four benchmarks, Clean-CT LeWM achieves 85.5%, 46.5%, 35.5%, and 75.0% success under clean, Gaussian, patch-masking, and history-masking conditions, respectively. The corresponding results for Clean-CT Sub-JEPA are 81.0%, 40.5%, 26.0%, and 78.5%.

The tables above provide all per-task comparisons. FAPI exceeds these matched controls in mean success under all four conditions for both backbones.

Thank you again for your time and constructive feedback. We welcome any further questions.

Sincerely,
Authors

Reviewer_EmLzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the sensitivity of JEPA-based models to asymmetric observation perturbations eg Gaussian noise, patch masking, and dropping frames in the observation history (the prediction target is not perturbed). The authors define "predictive brittleness gap" as the difference in latent MSE between predicting a clean future target from a perturbed context versus a clean context. To reduce this gap, they propose FAPI which continues pretraining a JEPA checkpoint while applying to data the aforementioned corruptions in the context branch. Experiments are conducted by adapting LeWM and Sub-JEPA for additional 5 epochs on four control tasks, and fine-tuning V-JEPA (on SSv2) and I-JEPA (on ImageNet) while evaluating latent prediction error and feature similarity under corruption (without reporting downstream performance).

Critical strengths
  1. Evaluating world models under noisy observations and missing history frames and its effect on downstream planning is a reasonable direction to explore.
Critical weaknesses
  1. The method is asymmetric data augmentation which has precedent in SSL literature (BYOL [1] and DINO [2]) applied during a few extra epochs of self-supervised training. Calling it "Future-Anchored Predictive Invariance" overstates the novelty.

  2. Proposition 3.1 is a trivial application of the inequality ∥a−b∥22≤2∥a∥22+2∥b∥22\Vert{}a - b\Vert{}_2^2 \le 2\Vert{}a\Vert{}_2^2 + 2\Vert{}b\Vert{}_2^2 and it does not take into account the assumption that the target encoder is a moving target during fine-tuning.

  3. The evaluation on V-JEPA and I-JEPA is flawed. The paper reports only latent prediction loss and cosine similarity with no downstream task metrics (e.g., linear probing or fine-tuned classification accuracy). In SSL, pred error can be trivially decreased by continual fine-tuning and possibly collapse without any downstream benefit.

  4. Fine-tuning in the paper is continued SSL training (wrong terminology) for more epochs. The planning experiments in Tables 1 to 3 omit the baselines where the method is "fine-tuned" on clean trajectories and only report the original clean checkpoints performance alongside their fine-tuning on corrupted trajectory which makes the comparisons unfair.

[1] Bootstrap your own latent: A new approach to self-supervised Learning

[2] Emerging Properties in Self-Supervised Vision Transformers

Questions for authors
  1. Can you provide downstream metrics for V-JEPA and I-JEPA for all methods?

  2. Can you provide eval metrics for planning baselines where fine-tuning is also done on clean trajectories?

5 comments
AuthorsOpenReview ↗

Response to Reviewer EmLz (1/4): Contribution and Prior Work

Dear Reviewer EmLz,

Thank you for your detailed comments. We have added downstream classification results and matched clean continued-training controls for planning. Below, we clarify our contribution and the scope of Proposition 3.1.

Weakness 1: Relationship to asymmetric augmentation

The method is asymmetric data augmentation which has precedent in SSL literature (BYOL [1] and DINO [2]) applied during a few extra epochs of self-supervised training. Calling it "Future-Anchored Predictive Invariance" overstates the novelty.

Reply: Thank you for your comment. We acknowledge the precedents in asymmetric self-supervised learning. Our contribution lies in formulating and empirically studying predictive brittleness in world-model planning, and in developing the future-anchored adaptation approach. We clarify the distinctions below.

(1) Different goal. BYOL and DINO relate views of the same image to learn visual representations. In our world-model experiments, we study whether degraded observations or missing history change the future predicted under a given action sequence. Such predictive drift can change a planner's ranking of candidate trajectories and ultimately its selected action.

(2) Different criterion. Not every transformation is appropriate for future prediction. For example, masking part of an observation history may remove information needed to infer motion, even though the degraded history remains associated with the same recorded future. We therefore formulate predictive admissibility in terms of whether a degraded context retains predictive value beyond an observation-null baseline, given the actions. The recoverability diagnostics in Appendix D examine this property empirically.

(3) Different target from image-view matching. For each transition, FAPI degrades the observation-derived context while retaining its recorded actions and the clean future observation used to construct the target representation. Different degraded contexts from that transition are trained toward targets derived from the same recorded future under the same actions. Compared with BYOL and DINO, the target is derived from an observed future rather than another view of the same image. This encourages stability of future predictions without imposing equality on intermediate context representations.

(4) Evidence for planning. Our matched controls compare FAPI with continued training on clean trajectories using the same initialization, data, optimization settings, and adaptation budget. Under Gaussian noise (σo=0.10\sigma_o=0.10), FAPI increases mean planning success across four tasks from 46.5% to 61.5% for LeWM and from 40.5% to 57.0% for Sub-JEPA. These results support the benefit of FAPI's context-degradation design relative to clean continued training.

References

[1] Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning.

[2] Emerging Properties in Self-Supervised Vision Transformers.

AuthorsOpenReview ↗

Response to Reviewer EmLz (2/4): Proposition 3.1

Weakness 2: Proposition 3.1 and the moving target

Proposition 3.1 is a trivial application of the inequality ∥a−b∥22≤2∥a∥22+2∥b∥22\lVert a-b\rVert_2^2 \leq 2\lVert a\rVert_2^2 + 2\lVert b\rVert_2^2 and it does not take into account the assumption that the target encoder is a moving target during fine-tuning.

Reply: Thank you for this observation. The purpose of Proposition 3.1 is to clarify why predicting a shared future target can encourage agreement between predictions from different perturbed contexts, without an explicit pairwise consistency loss.

We agree that the fixed-encoder assumption obscures the setting in which the target representation changes during adaptation. The agreement bound can be stated at each training snapshot without assuming that the target encoder remains fixed throughout training. We have revised Proposition 3.1 accordingly.

Let ss index a training snapshot, zs⋆(q)z_s^\star(q) be the clean future target for trajectory qq, and z^sξ(q)\hat z_s^\xi(q) be the prediction from a context transformed by ξ\xi. Define zˉs(q)=Eξ[z^sξ(q)]\bar z_s(q)=\mathbb{E}_{\xi}[\hat z_s^\xi(q)], the prediction risk

:= \frac{1}{d}\mathbb{E}_{q,\xi} \left\lVert \hat z_s^\xi(q)-z_s^\star(q)\right\rVert_2^2,$$ and the expected pairwise disagreement $$D_s := \frac{1}{d}\mathbb{E}_{q,\xi,\xi'} \left\lVert \hat z_s^\xi(q)-\hat z_s^{\xi'}(q)\right\rVert_2^2.$$ For independently sampled $\xi,\xi'$ from the same transformation distribution, the conditional variance identity gives $$D_s =2\mathcal{L}_s -\frac{2}{d}\mathbb{E}_{q} \left\lVert \bar z_s(q)-z_s^\star(q)\right\rVert_2^2 \leq 2\mathcal{L}_s.$$ Thus, at any snapshot, a small prediction risk bounds average disagreement when the transformed contexts share the same clean target. The encoder and target may change between snapshots. This statement does not characterize moving-target dynamics or guarantee convergence or the absence of representation collapse. We have updated Proposition 3.1 in the revised manuscript to state this scope explicitly.
AuthorsOpenReview ↗

Response to Reviewer EmLz (3/4): Downstream Metrics

Weakness 3: Downstream representation evaluation

The evaluation on V-JEPA and I-JEPA is flawed. The paper reports only latent prediction loss and cosine similarity with no downstream task metrics (e.g., linear probing or fine-tuned classification accuracy). In SSL, pred error can be trivially decreased by continual fine-tuning and possibly collapse without any downstream benefit.

Reply: Thank you for highlighting the need for downstream evaluation. Our submitted I-JEPA and V-JEPA experiments assess predictive robustness under degraded contexts. We agree that prediction loss and feature consistency alone do not establish downstream representation utility.

We complement these diagnostics with frozen-encoder linear probing, reporting Top-1 accuracy on ImageNet-1K for I-JEPA and Something-Something-v2 for V-JEPA. We follow the respective original linear-probing protocols. The clean Frozen baseline scores are taken from the corresponding papers. Fine-tuned denotes continued self-supervised adaptation without the additional FAPI context corruptions.

For each adapted checkpoint, a linear classifier is trained on the clean labeled training split using matched settings. The encoder and classifier remain fixed during clean and corrupted evaluation. Clean-CT denotes continued self-supervised training using the backbone's native training recipe, with the additional FAPI context corruptions disabled.

Table: Prediction and consistency diagnostics.

ModelClean lossCorrupt lossCosineRelative ℓ2\ell_2
Frozen V-JEPA0.5460.5700.9290.320
Clean-CT V-JEPA0.5880.5940.9770.143
FAPI+V-JEPA0.4850.4870.9940.062
Frozen I-JEPA0.1910.2230.8610.493
Clean-CT I-JEPA0.1930.2230.8670.481
FAPI+I-JEPA0.1870.2120.8880.437

For V-JEPA, corrupted prediction loss averages Gaussian noise (σo=0.02\sigma_o=0.02) and history masking (pt=0.05p_t=0.05). For I-JEPA, Gaussian noise uses σo=0.20\sigma_o=0.20.

Table: Frozen-encoder linear-probe Top-1 accuracy (%).

ModelCleanGaussianHistoryCombined
Frozen V-JEPA50.147.848.146.9
Clean-CT V-JEPA46.946.446.345.5
FAPI+V-JEPA52.351.551.550.6
Frozen I-JEPA79.377.1----
Clean-CT I-JEPA78.677.2----
FAPI+I-JEPA79.877.8----

V-JEPA uses σo=0.02\sigma_o=0.02 for Gaussian noise and pt=0.05p_t=0.05 for history masking; Combined applies both. I-JEPA uses σo=0.20\sigma_o=0.20. The datasets are SSv2 for V-JEPA and ImageNet-1K for I-JEPA. "--" denotes a condition that is not applicable.

These evaluations assess clean classification performance and robustness relative to the Clean-CT controls. Feature-variance and effective-rank analyses provide complementary representation diagnostics.

Question 1: Downstream metrics for all methods

Can you provide downstream metrics for V-JEPA and I-JEPA for all methods?

Reply: Thank you for highlighting the need for downstream evaluation. We have added frozen-encoder linear-probe Top-1 classification results for the Frozen, Fine-tuned, and FAPI models, as shown in the classification table above.

The evaluations cover clean and Gaussian-corrupted inputs on ImageNet-1K for I-JEPA, and clean, Gaussian, history-masked, and combined conditions on SSv2 for V-JEPA. For the Fine-tuned and FAPI checkpoints within each backbone, probes use matched training settings, and each trained probe remains fixed during clean and corrupted evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.