- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to make JEPA world models more reliable when observations contain noise, occlusion, or missing history.
- Research problem: Strong prediction accuracy under clean observations does not guarantee robustness to corrupted inputs. The authors introduce the predictive brittleness gap, which measures the additional future-prediction error caused by degraded observations.
- Core method: The proposed FAPI (Future-Anchored Predictive Invariance) adds noise, occlusion, or history masking to the observation context while keeping the action inputs and clean prediction target unchanged. Different degraded contexts are trained to predict the same future representation. FAPI fine-tunes existing JEPA models without changing their architecture or adding an explicit consistency loss.
- Experimental results: Across four control benchmarks, FAPI improves average planning success under Gaussian noise from 49.5% to 61.5% for LeWM and from 40% to 57% for Sub-JEPA. Experiments with V-JEPA on video prediction and I-JEPA on image representation prediction also show lower prediction errors and greater stability.
- Contribution and limitations: The paper provides an approach to evaluating and improving JEPA robustness by learning which observation changes should leave predictions unchanged. However, the degraded input must retain enough predictive information; severe information loss caused by occlusion remains difficult to address.
Critical strengths
-
The paper presents a clear problem formulation and a well-focused motivation. The brittleness gap is simple and intuitive, and Figure 1 supports the conclusion that “clean predictive accuracy and predictive stability capture distinct properties of a learned world model.”
-
The method is simple and broadly applicable, supported by substantial implementation and experimental effort. The authors demonstrate its effectiveness across multiple architectures and benchmarks.
Critical weaknesses
- The core method is essentially data augmentation that perturbs the input while keeping the target clean—a well-established approach closely related to denoising autoencoders. However, the paper lacks an adequate discussion of related work and does not cite relevant prior studies.
- The theoretical framing of Proposition 3.1 is overstated, while the result and its implications are not explained clearly enough.
- The proposition assumes a fixed target encoder, whereas LeWM is trained end to end. During FAPI training, the anchor itself can therefore change, yet the paper does not adequately discuss this discrepancy or its implications for the theoretical claim.
- The evaluation perturbations are similar in nature to the training perturbations: blur, cutout, and repetition versus noise, patch masking, and history masking. Improvements under these conditions are expected outcomes of data augmentation. Moreover, the explicit OOD evaluation is limited to TwoRoom. The method and experiments themselves are sound overall. I would raise my score if the authors could convincingly establish the novelty of their method relative to existing augmentation approaches.
Questions for authors
see weaknesses.
8 comments
Response to Reviewer 1QAU (2/3): Theory and the Moving Encoder
Weakness 2: The role and implications of Proposition 3.1
The theoretical framing of Proposition 3.1 is overstated, while the result and its implications are not explained clearly enough.
Reply: Thank you for highlighting this ambiguity. Proposition 3.1 explains how regression to a shared future anchor controls disagreement between predictions from transformed contexts. At a fixed training snapshot, small regression risk bounds average predictive disagreement. This does not guarantee monotonic improvement during training, prevent collapse, or establish planning benefits.
We also distinguish predictive drift from the excess-risk brittleness gap. For clean prediction , corrupted prediction , and their shared target ,
\frac{1}{d}\mathbb{E}\left\lVert q-p\right\rVert_2^2 +\frac{2}{d}\mathbb{E}\left\langle q-p,p-y\right\rangle.$$ The cross term means that a smaller brittleness gap alone does not imply smaller predictive drift. We will clarify these distinctions in Section 3.1 and Appendix K, and remove the claim that the bound establishes the empirical redundancy of pairwise consistency regularization. ### Weakness 3: A changing encoder during end-to-end training > The proposition assumes a fixed target encoder, whereas LeWM is trained end to end. During FAPI training, the anchor itself can therefore change, yet the paper does not adequately discuss this discrepancy or its implications for the theoretical claim. **Reply:** Thank you for highlighting the ambiguity in the fixed-encoder assumption. A forward prediction or multi-step rollout uses a single parameter setting, while end-to-end training updates the encoder across optimization iterations. We have revised Proposition 3.1 to state the comparison at each training snapshot $s$. For the same recorded trajectory $q$, all context transformations share the clean target $z_s^\star(q)$ computed with the current target encoder. Define $$\mathcal{L}_s := \frac{1}{d}\mathbb{E}_{q,\xi} \left\lVert \hat z_s^\xi(q)-z_s^\star(q)\right\rVert_2^2,$$ and $$D_s := \frac{1}{d}\mathbb{E}_{q,\xi,\xi'} \left\lVert \hat z_s^\xi(q)-\hat z_s^{\xi'}(q) \right\rVert_2^2.$$ For independent transformations drawn from the same distribution, the conditional variance identity yields $$D_s\leq 2\mathcal{L}_s.$$ This requires a shared target within the current snapshot and allows the encoder to change between snapshots. After an update, both the anchor and the conditional-uncertainty term can change. The proposition therefore does not control anchor drift across iterations or guarantee convergence during joint training. We will clarify this scope in Section 3.1 and Appendix K and document encoder updates and gradient flow in Appendix B.3.Response to Reviewer 1QAU (3/3): Held-Out Evaluation and Ablations
Weakness 4: Held-out operators and the scope of generalization
The evaluation perturbations are similar in nature to the training perturbations: blur, cutout, and repetition versus noise, patch masking, and history masking. Improvements under these conditions are expected outcomes of data augmentation. Moreover, the explicit OOD evaluation is limited to TwoRoom. The method and experiments themselves are sound overall. I would raise my score if the authors could convincingly establish the novelty of their method relative to existing augmentation approaches.
Reply: Thank you for this observation. We have completed matched clean continued-training controls for Gaussian blur, block cutout, and context repetition. These operators were excluded from adaptation of the evaluated checkpoints. Clean-CT denotes continued self-supervised training using the backbone's native training recipe, with the additional FAPI context corruptions disabled.
FAPI exceeds the Clean-CT control in all twelve settings, with average gains of 13.0, 9.5, and 5.5 percentage points under blur, cutout, and repetition, respectively. These transfer benefits persist when the additional training budget is matched.
Table: Held-out-operator planning success (%) for LeWM on TwoRoom.
| Operator | Level | Frozen | Clean-CT | FAPI |
|---|---|---|---|---|
| Gaussian blur | 86.0 | 82.0 | 94.0 | |
| Gaussian blur | 84.0 | 78.0 | 92.0 | |
| Gaussian blur | 78.0 | 76.0 | 88.0 | |
| Gaussian blur | 72.0 | 70.0 | 84.0 | |
| Block cutout | 88.0 | 84.0 | 92.0 | |
| Block cutout | 76.0 | 76.0 | 88.0 | |
| Block cutout | 78.0 | 74.0 | 84.0 | |
| Block cutout | 66.0 | 64.0 | 72.0 | |
| Context repetition | 88.0 | 88.0 | 94.0 | |
| Context repetition | 88.0 | 86.0 | 94.0 | |
| Context repetition | 86.0 | 86.0 | 90.0 | |
| Context repetition | 82.0 | 82.0 | 86.0 |
Blur settings are , , , and . Cutout sizes are , , , and on observations. Context repetition probabilities are , , , and for through , respectively.
The component-removal study uses fixed evaluation conditions across variants. FAPI+LeWM improves clean and combined-degradation success over Clean-CT LeWM by 8.0 and 14.0 percentage points. Removing noise, patch masking, or history masking during adaptation lowers observed combined-degradation success by 2.0, 6.0, and 4.0 points, respectively.
Table: TwoRoom component-removal planning success (%).
| Model / variant | Clean | Combined degradation |
|---|---|---|
| Frozen LeWM | 88.0 | 82.0 |
| Clean-CT LeWM | 88.0 | 78.0 |
| FAPI+LeWM | 96.0 | 92.0 |
| FAPI+LeWM w/o noise | 92.0 | 90.0 |
| FAPI+LeWM w/o patch | 92.0 | 86.0 |
| FAPI+LeWM w/o history | 90.0 | 88.0 |
Combined degradation uses Gaussian noise , patch masking ratio , and history masking ratio . Components are removed only during adaptation; evaluation conditions are fixed.
These results support benefits beyond additional clean training and transfer to held-out operators within related corruption families on TwoRoom. We will limit the generalization claim to this evidence. The contribution in predictive-brittleness diagnosis and future-anchored adaptation for planning will be distinguished from established augmentation principles.
Thank you again for your time and constructive feedback. We welcome any further questions.
Sincerely,
Authors
Response to Reviewer 1QAU (1/3): Related Work and Contribution
Dear Reviewer 1QAU,
Thank you for your constructive assessment. We have completed matched clean continued-training controls for all three held-out operators on TwoRoom. Below, we clarify the relationship to prior methods, the theoretical scope, and the extent of the empirical evidence.
Weakness 1: Related work and the contribution
Reply: Thank you for identifying this omission. Corrupted-input/clean-target learning is established in denoising autoencoders. We will cite Vincent et al. (ICML 2008) and expand the discussion of related self-distillation methods, including BYOL and DINO.
In our action-conditioned world-model experiments, FAPI trains degraded visual or historical contexts to predict a clean future representation while retaining the recorded action inputs. The objective concerns stability of future predictions rather than reconstruction of the corrupted observation; context representations themselves need not be identical.
Our contribution is the diagnosis of predictive brittleness and the controlled evaluation of context-degradation adaptation for prediction and planning. The completed matched controls below demonstrate planning benefits beyond additional clean training. Clean-CT denotes continued self-supervised training using the backbone's native training recipe, with the additional FAPI context corruptions disabled.
Table: Mean planning success (%) across PushT, OGB-Cube, TwoRoom, and DMC.
Gaussian noise uses σo=0.10; patch and history masking ratios are 0.30. Each per-task result uses 50 episodes. Fine-tuned denotes five-epoch clean continued self-supervised training with the same settings and budget as FAPI.
We have clarified the contribution and its relationship to prior denoising approaches.