acceptodds
Withdrawn from ICLR 2027

Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Joint-embedding predictive architectures (JEPAs) learn world models that predict in a compact latent space rather than in pixels, reducing the pressure to model nuisance appearance. Yet this provides no guarantee against visual perturbations: they can still alter the encoded representation and affect subsequent action-conditioned predictions. Bisimulation captures this requirement precisely: two observations should be treated as the same state only when their action-conditioned consequences agree. Guided by this criterion, we introduce Action-Conditioned Predictive Consistency (ACPC), a diagnostic that measures how far a clean history and a visually perturbed view of it diverge after being rolled forward under the same action sequence. We prove that this divergence bounds perturbation-induced changes in multi-step prediction error and planner cost, i.e., the change in predicted cost caused by a visual perturbation under the same action sequence. Building on pairwise ACPC, we define two complementary measures: Invariance Radius (IR) summarizes clean–perturbed rollout spread, while Separation Rate (SR) checks whether different states remain distinguishable after rollout. Experiments on four visual control tasks show that pairwise ACPC predicts perturbation-induced changes in prediction error and planner cost. On LeWM, the joint IR–SR screen transfers across tasks, and both IR and SR remain informative under blur and resize. PLDM exhibits similar diagnostic trends under a different architecture.

Primary area:
learning on time series and dynamical systems
Keywords:
world models, world model diagnostics, joint-embedding predictive architectures (JEPA), action-conditioned prediction, visual robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JqqjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Action-Conditioned Predictive Consistency (ACPC) for diagnosing sensitivity of JEPA world models to visual perturbations. ACPC measures the distance between multi-step rollouts from clean and perturbed (noisy) histories under the same action sequence. These distances are used to define two metrics: Invariance Radius (IR), which captures sensitivity to perturbations, and Separation Rate (SR), which checks whether different states remain distinguishable, serving as a check against collapse. The authors theoretically show that ACPC bounds perturbation-induced changes in prediction error and planner cost. Experiments with LeWM and PLDM on four control tasks show that IR and SR track success rates across checkpoints trained with different levels of Gaussian-noise augmentation.

Critical strengths

Developing a bisimulation-motivated diagnostic that incorporates the complementary properties of IR and SR and accounts for multi-step dynamics is a meaningful idea.

  1. An invariance metric alone cannot distinguish a robust model from a collapsed one, whereas SR is designed to detect this distinction. In Figure 3, the authors show that under strong perturbations, SR indeed rejects some checkpoints that pass the IR threshold.
  2. For predicting perturbation-induced changes in prediction error, Figure 4 compares a one-step baseline with two extensions: eight-step ACPC under modified actions and eight-step ACPC under recorded actions. The latter achieves the lowest error on all four tasks, indicating the value of capturing the multi-step consequences of the specific action sequence.
Critical weaknesses
  1. The practical utility of the diagnostic has not been demonstrated. IR–SR checkpoint selection is evaluated only on checkpoints that differ in the level of Gaussian-noise augmentation; the PLDM, blur, and resize evaluations use the same augmentation grid. It remains unclear whether the diagnostic is useful for selecting between models with different architectures or training procedures, or under natural changes in background, lighting, or camera pose.
  2. The theoretical contribution is overstated. The propositions are presented as one of the three main contributions, yet both primarily formalize the motivation for the metric. They describe the geometry of the model outputs without exploiting specific properties of the model or the perturbation, and therefore do not establish conditions under which ACPC is small. Also, neither theoretical bound is directly used in the proposed selection procedure.
Questions for authors
  1. Can IR–SR reliably select among checkpoints that differ in architecture, training objective, training duration, or data quality, rather than only in Gaussian-noise augmentation strength?
  2. What concrete practical use case does ACPC/IR–SR support beyond choosing a noise-augmentation level for a known test perturbation?
  3. What exactly is the projection П used for LeWM and PLDM, and how is it obtained in each case?
Reviewer_p4bjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Paper proposes the Action-Conditioned Predictive Consistency (ACPC) measure, which examines if visual perturbations diverge in latent space after rollouts of some action sequence. It also includes Invariance Radius (IR) which is mostly related to ACPC, and also a Separation Rate (SR) to ensure convergence is not purely explained by representing collapse.

Critical strengths

The paper's main strength is the introduction of a novel metric that considers how representation differences change over time. For example, an encoder might have learned to encode a noisy and noise-free version of an observation to roughly the same encoder (which might be desirable), but this could still derail planning if that small difference in encoder space grows over time given an identical action sequence.

Critical weaknesses

Larger weaknesses:

  1. I found the clarity around exactly what the “projection” operator \Pi was held the paper back a lot, I couldn’t understand if it meant a projection back into observation space, or some other type of projection. The paper is motivated by JEPAs, but these are lossy representations, which cannot (in general) project back to pixel space (or there’s generally a one-many mapping), so it’s unclear what is meant here. Line 192 would suggest it a projection into observation space (e.g. pixels), unless Y_a^H is an embedding. Line 216 mentions “fixed goal embedding g” which suggests the projection was not back into observation space, but from embedding space to some other type of (projected) embedding space (since it was used to construct x in Line 213, to compare with g in Line 219 (an equation number would be help there too).
  2. PS: in Line 192 are these the same shape, G_a(E(h)) is a concentration of weighted projections. Is Y_a^H the (single) H’th observation in future, or the full rollout (concatenation) of all H observations in future? The latter would make sense shape wise, but there’s still a difference here between terms weighted by alpha and unweighted term(s) Y_a^H.
  3. Utility of the theoretical results. For example, in Equation 4, there’s the triangle inequality, but how is this useful to the reader, what can/should they do with knowing this?
  4. Line 192: how does this equation (perhaps number it too) make sense? Is it the distance between projected and encoded images and vs raw images (non-projected, non-encoded)?

Smaller weaknesses:

  • L012: “model nuisance appearance” → “model nuisance information such as appearance”?
  • L013: “visual perturbations: they can still alter the encoded representation” - who is “they”, the perturbations? If so, change “they” → “which”
  • L015: “two observations should be treated as the same state only when their action-conditioned consequences agree” is partly true. A truer statement is perhaps “two observations should be treated as the same state only when executing any subsequent action sequence would yield identical rewards”.
  • Line 049: “Invariance Radius (IR)” - why is this called a radius? This is simply the distance between two points, right?
  • Figure 1: for the perturbed image, perhaps use a lower resolution? It took me some time staring at my screen to see how the shapes were perturbed. Only later, when I zoomed in could I see there was some slightly pixel noise, but I found this difficult to see initially. Also I don’t understand the caption “IR summarizes perturbation-induced rollout spread across histories”.
  • Line 084: unclear what “ACPC-based checkpoint screening” is?
  • Line 115: “checkpoint screen” unclear what this is: test if a model checkpoint is “good enough” ? Still unclear what a checkpoint is by Line 152 too: “To summarize ACPC across a checkpoint”.
  • Line 168: “We compare these predictions in the projected latent space” unclear what the projected space is, was this already introduced?
  • Line 168: “planner costs” unclear what this is
  • Line 170: “we assign nonnegative weights” is confusing to read at first since it doesn’t support the premise of “To represent the full H-step rollout with a single vector”. Only later in equation (3) does it follow why this was done. Also any intuition on how to choose alpha values?
  • Line 208: “candidate costs” → “candidates’ predicted costs”
  • Line 209: “For each candidate action sequence a^j” - possibly good to mention where these action trajectories might originate from / be sampled from?
  • Line 213: I don’t understand the use of the projector operator here \Pi that was used previously in sections 3.1 and 3.2, but only that and neglecting the G_a operator. Why use one part but not both?
  • Line 237: “We apply the visual perturbation” could be good to give some examples of what these might be, e.g. synthetic pixel noise, or something more structured or contrastive?
  • Line 239: “s_i, the median projected one-step clean motion over the next H transitions” still unsure what “s_i” is exactly
  • Equation 6: why h_i and a_i are both indexed by i? The same h can (should?) be assessed my multiple action sequences, is that correct? Otherwise is there a particular action sequence a_i associated with this particular state sample h_i?
  • Line 249: “For each eligible anchor i” what makes an anchor eligible?
Questions for authors
  1. How are action sequences chosen? E.g. under line 150 “under the same action sequence”.
  2. How do perturbations generally happen beyond additive standard Gaussian noise?
  3. Where does the projection \Pi come from, first introduced on Line 168?
  4. How choose alpha? (if simply uniform, then can we remove alpha?)
Reviewer_fXM1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes action-conditioned predictive consistency (ACPC), an evaluation metric for JEPA world models that measures the weighted distance between multi-step rollouts from a clean observation context and a visually perturbed version one under the same action sequence. From pairwise ACPC, the authors define two statistics, IR and SR where lower IR indicates less sensitivity to the visual perturbation, while higher SR indicates that different states remain separated beyond the perturbation-induced spread. The paper evaluates these metrics on checkpoints trained with varying levels of Gaussian noise augmentation across four control tasks using LeWM and PLDM.

Critical strengths
  1. The motivation behind the work ie defining diagnostic metrics for JEPA world model rollouts is valid.
Critical weaknesses
  1. The paper claims are trivial and provide no insight. Showing that models trained with Gaussian noise have smaller intra-rollout variance under Gaussian noise and achieve higher success rates when tested on the same noise distribution provides no insight into the actual rollout dynamics.

  2. The theory (two propositions) are also trivial. Proposition 1 and 2 are trivial applications of Triangle and Cauchy inequalities respectively, and I believe are used as window dressing to cover the lack of novelty and insight in the paper.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.