- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose Action-Conditioned Predictive Consistency (ACPC) for diagnosing sensitivity of JEPA world models to visual perturbations. ACPC measures the distance between multi-step rollouts from clean and perturbed (noisy) histories under the same action sequence. These distances are used to define two metrics: Invariance Radius (IR), which captures sensitivity to perturbations, and Separation Rate (SR), which checks whether different states remain distinguishable, serving as a check against collapse. The authors theoretically show that ACPC bounds perturbation-induced changes in prediction error and planner cost. Experiments with LeWM and PLDM on four control tasks show that IR and SR track success rates across checkpoints trained with different levels of Gaussian-noise augmentation.
Critical strengths
Developing a bisimulation-motivated diagnostic that incorporates the complementary properties of IR and SR and accounts for multi-step dynamics is a meaningful idea.
- An invariance metric alone cannot distinguish a robust model from a collapsed one, whereas SR is designed to detect this distinction. In Figure 3, the authors show that under strong perturbations, SR indeed rejects some checkpoints that pass the IR threshold.
- For predicting perturbation-induced changes in prediction error, Figure 4 compares a one-step baseline with two extensions: eight-step ACPC under modified actions and eight-step ACPC under recorded actions. The latter achieves the lowest error on all four tasks, indicating the value of capturing the multi-step consequences of the specific action sequence.
Critical weaknesses
- The practical utility of the diagnostic has not been demonstrated. IR–SR checkpoint selection is evaluated only on checkpoints that differ in the level of Gaussian-noise augmentation; the PLDM, blur, and resize evaluations use the same augmentation grid. It remains unclear whether the diagnostic is useful for selecting between models with different architectures or training procedures, or under natural changes in background, lighting, or camera pose.
- The theoretical contribution is overstated. The propositions are presented as one of the three main contributions, yet both primarily formalize the motivation for the metric. They describe the geometry of the model outputs without exploiting specific properties of the model or the perturbation, and therefore do not establish conditions under which ACPC is small. Also, neither theoretical bound is directly used in the proposed selection procedure.
Questions for authors
- Can IR–SR reliably select among checkpoints that differ in architecture, training objective, training duration, or data quality, rather than only in Gaussian-noise augmentation strength?
- What concrete practical use case does ACPC/IR–SR support beyond choosing a noise-augmentation level for a known test perturbation?
- What exactly is the projection П used for LeWM and PLDM, and how is it obtained in each case?