- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper questions whether physically impossible videos surprise score can be treated as evidence of intuitive physics. The authors decompose the squared surprise difference into magnitude, direction/alignment, and selection terms, and find that the signal mainly comes from the alignment between the model’s prediction and the future latent representation rather than from representation magnitude. They then show that, for frozen V-JEPA models, the reported physics score can change substantially depending on the encoder depth, readout, and aggregation rule. They also introduce a motion-only baseline showing that simple kinematic information can explain a substantial portion of the possible/impossible separability. The paper argues that surprise is not a fixed property of the model alone and should be accompanied by additional diagnostics such as discriminability, untrained floor, magnitude versus direction, global versus local effects, selection effects, and a motion-only baseline.
Critical strengths
A useful decomposition of the surprise score. The paper provides a decomposition of the squared violation-of-expectation score into magnitude, direction/alignment, and selection components, and shows empirically that the observed effect is driven primarily by the directional term rather than by simple changes in latent magnitude. This is consequential because it makes an otherwise opaque scalar more interpretable and provides a principled diagnostic for future work using prediction-error-based physics benchmarks.
Critical weaknesses
-
The readout experiments convincingly show that the surprise score is not invariant to changes in representation geometry, but the original V-JEPA/IntPhys-style evaluation does not insert a trained readout between the predictor and the target. In particular, training a VICReg head explicitly changes the metric space in which prediction error is measured, and its invariance term directly pushes prediction and target representations together. Therefore, the large performance swings under different readouts do not necessarily imply that the original predictor-based surprise score is unreliable.
-
Several proposed diagnostics seem to regulate benchmark design rather than directly challenge the predictive-physics interpretation. The benchmark already uses paired clips with highly similar content and controlled physical perturbations, making a small magnitude difference relatively expected; correspondingly, the paper finds that the signal is dominated by directional alignment rather than latent norm. Likewise, the selection term arises from the paper’s own per-arm top- aggregation, which the authors explicitly note is not the published protocol. The motion-only baseline is informative, but it is an engineered pipeline using DINOv2 features, tracking, hand-designed kinematic features, and supervised logistic regression, whereas the V-JEPA surprise score is obtained without task-specific supervision. Since the motion baseline still does not outperform V-JEPA 2.1, these results are also compatible with the interpretation that predictor-based surprise captures learned physical dynamics while remaining sensitive to evaluation design.
Questions for authors
Could the authors include a strictly matched encoder-only baseline and compare it directly with predictor-based surprise? The key unresolved question for me is whether the predictor provides substantial additional information beyond the encoder representation itself. Concretely, one could keep the same encoder fixed, remove the predictor, construct a surprise score using only latent dynamics, latent MSE, or representation differences, and compare it against the predictor-based score. If predictor-based surprise substantially outperforms this encoder-only baseline, that would suggest that the signal is not merely driven by representation or motion differences between possible and impossible clips, but depends meaningfully on learned predictive dynamics. This result could materially affect my assessment of the paper’s central claim.