acceptodds
Withdrawn from ICLR 2027

Does Surprise Measure Intuitive Physics? The Score Depends on How the Model Is Read

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent-predictive video models are said to acquire intuitive physics when they are more surprised by an impossible event than by a possible one (Garrido et al., 2025). That surprise is a single scalar, the difference in prediction error between two otherwise identical clips. The benchmark that defines that score chose the measure deliberately, naming one alternative protocol (Weihs et al., 2022) and reporting that the alternative had not yielded evidence of understanding (Bordes et al., 2025). Nothing has established what the difference contains. We decompose the squared form exactly into magnitude, direction and a selection term, and show it is not a fixed property of the model. We measure it on V-JEPA 2 (Assran et al., 2025) and V-JEPA 2.1 (Mur-Labadia et al., 2026). Both are publicly released latent-predictive encoders, and only V-JEPA 2.1 exposes intermediate depths to read at. Reading V-JEPA 2.1 at the four encoder depths its predictor was trained to read, with nothing attached on top, the reported accuracy spans 10.18 percentage points with the model, the clips and the readout all unchanged. Across the two encoders every term keeps its sign and none keeps its size, so the two standard normalisations disagree about which model is more direction-driven. A motion-only observer touching neither model shows that most or more of what the score detects is available from movement alone. We then move the score using a regulariser carrying no physics. An untrained random projection preserves accuracy at 88.39% against the frozen encoder's 89.55%. Training a small network in its place, under the coefficients published with VICReg (Bardes et al., 2022), drives it to 46.57%, a point estimate below chance whose 95% confidence interval contains chance. Raising one of those three coefficients recovers 94.62%. A violation-of-expectation score is not readable as physics evidence until six quantities are reported beside it. We specify those six and show what each one catches.

Primary area:
interpretability and explainable AI
Keywords:
intuitive physics, violation of expectation, evaluation methodology, video world models, latent-predictive models, V-JEPA, representation diagnostics, measurement validity

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_eRiCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper questions whether physically impossible videos surprise score can be treated as evidence of intuitive physics. The authors decompose the squared surprise difference into magnitude, direction/alignment, and selection terms, and find that the signal mainly comes from the alignment between the model’s prediction and the future latent representation rather than from representation magnitude. They then show that, for frozen V-JEPA models, the reported physics score can change substantially depending on the encoder depth, readout, and aggregation rule. They also introduce a motion-only baseline showing that simple kinematic information can explain a substantial portion of the possible/impossible separability. The paper argues that surprise is not a fixed property of the model alone and should be accompanied by additional diagnostics such as discriminability, untrained floor, magnitude versus direction, global versus local effects, selection effects, and a motion-only baseline.

Critical strengths

A useful decomposition of the surprise score. The paper provides a decomposition of the squared violation-of-expectation score into magnitude, direction/alignment, and selection components, and shows empirically that the observed effect is driven primarily by the directional term rather than by simple changes in latent magnitude. This is consequential because it makes an otherwise opaque scalar more interpretable and provides a principled diagnostic for future work using prediction-error-based physics benchmarks.

Critical weaknesses
  1. The readout experiments convincingly show that the surprise score is not invariant to changes in representation geometry, but the original V-JEPA/IntPhys-style evaluation does not insert a trained readout between the predictor and the target. In particular, training a VICReg head explicitly changes the metric space in which prediction error is measured, and its invariance term directly pushes prediction and target representations together. Therefore, the large performance swings under different readouts do not necessarily imply that the original predictor-based surprise score is unreliable.

  2. Several proposed diagnostics seem to regulate benchmark design rather than directly challenge the predictive-physics interpretation. The benchmark already uses paired clips with highly similar content and controlled physical perturbations, making a small magnitude difference relatively expected; correspondingly, the paper finds that the signal is dominated by directional alignment rather than latent norm. Likewise, the selection term arises from the paper’s own per-arm top-qq aggregation, which the authors explicitly note is not the published protocol. The motion-only baseline is informative, but it is an engineered pipeline using DINOv2 features, tracking, hand-designed kinematic features, and supervised logistic regression, whereas the V-JEPA surprise score is obtained without task-specific supervision. Since the motion baseline still does not outperform V-JEPA 2.1, these results are also compatible with the interpretation that predictor-based surprise captures learned physical dynamics while remaining sensitive to evaluation design.

Questions for authors

Could the authors include a strictly matched encoder-only baseline and compare it directly with predictor-based surprise? The key unresolved question for me is whether the predictor provides substantial additional information beyond the encoder representation itself. Concretely, one could keep the same encoder fixed, remove the predictor, construct a surprise score using only latent dynamics, latent MSE, or representation differences, and compare it against the predictor-based score. If predictor-based surprise substantially outperforms this encoder-only baseline, that would suggest that the signal is not merely driven by representation or motion differences between possible and impossible clips, but depends meaningfully on learned predictive dynamics. This result could materially affect my assessment of the paper’s central claim.

Reviewer_NKz1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines how violation-of-expectation (VoE) scores depend on evaluation choices for V-JEPA 2 and V-JEPA 2.1. It decomposes a squared prediction-error difference into magnitude, alignment, and selection terms, and studies the effects of encoder depth, aggregation, and VICReg-trained readouts using generated pairs of possible and impossible videos. It also compares against untrained models and a motion-based baseline. The authors argue that a VoE score cannot be interpreted as evidence of physical understanding unless six accompanying diagnostics are reported.

Critical strengths

The paper addresses a relevant question: what conclusions about physical understanding can be supported by prediction-error-based VoE scores? The decomposition and controlled comparisons provide useful information about how representation depth and readout training affect the measured result while the underlying encoder and predictor remain frozen. These observations can inform the design and reporting of VoE evaluations, even though I find the broader conclusions insufficiently supported.

Critical weaknesses
  1. Limited generalizable contribution and insufficient justification for the six diagnostics.

Prior V-JEPA work already analyzes evaluation hyperparameters (Garrido et al., 2025, Appendix B). Although this paper adds a decomposition and further interventions, it mainly demonstrates configuration sensitivity within the V-JEPA 2 family. Readout and scoring choices can be integral to a method; sensitivity alone does not establish that all six diagnostics are necessary under a fixed, justified VoE protocol. This is particularly unclear when some diagnostics arise from the paper’s squared-error and separate top-q formulation rather than the published protocols it critiques.

The evidence is also limited to a custom corpus and two closely related models. The rationale for controlled stimuli does not adequately justify omitting validation on established benchmarks such as IntPhys, IntPhys2, PhysicalConcept, and InfLevel, or relevant approaches such as PLATO and DINO-world. The generality of the conclusions therefore remains unestablished.

  1. Unclear training and evaluation protocols.

Although evaluation counts are reported, Appendix H does not clearly identify the source and composition of the 24,000 readout-training clips or their relationship to the 3,427 evaluation pairs. This prevents verification of training–evaluation separation and whether training includes physically implausible scenes. The grouping and held-out evaluation procedures for the motion baseline also remain unclear, limiting confidence in the comparisons.

Questions for authors
  1. Justification of the central claim: What evidence establishes that each of the six diagnostics is necessary for interpreting a VoE result under a fixed, justified protocol? In particular, how does this requirement apply to published L1-based protocols that do not use the paper’s separate top-q selection or additional trained readout?

  2. Scope of the conclusions: What evidence supports extending the main conclusions to established VoE benchmarks and approaches beyond the two closely related models studied here? If no such validation is available, what specific constraints prevent checking the principal findings under a published protocol, and which conclusions are intended to remain specific to the present setup?

Reviewer_wdaGOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors argue that the surprise score, as described in the literature, is insufficient to determine whether a model can understand physics. For this reason, they composed it of 3 components.

They tested it across multiple network configurations, including layer counts and readout settings, showing that the surprise score varies considerably across experimental setups.

Critical strengths

The basis of the paper is very simple, that surprise score has been used in literature as a proxy for a model to understand physics, while the authors argue and demonstrate that it is not that simple.

Critical weaknesses

The abstract and introduction are very confusing because they lack a clear motivation and a high-level overview of the problem and the solution. In my opinion, the title explains the paper better than the entire introduction. The work contains several inconsistencies; for example, it references 2 papers throughout the introduction and Section 2 that are never introduced; it describes surprise as an L1 norm, but then describes it as an L2 norm in Equation 1; and there are notation changes in Equation 2 and the following equations without a clear definition of the terms.

Questions for authors
  • I have a problem with the motivation for the work. I have not understood it fully. Why is this taxonomy important?
  • What are the 2 papers you are referring to as describing surprise throughout the introduction of Section 2?
  • Why is the surprise score defined as an L1 norm in Garrido et al., but then you define it as an L2 norm? That's not explained.
  • The notation of the equations is not consistent. You start with the two vectors P and A, and then, in the 2nd equation, you change the notation without explaining it. Also, there is no clear definition. Can you provide one?
  • The structure, grammar, and the way the paper is written are awkward. It is difficult to follow. It contains a lot of information but offers no motivation or high-level overview of the system. Similarly, it includes no qualitative images in the main text to help the reader understand the problem. I would ask the authors to review the whole text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.