- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors evaluate object permanence in video world models, mainly focusing on V-JEPA 2, with comparisons to VideoMAE and Cosmos. They find that these models struggle to track hidden objects and explore how different training strategies can improve this ability.
Critical strengths
- The authors try to explore potentially useful problem but relatively narrow problem.
Critical weaknesses
-
This paper is poorly written and hard to follow. The motivation and main contributions are not clearly presented, and important evaluation details are scattered throughout the appendix. It reads more like a project report, and do not meet a top-tier conference paper.
-
The methodological contribution is quite limited. The proposed probe is mainly based on simple feature comparisons, without much technical novelty or deeper insight.
-
The experiments not clear and do not convincingly support their main claims.
-
Some concern about their generalization, because training and testing seems to share the same renderer in several experiments.
Questions for authors
see weakness.
3 comments
Comment by Reviewer Hgqr
I appreciate the substantial revisions and experiments. However, concerns regarding methodological novelty and generalization. Thus, I currently only raise my score from 1 to 2.
Thank you for reading the revision and for raising your score. We answer the two remaining concerns in turn.
Methodological novelty. The paper introduces a new measurement. It reads, from a released predictor's own output, whether the model still expects an object that it can no longer see, at each time step while the object is hidden. To our knowledge no earlier work reads this. Existing tests give one score per clip, and tracking methods train for the task with supervision (Sec. 2). Layer-wise probes of video encoders and concurrent benchmarks of unobserved-state tracking ask neighbouring questions with other models and readouts (App. J). The contribution has three parts.
-
The readout. Two worlds are identical in every pixel the model sees and differ only in the hidden region. At each time step we ask which world's encoder representation the predictor's output on the hidden tokens is nearer to, against a control in which the object was never shown. The last step is a distance comparison by design: the readout has no trained parameters, so its answer cannot come from a probe that learned the task. It carries to VideoMAE and Cosmos-Predict1-4B (Sec. 4.6).
-
A reason and a test for what it measures. V-JEPA 2 trains its predictor with an L1 loss, so the best prediction for a hidden token is the median of its possible targets, and the decision reads whether the object is more likely present than absent (Sec. 3.2). We tested this on ViT-H predictors trained in worlds where a hidden ball persists with a known probability p. The probe decides present in at most 16% of scenes for p ≤ 0.25 and in at least 98% for p ≥ 0.75 (Sec. 4.1).
-
Evidence that the standard measures do not give this reading. The surprise score of violation of expectation rates a vanishing ball as less surprising than a continuing one in 91 and 93% of pairs for the released ViT-H and ViT-g, and in 86 and 97% for two predictors that the probe shows keeping the ball (App. K), because an empty region is easier to predict than a moving ball. Benchmark accuracy does not follow the belief either: for ViT-H, IntPhys-2019 stays at 93.6 to 94.7% across training worlds in which the probe's reading goes from 0 to 1 (Sec. 4.9).
The readout gives results that neither measure shows. The released predictors keep a hidden object where it was last seen and carry it forward only after seeing it move for long enough (Sec. 4.2). They lose a ball inside a moving box within two seconds, while their own encoder keeps its contents decodable for 3.5 s (Sec. 4.4).
Generalization. Your point 4 was that training and testing share one renderer. This holds where we trained predictors ourselves, in the known-answer test and the curricula. The released predictors, which carry the paper's main result, never saw the renderer in training; it only supplies their test scenes. We tested in three ways whether their results depend on those scenes. On 27 hiding events in real video they read a hidden object as present 0.28 to 0.38 above their control (Sec. 4.5). On the IntPhys-2019 videos, which come from a renderer we did not write, they keep a static object that vanishes in plain view in 73 to 100% of pairs, against at most 7% for the controls (App. O). On backgrounds that our renderer does not draw, they still decide a hidden stationary ball present more often than its control, in 13 to 78% of scenes against 5 to 14%; on our renderer the share is 71 to 90%, the highest we measured (App. P). The known-answer test shares the renderer on purpose, because it checks the readout on the scenes where we use it. For the curricula the paper reports the limit: the containment prior installed on our renderer does not transfer to real containers, and the abstract says so.
If you know of earlier work that reads this from a predictor, or of a claim that you find unsupported outside our renderer, please name it, and we will answer while the discussion is open.
Thank you for the review. We answer the four weaknesses in order. Please also see the general comment: during the rebuttal we found an error in our own evaluation, and the revision corrects it.
1. Writing, motivation and contributions
We rewrote the main text. The motivation is now stated in the first two sentences of the abstract: a video world model that plans has to keep state for objects it can no longer see, and we ask what a frozen V-JEPA 2 predictor keeps of a hidden object. The introduction ends with three contributions in place of six bullets:
On the evaluation details: Section 3 now states what the decision measures, which control sets its floor, and how each intervention is trained and scored. Table 13 lists every dataset with its use.
2. Methodological contribution
The probe is a feature comparison, and we kept it that simple on purpose. It needs no training of the model under test and no labelled classifier. What the paper adds around it is evidence that the comparison can be trusted and that it shows what the standard test misses.
3. Support for the claims
The review does not name an experiment, so we list what backs each main claim in the revision.
If a specific experiment or claim is still unclear, please point us to it and we will answer directly.
4. Training and testing on the same renderer
This concern was justified, and the revision tests it. The answer has two parts.
What we report about the released models holds outside our renderer. On 27 real hiding events they read 0.28 to 0.38 above their control. On the IntPhys-2019 videos, which come from a renderer we did not write, they keep a static object that vanishes in plain view in 73 to 100% of pairs, against at most 7% for the controls.
What the curriculum installs does not carry over. The containment prior reads 1.00 on our renderer and 0.85 to 0.97 over natural-video backgrounds. On 12 real containment events it is 0.04 above its control, with an interval that includes zero. The abstract and the conclusion now state this limit.