acceptodds
Under review as a conference paper at ICLR 2027

Tracking Is Not Permanence: What Video World Models Keep of a Hidden Object

Abstract

Video world models track objects they can see; we ask what they keep of objects they cannot. We hide an object from a frozen V-JEPA 2 predictor and compare its prediction for the hidden region with the encoder's representation of two worlds that differ only inside that region. Its decision keeps a stationary object in part and one carried inside a container not at all, and loses a moving one within 0.3 s, or 0.5 s under V-JEPA's own tube mask (ViT-H keeps it to 1.1 s at pretraining's 90% masking ratio); in projection a trace remains after the first tubelet, below the midpoint for a moving object and a closed box, at 14–60% of what a baseline copying the last view retains. The information is there: the encoder reads the object's presence at 1.00 and keeps a closed container's contents decodable for the 3.5 s of the window, while the predictor's output, read with the encoder's own probe, contains the ball in 2% of scenes once the box has been closed for half a second. On rendered scenes permanence is missing on the predictor's side, and training installs it cheaply, as a prior: three thousand predictor-only steps on synthetic containers take this belief from 0.05 to 1.00 against two matched controls. They also raise IntPhys-2019 from 84.2% to 93.3%, but so does a curriculum without containers, and which training habit the benchmark credits changes with its scoring rule: for these checkpoints, benchmark accuracy does not track the belief. Continued training with tube masks for 16k steps produces 1.1–1.6 s of moving-object carry-over on egocentric and internet-style video, so the deficit is not intrinsic to latent prediction; what persists changes with the masks and the data. VideoMAE's pixel reconstruction keeps almost nothing, and Cosmos's next-token prediction keeps a stationary hidden object but not one carried inside a moving container, where V-JEPA 2 and Cosmos fail alike.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
video world models, object permanence, self-supervised video pretraining, V-JEPA, masked prediction, intuitive physics, representation analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HgqrOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors evaluate object permanence in video world models, mainly focusing on V-JEPA 2, with comparisons to VideoMAE and Cosmos. They find that these models struggle to track hidden objects and explore how different training strategies can improve this ability.

Critical strengths
  1. The authors try to explore potentially useful problem but relatively narrow problem.
Critical weaknesses
  1. This paper is poorly written and hard to follow. The motivation and main contributions are not clearly presented, and important evaluation details are scattered throughout the appendix. It reads more like a project report, and do not meet a top-tier conference paper.

  2. The methodological contribution is quite limited. The proposed probe is mainly based on simple feature comparisons, without much technical novelty or deeper insight.

  3. The experiments not clear and do not convincingly support their main claims.

  4. Some concern about their generalization, because training and testing seems to share the same renderer in several experiments.

Questions for authors

see weakness.

3 comments
AuthorsOpenReview ↗

Thank you for the review. We answer the four weaknesses in order. Please also see the general comment: during the rebuttal we found an error in our own evaluation, and the revision corrects it.

1. Writing, motivation and contributions

We rewrote the main text. The motivation is now stated in the first two sentences of the abstract: a video world model that plans has to keep state for objects it can no longer see, and we ask what a frozen V-JEPA 2 predictor keeps of a hidden object. The introduction ends with three contributions in place of six bullets:

  1. The released predictors keep a hidden object where it was last seen and track it only after seeing it move for long enough. They do not keep a ball inside a moving box, although their encoder decodes the box's contents.
  2. The readout is validated on predictors trained with a known probability that hidden objects persist.
  3. Training sets what persists, and for ViT-H benchmark accuracy does not follow it.

On the evaluation details: Section 3 now states what the decision measures, which control sets its floor, and how each intervention is trained and scored. Table 13 lists every dataset with its use.

2. Methodological contribution

The probe is a feature comparison, and we kept it that simple on purpose. It needs no training of the model under test and no labelled classifier. What the paper adds around it is evidence that the comparison can be trusted and that it shows what the standard test misses.

  • It returns a known answer. For predictors trained in worlds where a hidden ball persists with probability p, it reads the ball as present in at most 16% of scenes for p ≤ 0.25 and in at least 98% for p ≥ 0.75 (Sec. 4.1).
  • The standard surprise test gets this case wrong. On our minimal pairs V-JEPA 2 finds a vanishing ball less surprising than a continuing one in 91 to 93% of pairs, and predictors that keep the ball do the same (App. K).
  • It separates the encoder from the predictor. A linear probe on the encoder decodes the contents of a closed box for 3.5 s, with accuracy 0.91 to 1.00, while the predictor's decision for a ball in a moving box stays at the level of its controls (Sec. 4.4).

3. Support for the claims

The review does not name an experiment, so we list what backs each main claim in the revision.

ClaimEvidence addedWhere
The readout measures what the predictor expectsknown-answer test at three model sizesSec. 4.1, App. I
Released predictors keep a hidden object in placeintervals that resample seeds; rerun with one seed per scene; 27 real hiding events; the IntPhys-2019 videosTable 2, App. M, Sec. 4.5, App. O
They do not keep it inside a moving containermatched controls on rendered scenes; 12 real containment eventsSecs. 4.4, 4.5
The IntPhys-2019 gain of the curriculum is not containmentofficial evaluation code; the same curriculum without container masksSec. 4.9, App. I

If a specific experiment or claim is still unclear, please point us to it and we will answer directly.

4. Training and testing on the same renderer

This concern was justified, and the revision tests it. The answer has two parts.

What we report about the released models holds outside our renderer. On 27 real hiding events they read 0.28 to 0.38 above their control. On the IntPhys-2019 videos, which come from a renderer we did not write, they keep a static object that vanishes in plain view in 73 to 100% of pairs, against at most 7% for the controls.

What the curriculum installs does not carry over. The containment prior reads 1.00 on our renderer and 0.85 to 0.97 over natural-video backgrounds. On 12 real containment events it is 0.04 above its control, with an interval that includes zero. The abstract and the conclusion now state this limit.

Reviewer_HgqrOpenReview ↗

Comment by Reviewer Hgqr

I appreciate the substantial revisions and experiments. However, concerns regarding methodological novelty and generalization. Thus, I currently only raise my score from 1 to 2.

Thank you for reading the revision and for raising your score. We answer the two remaining concerns in turn.

Methodological novelty. The paper introduces a new measurement. It reads, from a released predictor's own output, whether the model still expects an object that it can no longer see, at each time step while the object is hidden. To our knowledge no earlier work reads this. Existing tests give one score per clip, and tracking methods train for the task with supervision (Sec. 2). Layer-wise probes of video encoders and concurrent benchmarks of unobserved-state tracking ask neighbouring questions with other models and readouts (App. J). The contribution has three parts.

  1. The readout. Two worlds are identical in every pixel the model sees and differ only in the hidden region. At each time step we ask which world's encoder representation the predictor's output on the hidden tokens is nearer to, against a control in which the object was never shown. The last step is a distance comparison by design: the readout has no trained parameters, so its answer cannot come from a probe that learned the task. It carries to VideoMAE and Cosmos-Predict1-4B (Sec. 4.6).

  2. A reason and a test for what it measures. V-JEPA 2 trains its predictor with an L1 loss, so the best prediction for a hidden token is the median of its possible targets, and the decision reads whether the object is more likely present than absent (Sec. 3.2). We tested this on ViT-H predictors trained in worlds where a hidden ball persists with a known probability p. The probe decides present in at most 16% of scenes for p ≤ 0.25 and in at least 98% for p ≥ 0.75 (Sec. 4.1).

  3. Evidence that the standard measures do not give this reading. The surprise score of violation of expectation rates a vanishing ball as less surprising than a continuing one in 91 and 93% of pairs for the released ViT-H and ViT-g, and in 86 and 97% for two predictors that the probe shows keeping the ball (App. K), because an empty region is easier to predict than a moving ball. Benchmark accuracy does not follow the belief either: for ViT-H, IntPhys-2019 stays at 93.6 to 94.7% across training worlds in which the probe's reading goes from 0 to 1 (Sec. 4.9).

The readout gives results that neither measure shows. The released predictors keep a hidden object where it was last seen and carry it forward only after seeing it move for long enough (Sec. 4.2). They lose a ball inside a moving box within two seconds, while their own encoder keeps its contents decodable for 3.5 s (Sec. 4.4).

Generalization. Your point 4 was that training and testing share one renderer. This holds where we trained predictors ourselves, in the known-answer test and the curricula. The released predictors, which carry the paper's main result, never saw the renderer in training; it only supplies their test scenes. We tested in three ways whether their results depend on those scenes. On 27 hiding events in real video they read a hidden object as present 0.28 to 0.38 above their control (Sec. 4.5). On the IntPhys-2019 videos, which come from a renderer we did not write, they keep a static object that vanishes in plain view in 73 to 100% of pairs, against at most 7% for the controls (App. O). On backgrounds that our renderer does not draw, they still decide a hidden stationary ball present more often than its control, in 13 to 78% of scenes against 5 to 14%; on our renderer the share is 71 to 90%, the highest we measured (App. P). The known-answer test shares the renderer on purpose, because it checks the readout on the scenes where we use it. For the curricula the paper reports the limit: the containment prior installed on our renderer does not transfer to real containers, and the abstract says so.

If you know of earlier work that reads this from a predictor, or of a claim that you find unsupported outside our renderer, please name it, and we will answer while the discussion is open.

Reviewer_5AgVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether video world models keep an object in mind once it is hidden. Given some history of frames with a ball, they hide the ball in the next frames using a mask and use the frozen V-JEPA 2 predictor to see what it infills in this mask. The predictor is able to infill the ball for the first few frames, very locally, but in the subsequent frames it loses track of the ball. The problem still remains when the first few frames and the last few frames have the ball and only the frames in the middle are masked out. Their primary argument is that the ball being present is well encoded in the encoder (a linear probe on the encoder's tokens of the full, unhidden scene detects it), but the predictor is not able to reconstruct it (the encoder's probe applied to the predictor's output finds the ball in only 2% of scenes once a box has closed over it), so the predictor does not have the idea of object permanence. They also identify a problem in the training paradigm and provide a solution: if you train the predictor with their methodology (different masks, or a short curriculum on synthetic containers), it is able to better represent object permanence. They also compare against VideoMAE and Cosmos.

Critical strengths
  • The paper represents something novel and asks an important question: how well are these models able to represent the idea of object permanence. People building world models want to understand if the model knows where an object is in the scene, even when it cannot be seen.

  • They support their claims with a large number of experiments and controls: a moving and a stationary ball, revealing the ball at the end, a ball inside a box and controls where the object was never shown (all on rendered scenes), plus real hands closing over a coin.

  • They also provide a way to fix it through training.

  • The code is provided in the supplementary material.

Critical weaknesses
  • The paper could have had better writing. It is very hard to follow: most sentences pack many numbers and conditions together, so the main story is hard to pull out. It still reads like it was written by an LLM (which they also say in their AI use statement), and that makes the main argument harder to follow.

  • They do not have enough evaluation. Most of the results are on rendered scenes with 60 scenes per point, the curricula are trained on the same renderer the probe uses, and the held-out real-hand test is only a few events (the authors list these limits themselves in Appendix J).

  • I could not visualize what they are doing with the other datasets (EgoDex, EPIC-KITCHENS, DROID, FunQA, MagicCATs, MMC). The idea is sound and the methodology is there, but I would appreciate if I could have seen clips and/or experiments across these other datasets. A figure or a table showing what each dataset is used for would also help.

Questions for authors
  1. What exactly is each of the other datasets used for, and what does an example clip look like?

  2. Can you show the belief probe on the released models with real occlusion video (for example the real test set of TCOW or the cups game of the Perception Test, which you mention in Appendix J), not only on rendered scenes?

2 comments
AuthorsOpenReview ↗

Thank you for the review and for the two concrete requests. Both are in the revision. Please read the general comment first. We found an error in how we queried the released models, and it changes part of what your summary describes.

The released predictors do keep a hidden ball, in place. What they do not keep is a ball inside a moving box. The 2% you cite, for the encoder's probe applied to the predictor's output once a box has closed, is now 62% half a second after closing and 7% after two seconds (controls 22% and 2%).

The criticism is fair. We rewrote the main text. The results sections now open with the finding, Section 4 follows the order of the findings, and the introduction ends with three contributions in place of six bullets. As an example, the first results paragraph used to begin

"... keeps a hidden moving object for one tubelet and then leans absent. Figure 2a and Table 3 (Appendix A) give the curve for ViT-H: P=0.92 at 0 s and 0.00 at 0.27 s, against a geometry-matched control, the object never rendered, of 0.30 and 0.00. ViT-L (0.58, 0.00) and ViT-g (0.98, 0.17, 0.10) behave the same way ..."

and now begins

"Under the main query the three released predictors keep a stationary hidden ball to the end of the window and a moving one for 0.5-1.1 s, after which the prediction stays nearest the world in which the ball stopped."

Some paragraphs still carry many numbers, because they report three models with their controls. Those numbers now come after the sentence that says what they show.

Evaluation

Three things were added. The probe now runs on real occlusion video (your Q2, below). It also runs on the IntPhys-2019 videos, which come from a renderer we did not write (App. O). There the released predictors keep a static object that vanishes in plain view in 73 to 100% of pairs, against at most 7% for geometry-matched controls. And every interval over rendered scenes now resamples renderer seeds, with a rerun that uses a fresh seed for each scene (App. M).

Two of the limits you name remain. The held-out real-hand test is small (19 and 11 events). The curricula are trained on the renderer that the probe uses, and this now has a measured consequence, described under Q2.

Q1: what each dataset is used for

Table 13 (App. B) lists each dataset with its content and its use. In short:

DatasetUsed for
Rendered scenes (ours)the probe and its controls; the synthetic curricula
EgoDexnatural-video training; real-hand training events; 19 held-out hand events; 300 held-out clips for the prediction cost
EPIC-KITCHENS-100natural-video training; real-hand training events; 11 held-out closing events
DROIDnatural-video training
FunQAcontinued pretraining (2,435 clips); magic readouts
MagicCATs, MMCmagic readouts; transfer of the real-hand recipe (81 and 19 events)
IntPhys-2019, IntPhys 2benchmark; the IntPhys-2019 videos also for the probe
TCOW real setthe probe on real occlusion video

For example clips, Figure 9 shows a held-out closing hand from EPIC-KITCHENS with each token's decision, and Figure 10 an IntPhys-2019 video. We show no frames from MagicCATs and MMC, which we received from their authors on request, or from the office recordings of TCOW, which carry no licence.

Q2: the probe on real occlusion video

Done, on the TCOW real set you suggested, which includes Perception Test cup games (Sec. 4.5, Table 3, App. L). An event is a run of frames in which the object is fully hidden and then reappears; 27 events from 25 videos fit a 32-frame window. The "present" world is the video as filmed. For the "absent" world, video inpainting removes the object from the moment it is hidden. For the control it removes the object from every frame.

ReleasedP(present)ControlDifference (before rounding)
ViT-L0.570.280.28 [0.14, 0.43]
ViT-H0.440.140.31 [0.17, 0.46]
ViT-g0.650.280.38 [0.24, 0.51]

The released predictors keep a hidden object on real video too, about as much as the three predictors we trained (0.28 to 0.41 above control). Containment is different. In the 12 containment events, six of them cup games, the object is hidden for 2.9 to 7.1 s, and no model is more than 0.08 above its control. That includes the predictor trained on rendered containers. Its prior does not transfer to real containers, and the abstract now says so.

The set is small, and inpainting leaves some artefacts, which App. L lists.

Thank you for the additional tests. I maintain my original score.

Reviewer_wNX6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares a model's prediction with two versions of a scene: one where the object remains and one where it is gone. Most experiments use V-JEPA 2, with additional tests on VideoMAE and Cosmos. The authors find weak predictions of hidden objects in simple rendered scenes and improve these predictions through further training. They also show that higher IntPhys-2019 scores can occur without learning to keep track of an object inside a container.

Critical strengths
  • The problem is technical-sound. A model used for planning should retain useful information about objects that are temporarily out of view. Checking this ability directly is worthwhile.
  • The experiments include useful controls: an object that was never present, one that left the scene, stationary and moving objects, and an object revealed again later. Comparing against simply copying the last visible features is also helpful.
  • The container-training result is promising within the tested setting. The authors report that 3,000 steps improve the hidden-object test substantially. They repeat the main ViT-H IntPhys result with additional training seeds and check that changing the scoring head alone does not explain the gain.
Critical weaknesses
  • The importance of the problem is clear, but the meaning of the main measurement needs stronger support. Choosing the "object absent" reference does not establish how much useful memory the model has lost. In Table 9, a separate classifier can still distinguish the main condition from control conditions with 99-100% accuracy using the predictor's output. Other measurements also find a lasting trace of the object. The authors acknowledge this. However, a test with a known correct answer about memory, or a link to useful behavior, is needed to justify treating the chosen score as a measure of permanence. The timing is also conditional: ViT-H keeps the moving object to 1.1 seconds when the masking setup is closer to training, compared with the headline loss within about 0.3 seconds under the main setup.

  • The method appears mostly incremental relative to the prior work discussed in the paper. Comparing possible and impossible futures is already central to violation-of-expectation testing [1]. Counterfactual inputs and structured masking have been used to study video models [2], and tracking through occlusion and containers is already studied in TCOW [3]. The most distinct contribution here is the detailed comparison between information in the encoder and information expressed in the predictor's output. The training recipes mainly adapt existing models through different masks and data. The paper should explain more clearly what new information its test adds beyond earlier tracking and counterfactual tests.

  • Detailed visual examples are missing from the results. Figure 1 shows the basic idea, but curves and tables alone make it hard to understand the model's successes and failures. The paper should show example frames before, during, and after an object is hidden, mark what the model can see, and show its decision at each step. Examples from both rendered scenes and real-hand videos, before and after training, would help explain the findings.

  • Cheap training on these scenes does not yet establish an efficient general solution. The strong container result uses the same family of rendered scenes for training and testing. Replacing the background with natural video is a useful check, but leaves the basic objects and events similar. The strongest real-hand results use only 8 held-out EgoDex events and 4 events from unseen participants; only 3 and 2, respectively, have a valid removed-object control. IntPhys 2 shows no clear improvement. The more involved real-hand training also lowers IntPhys-2019 accuracy to 27.2% and 30.6%, compared with the released ViT-H's 84.2% (Table 10). These results leave the wider benefit and cost unresolved.

  • The benchmark evidence supports a limited conclusion. The container curriculum improves IntPhys-2019 from 84.2% to 93.3%, but training without container masks reaches 94.3% on average (Table 1). This is useful evidence that a higher score does not specifically demonstrate containment. However, all comparisons use the development set, and the alternative scoring implementation does not reproduce the published baseline (Appendices I and J). The reported changes in model rankings therefore need confirmation with a validated evaluation setup and an untouched test set before drawing a broader conclusion about the benchmark.

[1] Garrido, Quentin, et al. "Intuitive physics understanding emerges from self-supervised pretraining on natural videos". 2025.

[2] Bear, Daniel M., et al. "Unifying (Machine) Vision via Counterfactual World Modeling". 2023.

[3] Van Hoorick, Basile, et al. "Tracking Through Containers and Occluders in the Wild." CVPR 2023.

Questions for authors
  • Can you validate the test on a model known to remember hidden objects? How should a low score be interpreted when another classifier can still recover the difference between the main and control conditions from the same output?

  • Does the container curriculum help on a different scene generator or real occlusion videos, compared with the same amount of training without container masks? What other abilities improve or worsen?

  • Can you build a control that also breaks object identity and whether the object reappears, while keeping individual frames similar? Does the claimed contribution from learning the appearance of the data still hold?

  • Can you reproduce the established IntPhys evaluation first, then compare the curricula on an untouched test set and provide the individual video scores?

  • Can you use independent scene seeds for each setting, or recalculate uncertainty by grouping scenes that share a seed? Do the main comparisons and estimated memory durations remain convincing?

4 comments
AuthorsOpenReview ↗

Thank you for the careful review. Your Q4 is what led us to the error described in the general comment, and we are grateful for it. All numbers below are the corrected ones.

W1, Q1: what the decision measures, and a test with a known answer

We ran the test you asked for. We trained ViT-H predictors on our scenes, under our query, in worlds where a hidden ball continues with probability p and vanishes otherwise (3,000 predictor-only steps, two seeds per p). A readout of what the predictor expects should say "absent" for small p and "present" for large p. It does:

pmoving ballstationary ballcontrol
00.000.050.00
0.250.000.16≤ 0.01
0.50.110.91≤ 0.01
0.750.980.98≤ 0.01
11.000.99≤ 0.01

These are shares of scenes read as present, averaged over 0.53 to 3.47 s after hiding (Sec. 4.1, Table 1). ViT-L and ViT-g, with one seed each at p = 0, 1/2 and 1, behave the same way (App. I).

The switch near one half is what the training loss predicts. V-JEPA 2 trains its predictor with an L1 loss, so the best prediction for a hidden token is the median of its possible targets: the representation with the ball when p > 1/2 and the one without it when p < 1/2. At exactly 1/2 the argument is silent, and there the stationary ball reads present and the moving one absent.

You also write that choosing the absent reference does not show how much memory was lost. We agree, the decision is binary. For a graded measure we report the projection α next to it. In the same test α rises steadily, from 0.16 at p = 0 to 0.86 at p = 1 for a moving ball, with the control near 0.1.

The 99-100% classifier. You are right that the predictor's output still separates the main condition from the controls. The corrected runs give 0.98 to 1.00 (Table 14). We read this as the answer to a different question. A classifier trained with labels on the output detects that the two contexts differed. The decision asks whether the output is closer to the encoder's representation of the scene with the object than to the one without it. The two uses of the prediction that we know of make this kind of comparison. V-JEPA 2-AC plans by minimising the L1 distance between predicted and goal representations in this space (Assran et al. 2025, Eq. 5), and the surprise test is an L1 error there. Neither has a labelled classifier.

The trace you point to is now reported next to the decision. For ViT-H the encoder's own probe still finds the moving ball in 42 to 52% of scenes between 0.8 and 1.9 s, after the decision has turned to absent, and Sec. 4.2 says that "kept" and "lost" refer to the decision.

On a link to behaviour: we did not run a planning experiment. The link we offer is the argument above, that V-JEPA 2-AC's planner reads the same L1 distance as our decision.

The timing is conditional. It is, and the abstract now states the conditions. Both numbers you quote, 0.3 s and 1.1 s, were computed with the untrained token. With the trained token the three models keep a moving ball for 1.07, 0.53 and 1.07 s (ViT-L, H, g) under our query, in which the ball is seen for half a second before it is hidden. With 90% of the remaining context hidden, as in pretraining, this becomes 1.33, 1.60 and 2.40 s. Under V-JEPA 2's own tube geometry, with the ball seen for 2.1 s, they keep it for 1.87, 1.60 and 1.87 s, and 1.87 s is the last time point of that window. The duration still depends on the query. Table 2 has the intervals, and Table 16 separates the length of the visible approach from the mask geometry.

W2: what the test adds to VoE, counterfactual world models and TCOW

The counterfactual input is not new and we do not present it as ours. What the paper adds is a readout of what a frozen predictor expects in a hidden region at each time step, checked on predictors with a known answer and set against what the encoder holds.

Against VoE the difference can be measured. We ran the surprise test of Garrido et al. on minimal pairs from our renderer (App. K). The released ViT-H and ViT-g find a ball that vanishes less surprising than one that continues, in 91% and 93% of pairs. Two predictors that our probe shows keeping the ball do the same (86% and 97%). An empty region is easier to predict than a moving ball, so a low surprise cannot tell a model that dropped the object from one that kept it. The probe can, because it compares one prediction with both outcomes on the hidden tokens and has a matched control.

Bear et al. use counterfactual prompts to extract flow and segments; they do not ask what persists over time. TCOW trains a tracker with mask supervision, while we test a model as released. We now use TCOW's real videos as a test set (Sec. 4.5). Related Work discusses all three.

W3: visual examples

Added. Figure 4 shows the frames the model sees, the hidden tokens and each token's decision, for a moving ball and for a ball in a box, released and trained. Figure 9 does the same for a held-out real closing hand, and Figure 10 for an IntPhys-2019 video.

W4, Q2: does the container curriculum help elsewhere, and at what cost?

On real containers it does not help, and the abstract now says so.

  • Real video (Sec. 4.5). On 27 real hiding events the container curriculum reads 0.41 above its control, the same curriculum without container masks 0.28 and the released ViT-H 0.31; the intervals overlap. On 12 real containment events no model is more than 0.08 above its control, and the container curriculum is at 0.04 [−0.09, 0.18].
  • Another generator (App. O). On the IntPhys-2019 videos the container curriculum agrees with the released ViT-H in 95% of decisions, so it changes little there.
  • What improves. On our renderer the curriculum installs containment (1.00 against controls of at most 0.02) and lengthens the time a hidden moving ball is kept from 0.53 to 1.6 s.
  • What it costs. Only the predictor is trained, so the encoder is unchanged. The predictor's own error on 300 held-out natural clips rises by 0.4 to 0.7% (Table 11). IntPhys 2 shows no gain: 51.2 to 54.2%, against 52.8 to 55.3% for the released model.

On the real-hand results your count is right; it refers to the short closures. Over all held-out events (19 EgoDex, 11 EPIC-KITCHENS) the paired difference between coin-in and never has 95% intervals of [0.58, 0.95] and [0.68, 1.00] on EgoDex and of [0.09, 0.64] and [0.36, 0.91] on EPIC-KITCHENS, for the two seeds (App. G). A removed-object control exists for 8 and 4 of these events. The samples are small and the limitations say so.

We also corrected a statement here. The two EPIC-KITCHENS participants were not unseen: their closing events were held out of the real-hand training, but their unlabelled videos were part of the natural-video data that the recipe builds on.

The real-hand recipe is the costly one, as you note. It scores 27.2% and 30.8% on IntPhys-2019 (Table 17; the 30.6% in the submission was a transcription slip), although its prediction error on held-out natural video falls by 4 to 7%. We have no fix for this, and it is reported as the price of that recipe (Sec. 4.8).

W5, Q4: IntPhys-2019 under the official evaluation

We ran the official evaluation code of Garrido et al. on the released checkpoints. Once the model is built the way V-JEPA 2 was trained, the code reproduces the published number: 87.8% for ViT-H under the rule behind the published 87.2%, and 86.1% [81.1, 90.6] with two context frames. Under this protocol, for ViT-H:

two context framesminimum rule
released86.184.4
container curriculum92.290.0
same steps, no container masks91.791.1
known persistence, p = 092.290.6
known persistence, p = 193.391.1

The container curriculum gains 6.1 points [1.7, 10.6] and differs from the curriculum without container masks by 0.6 [−1.1, 2.8]. The limited conclusion you describe therefore holds under the validated setup: the gain is there, and it is not containment. Per-video scores under both pipelines are in the supplement.

We did not evaluate on the test set, which is scored by an evaluation server. All comparisons are on the dev set, and the limitations say so. The dev set is also small: switching between fp16 and bf16 arithmetic alone moves 6 of the 180 official pairs.

Q3: a control that breaks identity and reappearance while keeping frames similar

Two controls bear on this, and one of them is new.

The new one is the known-persistence worlds of our first reply. The frames before hiding are the same for every p. Only how often the hidden ball comes back differs. For ViT-H, IntPhys-2019 does not move with p: 93.6 to 94.7% over all seeds and values of p, and +1.1 [0.0, 2.8] for p = 1 against p = 0 under the official protocol. The probe meanwhile goes from 0 to 1.

The other is the non-persistent curriculum of the submission, in which every object is redrawn at a random position in each frame. This keeps the renderer's appearance and leaves no object to follow from one frame to the next. Against the corrected baseline it gains 2.3 points [−1.4, 6.0] over the released ViT-H, where the same curriculum with normal motion gains 7.9 [4.1, 11.9] (3.3 and 6.1 under the official protocol). We did not run a variant that also changes what an object looks like from frame to frame.

So the appearance claim holds in a weaker form than we submitted. For ViT-H appearance accounts for 2 to 3 of the 6 to 8 points, with an interval that reaches zero, and for ViT-g it adds nothing. One more caveat belongs here. For ViT-L, with one seed per p, IntPhys-2019 does rise with p, by 5.0 points [2.8, 7.5]. The statement that benchmark accuracy does not follow the belief is therefore made for ViT-H only.

Q5: seeds

You are right. The 12 radius-by-distractor cells of a time point shared five renderer seeds, so its 60 scenes form five clusters of 12. Every interval over rendered scenes now resamples seeds, and we reran the main conditions with a fresh seed for each scene (App. M, Table 19).

Releasedmoving ball kept forseed bootstrapone seed per scene
ViT-L1.07 s[0.80, 1.33]1.07 [1.07, 1.33]
ViT-H0.53 s[0.53, 0.53]0.53 [0.53, 0.53]
ViT-g1.07 s[1.07, 1.33]1.07 [1.07, 1.07]

A stationary ball is kept to the end of the window in both analyses. Not every comparison survived. The 1.1 to 1.6 s that the submission credited to continued tube-mask training is at most two tubelets above the corrected released ViT-L (1.07 s [0.80, 1.33]), and we no longer present it as a gain.

Dear Reviewer wNX6,

Two of the new results exist because you asked for them: the known-answer test of the readout (W1, Q1; Sec. 4.1) and the official IntPhys-2019 evaluation (W5, Q4; App. I). Our summary comment says where the other new experiments are.

Does the known-answer test settle your concern about what the readout measures? If it does not, or if another point is still open, please tell us which, and we will answer as soon as we can.

Comments on the paper

3 comments
AuthorsOpenReview ↗

We thank the three reviewers. Before the individual replies we have to report an error in our own evaluation, because it changes several results of the submission.

What went wrong. Reviewer wNX6 asked us to reproduce the official IntPhys-2019 evaluation (Q4). While doing this we found that we had queried the released V-JEPA 2 predictors with the wrong mask token. Each released checkpoint stores ten mask tokens. Only token 0 was trained; tokens 1 to 9 are exactly zero. The transformers implementation that we used fills hidden positions with token 1. Every result in the submission that queried a released predictor was therefore computed with a zero vector that the predictor had never seen in training.

The predictors we trained ourselves are unaffected, because they learned token 1 and are queried with it. The linear probes on the encoder's own tokens and the VideoMAE and Cosmos experiments do not use this predictor. We reran the released-predictor results with token 0, a one-line change. Appendix H states the two exceptions: the magic readouts were rerun for ViT-H only, and the FunQA humour-versus-magic contrast was not rerun. The revision marks changes in blue, and Table 20 (Appendix N) puts submitted and corrected values side by side.

What it changes.

Released predictors (ViT-L / H / g)SubmittedCorrected
Hidden moving ball, read as present up to0 / 0 / 0 s1.07 / 0.53 / 1.07 s
Hidden stationary ball, scenes read as present from 0.53 s on16 / 41 / 13%70 / 91 / 85%
Ball in a moving box at opening, ViT-H0-5%10-18% (controls 5-17%)
Encoder's probe on the predictor's output, ball in a moving box, ViT-H, after 0.53 / 1.07 / 2.00 s2 / 0 / 0%62 / 40 / 7%
IntPhys-2019, two context frames61.4 / 84.2 / 54.2%64.2 / 86.4 / 70.6%

The submission's headline, that the released predictors lose a hidden moving object within 0.3 s, was an artefact of this error. The corrected picture is as follows.

  1. The released predictors keep a hidden object where it was last seen. A stationary ball stays present to the end of the 3.5 s window. A moving ball seen for half a second stays present for 0.5 to 1.1 s, and the prediction is then nearest the world in which the ball stopped.
  2. They carry a moving ball forward, for about a second, only after seeing it move for two seconds: all three models under V-JEPA 2's own mask geometry, ViT-L and ViT-g also under our query.
  3. They do not keep a ball carried inside a moving box, although their encoder decodes the box's contents for 3.5 s. This part of the submission stands. One number in it moved (last row of the table): the predictor's output still carries the ball half a second after the box closes and has lost it after two seconds.

The distinction in the title holds with a different content: permanence in place, tracking only after enough visible motion, no containment.

What we withdraw, in the submission's words:

  1. "loses a moving one within 0.3 s", "scale does not help under this query" and "the predictor fills in locally". With the trained token, showing the ball again at the end of the clip raises the belief through the whole hidden gap (Sec. 4.3).
  2. "On rendered scenes permanence is missing on the predictor's side". This holds for containment only.
  3. The 5.1-point advantage of the static over the non-persistent curriculum on IntPhys-2019. Under the official pairing it is −0.4 [−4.4, 3.3], and we no longer interpret the ordering.
  4. The 1.1 to 1.6 s of carry-over after continued tube-mask training, as a gain. The corrected released ViT-L already reads 1.07 s [0.80, 1.33].

What stands. The containment result, the effect of the container curriculum on our renderer, the real-hand intervention, the finding that the IntPhys-2019 gain is not containment, and the VideoMAE and Cosmos results.

We apologise for the error, and we thank Reviewer wNX6 for the question that exposed it.

Most of what is new in the revision was asked for in the reviews. The individual replies give the results; this is the list.

AdditionWhereAsked by
Test of the readout on predictors with a known answerSec. 4.1, App. IwNX6
The probe on real occlusion video (TCOW real set, which includes Perception Test cup games)Sec. 4.5, App. L5AgV, wNX6
The probe on the IntPhys-2019 videos, a renderer we did not writeApp. OwNX6, Hgqr
The official IntPhys-2019 evaluation code, with per-video scoresApp. I, supplementwNX6
Intervals that resample renderer seeds, and a rerun with one seed per sceneApp. MwNX6
The standard surprise test on our minimal pairsApp. KwNX6
Example figures and a table of datasetsFigs. 4, 9, 10, Table 13wNX6, 5AgV
A rewritten main textSecs. 1 and 45AgV, Hgqr

Two of these results limit our own claims, and the revision reports them as such. The containment prior that the container curriculum installs on our renderer does not transfer to real containers (Sec. 4.5). And the statement that benchmark accuracy does not follow the belief is now made for ViT-H only, because for ViT-L the IntPhys-2019 score does rise with the persistence probability (App. I).

Numbering. The tables cited in the reviews have moved: submitted Table 1 is now Table 4, Table 9 is Table 14 and Table 10 is Table 17. Appendices A to J keep their letters, and K to O are new. The title is unchanged, and the main text now uses its tenth page.

Supplement. It now contains the corrected code, the per-video IntPhys-2019 scores under both evaluation pipelines, and the job scripts of the new experiments.

AuthorsOpenReview ↗

Final summary for the Area Chair and the reviewers

Dear All,

Thank you for handling our paper. As the discussion closes, we summarize what the paper establishes and show the most funny part we have written in the papr:

Summary

A coin that a hand closes over is still in the hand, and a model that plans has to know this. The paper asks whether video world models do: once an object is hidden, does the model still expect it to be there?

We introduce a new measurement that puts this question to the model directly. We hide one region of a video from the model and prepare two versions of the video. They are identical in everything the model sees and differ only inside the hidden region: in one the object is still there, in the other it is gone. The model fills in the hidden region, and we check which version its prediction is closer to, at each moment while the object is hidden, against a control in which the object was never shown. Nothing is trained for this comparison, so the answer is the model's own. We checked the measurement on models we trained so that the right answer is known.

For the released V-JEPA 2 models the answer is the title, Tracking Is Not Permanence: they remember a hidden object where they last saw it, follow it only after watching it move for long enough, and lose it once it is carried away inside a box.

Main findings

  1. For a hidden object, the standard test points the wrong way. It asks whether an impossible video surprises the model more than a possible one. But an empty region is easier to predict than a moving ball, so the released models find a ball that vanishes less surprising than one that continues, in 91 and 93% of pairs (ViT-H, ViT-g; App. K). Predictors that keep the ball according to our measurement do the same. A low surprise score therefore does not show that a model forgot the object.

  2. Remembering is not tracking. A ball hidden while at rest is still expected: in our rendered scenes it is read as present in 70 to 91% of scenes over a 3.5 s window at the three model sizes, against at most 10% for the control. A ball hidden after half a second of visible motion is expected where it was last seen, for 0.5 to 1.1 s, and not further along its path. The models move it along only after watching it move for longer (Sec. 4.2).

  3. The encoder still holds what the prediction has lost. When a ball is closed inside a box and the box moves on, a linear probe can still read the ball from the model's own encoder, the part that perceives the video, for the 3.5 s after the box closes. The predictor, the part that fills in what the model cannot see, has lost the ball within two seconds (Sec. 4.4).

  4. Training changes the belief, and for ViT-H benchmark accuracy does not follow it. Training the ViT-H predictor alone for 3,000 steps on rendered boxes teaches it to keep the ball in the box on our renderer, though not in real containers (Sec. 4.8). ViT-H predictors trained in worlds where hidden balls always persist, and in worlds where they never do, all score 93.6 to 94.7% on IntPhys-2019, while our measurement goes from "present" in almost every scene to almost none (Sec. 4.9).

Core contributions

  1. A new measurement. It reads from a video predictor's own output, moment by moment, whether the model still expects a hidden object. To our knowledge no earlier work reads this; concurrent work asks neighbouring questions with other models and readouts (App. J).

  2. Evidence that the measurement works and that it is needed. On predictors trained in worlds where a hidden ball persists with a set probability, the reading switches from absent to present as that probability passes one half (Sec. 4.1). Surprise scores and, for ViT-H, benchmark accuracy do not give this reading.

  3. A picture of what released models keep of a hidden object: permanence in place, tracking only with enough visible motion, no containment. It holds at the three model sizes. The released models never saw our renderer in training; they also keep a hidden object on real video (Sec. 4.5) and a static one on the IntPhys-2019 videos, which come from a renderer we did not write (App. O). The measurement carries to VideoMAE and Cosmos-Predict1-4B (Sec. 4.6).

The numbers above are those of the revised paper. The revision adds the tests the reviewers asked for: the known-answer test (Sec. 4.1), real video (Sec. 4.5), the official IntPhys-2019 evaluation (App. I) and intervals that resample the renderer seeds (App. M). It also corrects an error in how we queried the released models, which we found while running the official evaluation (App. N). We thank the reviewers and the Area Chair.

Thanks in advance

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.