acceptodds
Under review as a conference paper at ICLR 2027

Video Foundation Models Bind Objects but Do Not Maintain Them Under Occlusion

Abstract

Video foundation models are widely assumed to acquire object-centric structure, yet whether that structure is causally used, and whether it survives the moment an object stops being visible, has not been tested. Decoding is blind to both: linear readouts recover object identity from frozen features, yet a probe trained to report the hidden object retrieves the occluder instead. We measure object binding inside the model's own prediction machinery: delete object A's early tokens from a frozen predictor's context, then read the per-object degradation of its future predictions. A leakage ratio L summarizes this against endpoints measured in the same pipeline, at K+1 forward passes per clip with no fitted parameters. In a predictive video foundation model (V-JEPA2) and a diffusion one (CogVideoX), objects are strongly bound, and in V-JEPA2 the binding is learned: L = 0.23–0.43 against untrained 1.00 and label-shuffled 1.16, surviving distance- and appearance-matched controls and generalizing to real video. The same models have no object permanence. Under occlusion the fill contrast POperm, hidden object minus occluder, is never significantly positive on our pilot scenes in five video foundation models spanning three training paradigms; where it turns positive at scale (V-JEPA2-H) it is as positive when the object never existed, and V-JEPA2's significant deficits survive subtracting the readout's own measured floor. An object-centric model with per-object recurrent state reaches +0.264; on the 62 clips both cover, the paired gap is +0.292 with the object-centric model higher on 95% of them. The obvious fix works and does not do what it appears to: two epochs of occlusion-completion fine-tuning turn permanence positive (−0.024 → +0.070 over three seeds), and eight lift occluded-object identification from 0.369 to 0.522, yet a paired counterfactual in which the object never existed reproduces 96–108% of the fill gain, identifying the fix as a largely history-blind de-occlusion prior. On this class of feedforward predictors, internal binding does not imply object permanence, and the cheap fix does not install it.

Primary area:
interpretability and explainable AI
Keywords:
video foundation models, object permanence, object binding, occlusion, counterfactual probing, interpretability, object-centric learning, world models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gkBUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Correction: the previous text was pasted from a review of another submission. It is withdrawn and superseded in full by the review below. I apologize for the error.

This paper studies object binding and object permanence in video foundation models. It removes an object's earlier tokens from a frozen predictor's context and measures object-selective prediction degradation using a calibrated leakage ratio. A separate readout compares predicted features at occluded positions with hidden-object and occluder templates. Experiments across several foundation models, synthetic and real videos, and an object-centric VideoSAUR control find object-selective predictive dependencies but limited hidden-object preference under the proposed readout. Occlusion-completion fine-tuning improves feature and identification scores, while an erased-history counterfactual reproduces most of the feature-level gain. The main contribution is an intervention-based diagnostic separating predictive binding, hidden-object preference, and dependence on object history.

Critical strengths

1. A well-chosen research question and useful counterfactual diagnostic.

Distinguishing decodable object information from information used by a model's own predictions is meaningful for evaluating video representations. The erased-history experiment is particularly valuable: comparing fine-tuning gains when the object did and did not previously exist directly challenges a tempting interpretation of improved occlusion scores. The paper deserves credit for testing and qualifying its own apparent positive result.

2. Substantial controls within the investigated setting.

The evidence is considerably more extensive than a single negative probe result. It includes untrained and shuffled-label endpoints, distance and appearance controls, independent clips, several foundation models, real-video tests, readout-floor calibration, and a paired comparison on 62 clips. These support the reported predictive-dependency and readout-level observations. The acknowledgement of residual history traces and limits on cross-model score comparability is also valuable.

Critical weaknesses

1. The permanence readout and cross-model comparison do not fully identify the stronger maintenance claim (Sections 3.2 and 5).

A predictor trained on visible content may reasonably produce occluder-like features at an occluded location while preserving some object information elsewhere. A non-positive POperm therefore establishes failure to favor the hidden object in this particular output-space contrast, rather than absence of persistent object information in general. The visible-patch floor correction addresses a concrete estimator bias, but does not resolve this distinction. Section 6.2 itself finds subdominant object-history traces.

The VideoSAUR comparison makes this issue consequential: its positive score is measured in persistent slot space, whereas the foundation-model score is measured in predicted fill features. VideoSAUR's own mask readout is negative (-0.261). Matching the same clips removes a data-composition explanation but does not make these readouts equivalent. The paper acknowledges this limitation, yet the title and some main-text statements remain stronger than the common measurement supports. This is my principal concern: the empirical diagnostic is useful, but the broader claim about maintenance requires a more precise operational scope or a shared behavioral test.

2. The fine-tuning evidence constrains the mechanism without uniquely establishing it (Section 6.2).

The reproduction of 96-108% of the fill gain in erased-history worlds is persuasive evidence that this continuous feature-level gain does not require the object's history. However, the same section reports that the original-minus-erased identification gap increases from .058 to .156, and that the violation-of-expectation readout changes from -.021 to +.012. These results make it important to distinguish a history-independent feature shift from improved history-dependent behavior. The proposed explanation that a generic prior amplifies an existing weak trace is plausible, but its direct event-level prediction is supported only in direction (Spearman +.18, p=.176). The categorical statement that the gain adds no tracking therefore needs tighter qualification.

This interpretive limitation is compounded by the evaluated scope: all tested foundation models are feedforward predictors without persistent per-object state, the positive object-centric side uses one architecture, and real-video occlusions are pasted. These are acknowledged limitations, but they constrain how broadly the reported dissociation can be generalized. My concern is whether the existing observations sufficiently establish the claimed mechanism, rather than a lack of experimental effort.

Questions for authors

1. Operational meaning of permanence. Can the authors state precisely which maintenance claim is falsified by non-positive POperm, and distinguish it from retention of subdominant object information or prediction of visible content? How should VideoSAUR's positive slot score and negative mask readout be reconciled when interpreting the paired cross-model gap? A common identity- or future-behavior readout would be especially informative if available.

2. Feature-level versus behavioral history dependence. How do the authors reconcile the nearly history-independent fill gain with the increased original-minus-erased identification gap and the positive violation-of-expectation result after fine-tuning? What existing evidence separates amplification of a fixed trace from an improvement in history-dependent processing? Please distinguish results that rule out a history-dependent explanation of the fill gain from those that establish the proposed prior-amplification mechanism.

3. Scope of the conclusion. Would the authors narrow the main claim to the tested feedforward predictors and the specified readouts, as the Limitations section already suggests? Broader tests on persistent-state video models and natural occlusions would be valuable future work; my present assessment primarily depends on clarifying the existing evidence in Questions 1-2.

Reviewer_Q5A1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigated whether object permanence and object binding emerge spontaneously in video-prediction models such as V-JEPA without explicitly include them as a part of training objectives or assumptions of the model. This is interesting and potentially important because these video-prediction models are used for world models. The authors showed that while the models exhibit some evidence of object binding, it does not show object permanence. The results do provide some insights and potentially useful approach for evaluating object permanence, and binding, but the paper does not demonstrate how to instill object permanence in world models.

Critical strengths
  1. Investigating spontaneous emergence of binding and permanence is legitimate and well-motivated.
  2. Developed a way to evaluate object permanence and showed failure in object permanence, interesting negative findings, though not surprising.
  3. Erase-history counterfactual approach is a novel and a useful contribution.
Critical weaknesses
  1. The results are not too surprising though by no means a foregone conclusion.
  2. Did not introduce approach or demonstrate how to instill object permanence in world model.
  3. The paper is descriptive rather than explanatory. It does not explain why binding is there and permanence is not.
  4. The proposed metrics for object binding and permanence while not unreasonable, may not completely rule out other aspects of the internal states that will produce behaviors consistent with having object binding and permanence.
Questions for authors

Can you answer Questions 2-4 in the Critical Weaknesses section?

Reviewer_J3EiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tests whether video foundation models learn to represent individual objects and whether they can keep track of those objects when they become occluded. To test object binding, the authors remove an object’s earlier tokens from the context of a frozen predictor and measure how this affects future predictions for different objects. They find that a trained V-JEPA 2 model shows much less interference between objects than an untrained model or one with shuffled object labels. Tests on object permanence shows that the predictions from video foundation models fail to preserve the identity of an object hidden behind an occluder. On the other hand, object-centric VideoSAUR model succeeds. Further experiments show that the predicted content tends to follow the object currently covering the hidden region rather than the object that was there before. Fine-tuning on occlusion completion improves these results, but a key control shows that similar gains occur even when the model is given no history of the hidden object. This suggests that fine-tuning teaches a general prior about what should appear behind an occluder rather than enabling the model to remember specific objects. Overall, the paper argues that current video models can distinguish objects but struggle to maintain their identities through occlusion. It proposes the erased-history test as a way to check whether video models actually learn object permanence.

Critical strengths
  • The paper distinguishes between whether a video model contains information about objects and whether it actually uses that information to keep track of objects through occlusion.
  • Testing the model’s own predictions instead of using separate probes is a good approach.
Critical weaknesses
  • I found the interpretation of POperm somewhat unclear. For a predictive model trained to predict visible content, it seems reasonable for the model to predict the occluder’s features at positions where the object is hidden. So, a low POperm may reflect the expected visible-content prediction rather than a lack of object permanence. Could the authors better explain why a model should predict the hidden object rather than the occluder, and how this readout distinguishes object permanence from visible-content prediction? The violation-of-expectation readout seems to provide a more direct test of permanence. I would encourage the authors to give it a more central role and evaluate it consistently across all foundation models.

  • The title makes a broad claim about video foundation models, but the experiments only cover models without persistent memory. It would be helpful to test models that maintain information over time, such as streaming video encoders or SAM 2. The real-video evaluation is also limited to two datasets with pasted occluders. Testing on more natural occlusions would help show how well the findings generalize to real-world videos.

Questions for authors
  • Could the authors better explain why a model should predict the hidden object rather than the occluder, and how this readout distinguishes object permanence from visible-content prediction?

  • The VoE test should calibrate the score using clips where object A is never occluded but is painted out at the same positions. This would help clarify whether the score reflects object permanence or simply differences in how easy the two scenes are to predict.

  • How is the paint-out procedure performed for VoE?

  • Could the authors report VoE scores for all foundation models and VideoSAUR, to make the comparison more complete?

  • Could the authors also evaluate the models on available benchmarks (IntPhys, IntPhys 2, etc.)? This would help connect the results to prior reports of strong V-JEPA performance on physical understanding tasks.

  • Do these results extend to video models with memory?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.