acceptodds
Under review as a conference paper at ICLR 2027

Which Latent Representations Support Object-Permanent Generation?

Abstract

Video generators often fail to preserve objects that leave the view: occluded ob- jects change identity, fail to return or evolve inconsistently. We ask whether the latent a generator is trained on can natively support object-permanent generation. Our intuition is that a latent encoding the whole scene across cameras and time, rather than one camera’s partial view, can represent objects beyond their visibil- ity. We build PERMAKUBE, a synchronized multi-camera dataset with controlled occlusions and object-level annotations, design PERMATOK, a scene tokenizer trained to render unobserved cameras and times, and train generators on frozen video and scene latents. Under every tokenizer supervision we test, scene la- tents preserve occluded objects more often than video latents despite lower recon- struction fidelity (78.7% versus 47.7% preservation), while accurate motion after occlusion remains challenging for all latents. Cross-view targets add what same- camera targets lack: their encoded scene tokens let a probe trained only on visible objects recover objects hidden from one camera, and their generations reach twice the hide-and-reappear success at similar identity preservation. PERMATOK also adapts to monocular input while retaining strong object-permanent generation, and scene latents are effective diffusion targets built from sparse multi-camera ob- servations. These findings favor scene latents for object-permanent generation and show that identity preservation does not guarantee a readily recoverable hidden- object state, motivating latent probing alongside frame-level evaluation.

Primary area:
generative models
Keywords:
latent representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_euwQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of Object Permanence in video generation. The authors construct a multi-camera synthetic dataset, PermaKube, and an evaluation benchmark, PermaEval, and propose a scene representation based on cross-view and cross-time supervision, which encodes multi-view videos into a unified Scene Latent. Compared with traditional Video Latents, Scene Latents significantly improve object identity preservation after occlusion.

Critical strengths
  1. The paper investigates Object Permanence from the perspective of Latent Representation, which is meaningful for addressing long-term consistency issues in video generation.

  2. The proposed PermaKube and PermaEval support controlled occlusions and multi-view evaluation, and relatively extensive ablation experiments are conducted with different supervision strategies.

  3. The model achieves good performance, with PermaTok reaching an object identity preservation rate of 78.7%, significantly outperforming WF-VAE's 47.7%.

Critical weaknesses
  1. The experimental evaluation lacks validation on real-world videos or more complex dynamic scenes. More complex datasets may be needed to demonstrate the effectiveness of the model.

  2. The comparison between Scene Latents and Video Latents involves differences in network architecture, latent dimensionality, temporal sampling, and training budgets. The ablation studies do not sufficiently control these variables, and more experimental results are needed.

  3. Latent Probing is mainly performed on Scene Tokens before compression, rather than the compressed Latents actually used for generation. This is a clear distinction and may require further validation.

Questions for authors
  1. I am curious whether there are more realistic or complex datasets in this field. Could the authors further evaluate the generalization ability of PermaTok on real-world videos, complex occlusions, and nonlinear motion scenarios?

  2. If Probing is directly performed on the compressed Latents, or even the Latents predicted by the generator, would similar Hidden-object Recovery capabilities still be observed?

Since there is no borderline option in the current rating scale, I assigned this score.

1 comment

Thank you for the experiments. They addressed several of my concerns. However, I remain concerned that, the performance gains cannot be fully attributed to the scene latent representation (the tokenizer architectures are different). And in particular, PermaTok-SC achieves 74.7% identity preservation but shows little hidden-object recovery in probing. This raises questions about the relationship between hidden-object representations and generation performance. And, the lack of evaluation on realistic scenes and complex motion remains a limitation (this may need further reviewer-ac discussion).

I appreciate the authors' efforts, but I will keep my rating unchanged for now.

Reviewer_RJcqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper raises a critical question on the latent representation used in video models: which representation supports object permanance understanding during generation? To answer it, the authors built PermaKube and PermaEval, a playground for multi-camera, object visibility task training and evaluation. The proposed method, PermaTok, encodes the observation into a compact scene latent, which is then used as the generation target for a diffusion transformer. It is trained with cross-view and cross-time supervision, encouraging the latent to capture scene information beyond what is directly visible from a particular viewpoint.

Critical strengths
  1. The scene latent representation generated by PermaTok shows significant performance improvement over traditional WF-VAE
  2. A latent probing analysis is provided. The authors show that models with similar identity preservation can differ significantly in hidden-object recovery, highlighting why evaluating latent representations matters beyond generation quality.
  3. The single view experiments widens the practical significance and application field, making it feasible in most common single camera settings.
Critical weaknesses
  1. The authors did not provide video supplementary material, making it difficult to thoroughly assess the effectiveness of the proposed method.
  2. The presented PermaKube, which is built on Kubrics, is overly simplistic and idealistic. More evaluation results on real-world objects and scenes are needed to prove the generalizability and robustness of the method.
  3. The main ablation study comparing PermaTok with WF-VAE and SD-VAE, contains variables more than latent repersentations. For example, the compression ratio, and the token budget are different, which may cause performance gap. I think the authors should control these variables to see the influence.
  4. Although identity preservation has improved, predicting motion after occlusion is still difficult. It would be helpful to test more temporal samples or different motion aware representations to see whether the problem comes from having too few temporal anchors or from the scene-latent representation itself.
  5. The probing is only performed on the uncompressed scene tokens, rather than the latents actually used by the generator. It's therefore unclear whether the hidden-object information is preserved during compression and generation.
Questions for authors

Please see the weakness part.

Reviewer_r5CNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper targets video generation that handles re-appearing objects, given a context of multi-camera frames at synchronized times. Such a context is supposed to enable a state-preserving scene latent with two aspects: object permanence and evolution of its motion. The paper claims benefits for the former while showing that the latter remains challenging. The title hints at a study of latent representations, yet all comparisons are against two image/video autoencoders that differ in many uncontrolled ways (training data, context frames). The abstract advertises improvement in the metric introduced only in the appendix (hide-and-reappear success, Appendix B.2).

The paper makes highly unrealistic assumptions: it stays within a purely synthetic custom dataset, does not do any tests on real data, and does not compare to any off-the-shelf baselines. For real-world data collections, this setting is expensive and extremely challenging, making future applicability of the work questionable.

In presentation, the paper leans heavily on technical details (hyperparameters, exact shapes, frame time ids) while lacking convincing motivation and omitting details crucial to understanding the proposed architectures.

Given these fundamental concerns, I recommend rejection.

Critical strengths
  • An interesting design concept (a scene-level latent target for permanence).
  • Fair limitations are stated in the main text.

Unfortunately, neither is sufficient for acceptance on its own.

Critical weaknesses

Highly restrictive setting and limited scope.

  • Multi-camera context is under-motivated. Not convincing why expensive/synthetic multi-camera context is needed, if a single-video context may already contain useful past frames with an object revealing both identity and initial motion.
  • All experiments stay within the synthetic PermaKube arena, including the monocular evaluation (Sec. 5.2), which could otherwise have been informative on real data.
  • Despite the proposed problem decomposition and restrictive assumptions, motion preservation is still reported as challenging.

Problematic evaluation.

  • No comparison against existing permanence/memory methods. The two baseline tokenizers differ in training data and context frames.
  • Evaluation metrics lack detail and breadth. No VLM judge prompt is given, and there is no justification for PSNR as the only reconstruction metric (no SSIM, DINO, etc.).

Weak motivation behind design choices.

  • Weak placement in the context of related work. The related work states what is done differently but does not contrast existing approaches or explain what they lack.
  • The full description of the framework is missing (inputs and outputs of tokenizer and generator at training and inference, context/query separation across cameras and time). This makes the relevance of design choices hard to assess. For example, technical details (hyperparameters, shapes) precede motivation. The new dataset (Sec. 3) precedes the method that needs it (Sec. 4).
  • Under-motivated latent probing posed as contribution. Circular logic: state that this metric is considered a needed property, then show favors the method and incorrectly conclude measuring it is motivated.
Questions for authors

PermaKube.

  • Why are existing multi-camera 4D datasets (e.g., the cited ones) not usable? What do they lack?
  • Why are Base, Large, and Extension sizes needed?
  • Use of test masks is unclear: what do they contain, how are they used for scoring, and how do training masks weight the tokenizer’s reconstruction loss? (The weighting appears only in Appendix A.2 and should be in the main text.)

Architectural design.

  • The VecSet encoder (Zhang et al., 2023) is a shape/point-cloud encoder. How is it used on scene frame tokens?
  • How is the generator modified to receive both the context and noisy target tokens? Which decoder is used in generation? In particular, why is the output in-distribution for PermaTok’s decoder if the generation is a denoised query-camera latent rather than a full scene latent?
  • Why are learnable latent tokens ee needed in EncsceneEnc_{scene}?
  • Why does the context use specifically frames 2, 8, 14? Wouldn’t consecutive frames be more realistic? Also, the tokenizer (Sec. 4.2) is described with six input times, but the generator’s context (Sec. 4.3) has only three. Doesn’t this create a train/inference distribution shift?

Evaluation.

  • The introduction advertises fewer identity changes or deletions. How does the judge define and detect them?
  • How is motion success in Fig. 3 measured?
  • In Fig. 3 PermaTok has the highest motion success. Why is dynamics concluded to be equally challenging for all latents?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.