acceptodds
Withdrawn from ICLR 2027

SLES: Self-Localized Entity Supervision for Long-Horizon Dynamic Memory in Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon simulation of a persistent world requires maintaining the state of dynamic entities even while they lie outside the field of view, recovering each with its identity and evolving state intact upon re-entry; we refer to this capability as long-horizon dynamic memory. Despite rapid advances in generation quality and controllability, existing models frequently fail to recover entities that have left the field of view, or do so with a different identity or implausible dynamics. We study this problem under full-context bidirectional generation, where the complete observation history remains accessible, and find that failures persist, indicating that information utilization, rather than access, is the bottleneck. Probing self-attention reveals an observability–controllability gap: entity retrieval is encoded in reproducible attention structure, yet directly steering this routing does not reliably improve generation and can degrade identity consistency. We therefore use attention only to localize intervention. In this paper, we propose Self-Localized Entity Supervision (SLES), which derives a lightweight spatial entity support from text prior and reuses it to up-weight entity-aware supervision during training and to localize a Langevin-inspired overshooting sampler that concentrates stochastic correction on the entity region during inference, requiring no external memory, architectural modification, or target-side masks. We further introduce a phase-aware protocol separating reappearance, subject consistency, and temporal recovery. Across backbones and ablations, self-derived support approaches ground-truth mask supervision for identity recovery, while localized overshooting improves post-re-entry dynamics with limited collateral disturbance.

Primary area:
generative models
Keywords:
video world models, video generation, long-horizon consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XL45OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SLES reads text cross-attention to locate a subject region, doubles the flow-matching loss weight there, and applies localized overshooting and re-noising during sampling. The text signal selects where to intervene; the ground-truth training target is unchanged. The main paper and appendix report identity-recovery and temporal-stability gains under the stated full-context bidirectional protocol, including a 345-scene evaluation and transfer to LTX-2.5.

Critical strengths
  1. The appendix strengthens the controlled recovery evidence. On all 345 held-out scenes, training raises SC from 0.8078 to 0.8206 and reappearance from 96.7% to 98.3% (Table D6). Together with the GT-mask and scrambled-support controls, this supports useful spatially focused supervision without per-sample target masks after calibration. 2.The additional analyses make the contribution easier to assess. Appendix B documents retrieval shifts and unsuccessful direct attention steering; A.3 and Fig. D2 clarify the complementary identity and temporal evidence. The sampling dose-response analysis in E.1 supports a trade-off between temporal-stability gains and full-frame perceptual change, rather than a claim that localization always maximizes TCM.
Critical weaknesses
  1. Causal training is needed to establish the method's practical value for autoregressive world models. I did not find a causally trained SLES comparison in the supplied main paper or appendix. A matched causal baseline and SLES, trained and rolled out with exit and re-entry in different generation blocks, would test this practical memory claim. Applying a causal mask only at inference would introduce a training-inference mismatch. 2.The appendix documents missed support near re-entry. Fig. C1 shows misses around leaving and returning transitions; across 94 validation scenes, only 31% of frames with 1-5-token subjects receive support, versus 93% for subjects larger than 40 tokens. Unsupported tokens receive no extra loss weight or local sampling correction. These diagnostics are useful, but aggregate recovery scores do not show how much SLES helps small returning subjects or the first post-entry frames.
Questions for authors
  1. Could you compare a baseline and SLES both trained with the same frame- or block-causal mask, then run multiple generation blocks with earlier outputs fixed and exit/re-entry spanning different blocks?
  2. How much of the recovery gain remains for small returning subjects and the first few post-entry frames?
Reviewer_Hoz2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SLES to improve entity identity and dynamics after disappearance and re-entry in video world models. It uses the model’s own text cross-attention to localize entity regions, strengthen training supervision, and guide localized stochastic correction during sampling.

Critical strengths
  • The paper addresses a relevant problem in video world modeling.
  • The method requires no architectural changes, and the ablations help assess the contributions of localized supervision and sampling.
Critical weaknesses
  • My major concen is the limited evidence of generalization. The formulation focuses on a designated subject and extracts support around a dominant attention peak. It is unclear how this extends to complex scenes containing many entities, interactions, or substantial background changes. Experiments with more subjects would help establish whether the method supports broader world consistency.
  • No video results are provided. It's hard to verify the visual performance. The provided material does not include playable video demonstrations. Selected frames and aggregate metrics are insufficient to assess temporal coherence, identity preservation, and overall visual quality throughout disappearance and re-entry.
Questions for authors

My major concen is the generalization ability of the proposed method.

Reviewer_Sq5oOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of preserving the state of dynamic entities in long-horizon video generation, particularly when entities reappear after leaving the frame. The authors leverage the emergent semantic grounding capability of video diffusion models to localize entity regions only in frames where the entity is visible, and train these regions with different noise levels. On newly proposed benchmarks, the method shows performance gains over baselines.

Critical strengths

Clear motivation and problem definition. As illustrated in Figure 1, the authors identify inconsistencies in dynamic objects that existing evaluation metrics fail to capture. This makes a convincing case for the importance of the task.

Critical weaknesses
  1. Insufficient analysis and use of the observability–controllability gap. Section 3.2 lays the foundation for the main method, but it needs more detailed explanation and supporting analysis experiments. In addition, the method uses text-to-visual attention only to localize entity regions. An off-the-shelf segmentation model such as SAM2 could likely fill the same role at little additional cost, which weakens the contribution. The emergent entity features or correspondences could be used more directly in the generation process, and doing so would strengthen the paper’s contribution.

  2. Limited qualitative results. The paper includes few qualitative figures and provides no supplementary videos. For a video generation paper, this makes it difficult to assess temporal consistency and weakens the authors’ claims.

Questions for authors

In Section 5.1 (L372–373), the authors state that “Rea-Score decreases from 99.2% to 97.2%, but this direction is not stable across sampling seeds, so we do not interpret it as a systematic trade-off.” Could the authors clarify what this instability refers to? Does it indicate that the proposed Rea-Score metric itself is unreliable across seeds, or that the sampling method produces inconsistent results? Please also provide a fuller interpretation of the performance drop observed in SLES-Full.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.