acceptodds
Withdrawn from ICLR 2027

ReCollage: Self-Supervised Token Collages as Memory for Controllable Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long video generation requires compact memory that preserves past content despite prolonged absences, repetitive observations, and distractors. Existing approaches often rely on hand-designed compression rules that overlook uneven information density, favor recent observations, or learn new memory representations that require additional adaptation by the generator. An effective compressor should allocate memory according to content, preserve informative evidence across the full history, and remain compatible with the generator’s native representation. Guided by these principles, we introduce ReCollage, a self-supervised reconstruction method that learns to select historical VAE tokens as collage memory for future generation. A context encoder uses learnable queries to form the collage memory, which is supplied as an inpainting condition to a video diffusion model for reconstruction. During training, content-dependent selections that retain informative regions while reducing redundant storage emerge, making collage an effective history representation. The frozen selector supports three interfaces: sparse memory conditioning, frame-level conditioning, and autoregressive KV-cache management, with the latter two requiring no downstream training. We evaluate these interfaces on three revisit benchmarks spanning hand-controlled, camera-controlled, and text-to-video generation. Under matched memory budgets, ReCollage improves recovery of historical objects, scenes, and subjects despite intervening distractors and long repetitive intervals, demonstrating the effectiveness of reconstruction-trained, content-adaptive token selection for long video history compression.

Primary area:
generative models
Keywords:
Memory Compression, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JrrbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReCollage, a self-supervised history memory compression method for long video generation, aiming to address the difficulty of existing methods in allocating limited memory budgets according to content importance. The authors use learnable queries to select important information from historical VAE tokens and train the token selector using the reconstruction loss of a video diffusion model through a differentiable Collage-and-Scatter mechanism.

Critical strengths
  1. History memory compression in long video generation is an important problem, with clear motivation and practical value.

  2. The Collage-and-Scatter design is quite interesting. It learns content-adaptive token selection through self-supervised reconstruction while preserving the original VAE tokens and their positional information.

  3. The experimental performance is good, showing significant improvements in historical content recovery and long-term consistency.

Critical weaknesses
  1. The computational overhead of the method appears relatively high. The context encoder contains multiple Transformer blocks and needs to process historical tokens and learnable queries, but the paper lacks quantitative statistics on additional inference latency, memory consumption, and computational complexity.

  2. The baseline settings in some experiments are problematic, and comparisons with more state-of-the-art baseline methods are needed. For example, Table 2 only directly compares against Uniform and Random, lacking comparisons with other advanced memory selection or retrieval methods.

  3. The experimental scenarios are relatively limited and are mostly synthetic. Some tests simulate long-term memory tasks by repeating static frames, artificially constructing distractor segments, or synthesizing camera trajectories, lacking validation in more complex and natural long video generation scenarios.

Questions for authors
  1. ReCollage mainly learns token selection through historical reconstruction, but information important for reconstruction is not necessarily what future generation requires, correct?

  2. For longer video histories, how do the computational cost and memory consumption of the context encoder change? Could the authors provide corresponding efficiency comparisons?

  3. Could the authors include more complex real-world long video scenarios and comparisons with other advanced memory management methods to further validate the generalization ability of the proposed method?

Reviewer_iVSdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses long-term consistency degradation in video generation, aiming to preserve consistency with a limited history context. The authors propose a collage-and-scatter strategy for token selection and spatiotemporal placement. Experimental results demonstrate that the proposed method performs well across hand-controlled, camera-controlled, and text-conditioned video generation, achieving strong scene consistency under matched memory budgets.

Critical strengths
  1. The motivating observation is well founded: informative content is unevenly distributed across video history. Brief appearances of important objects or subjects may be more valuable than long, repetitive intervals, providing a reasonable motivation for content-adaptive memory allocation.

  2. The authors present a coherent framework, including a differentiable collage-and-scatter mechanism that jointly learns token selection and placement at source positions. This enables the selection policy to be learned through self-supervised reconstruction without explicit token-importance annotations.

Critical weaknesses
  1. The benchmark selection is somewhat unconventional. The evaluation relies on specialized hand-controlled and camera-controlled generation settings, together with customized recall protocols. Although a text-conditioned generation experiment is included, the lack of evaluation on widely adopted long-video generation benchmarks makes it difficult to assess the method’s broader applicability.

  2. The coverage of baseline methods is insufficiently comprehensive. Representative history-compression approaches, such as FramePack and LoViC, are missing from the direct quantitative comparisons. Including these methods under comparable experimental conditions would help clarify the advantages of the proposed approach over existing compression strategies.

Questions for authors
  1. Could the authors provide a baseline that uses the full, uncompressed history without collage-based selection? A controlled comparison using the same generator, at feasible history lengths, would help quantify the quality trade-off introduced by compression.

  2. Could the authors report end-to-end inference speed, including FPS or latency, as well as peak GPU memory usage? These measurements should include the overhead of the context encoder and token selection, with the hardware, resolution, and history length specified. This would clarify whether collage memory provides practical acceleration or primarily enables better use of a limited context budget.

  3. Could the authors supplement the evaluation with a more widely used benchmark, such as generation from MovieGenBench prompts evaluated using VBench-Long? This would complement the customized recall tests and facilitate comparison with existing long-video generation methods.

Reviewer_DCqUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses history compression for long-video generation with a self-supervised method, ReCollage. A context encoder uses learnable queries to attention-pool native VAE tokens into a token collage; to make token selection differentiable, the collage is softly scattered back to the source grid via the transpose of the assignment matrix, and a video diffusion model reconstructs the history under an inpainting condition, with reconstruction gradients training both selection and placement. The selector is then frozen and supports three interfaces: sparse token conditioning , frame-level conditioning, and online retention/eviction for autoregressive KV caches. The method is evaluated on hand-controlled, camera-controlled, and streaming text-conditioned benchmarks.

Critical strengths

The collage-and-scatter construction is a clean differentiable design: by softly scattering back to the source grid it avoids the gradient-breaking caused by discrete token selection, and it keeps memory in the native VAE token space with source coordinates, which facilitates downstream conditional generation. The frozen selector is shown to work across three different interfaces and benchmarks, including two training-free policies, and it compares against TinyHistory under matched memory budgets, winning most metrics, with ablations over slot counts and chunk-wise versus one-shot selection.

Critical weaknesses
  1. The three properties (content-adaptivity, full-history coverage, generator compatibility) are a standard checklist of memory requirements with little depth; the paper neither derives the collage design from them nor identifies a shared root cause of prior failures, leaving the causal chain between properties and method missing.

  2. The selector itself has a train-test gap: it is trained with soft assignments but deployed with hard argmax or frame scores, and the performance loss of hard selection relative to soft selection is not quantified.

  3. The evaluation is fragmented: the three interfaces span three benchmarks with disjoint metric sets and no unified head-to-head comparison.

Questions for authors
  1. Can you identify a shared root cause of the failure of the three prior families and formally derive the collage design from the three properties?

  2. What is the performance gap between soft selection (training) and hard argmax selection (deployment)?

  3. Can the three interfaces be evaluated on a shared benchmark with a common metric set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.