ReCollage: Self-Supervised Token Collages as Memory for Controllable Video Generation

Long video generation requires compact memory that preserves past content despite prolonged absences, repetitive observations, and distractors. Existing approaches often rely on hand-designed compression rules that overlook uneven information density, favor recent observations, or learn new memory representations that require additional adaptation by the generator. An effective compressor should allocate memory according to content, preserve informative evidence across the full history, and remain compatible with the generator’s native representation. Guided by these principles, we introduce ReCollage, a self-supervised reconstruction method that learns to select historical VAE tokens as collage memory for future generation. A context encoder uses learnable queries to form the collage memory, which is supplied as an inpainting condition to a video diffusion model for reconstruction. During training, content-dependent selections that retain informative regions while reducing redundant storage emerge, making collage an effective history representation. The frozen selector supports three interfaces: sparse memory conditioning, frame-level conditioning, and autoregressive KV-cache management, with the latter two requiring no downstream training. We evaluate these interfaces on three revisit benchmarks spanning hand-controlled, camera-controlled, and text-to-video generation. Under matched memory budgets, ReCollage improves recovery of historical objects, scenes, and subjects despite intervening distractors and long repetitive intervals, demonstrating the effectiveness of reconstruction-trained, content-adaptive token selection for long video history compression.