acceptodds
Under review as a conference paper at ICLR 2027

GenMem: Learning Ultra-Compact Video Memory by Generative Supervision

Abstract

Recent video generation models can produce long videos, but maintaining long-term consistency remains challenging because conditioning on extensive visual history requires prohibitive memory and computation. We introduce GenMem, a compact visual memory for long-context video generation. GenMem can compress 4 video frames of 480p resolution into only 32 tokens while retaining high-fidelity visual information. Our memory combines flexible Q-Former-based token compression with a hierarchical representation that prioritizes important semantics, appearance, and dynamics within a small token budget. To train our memory encoder, we find that supervision from the generation loss enables more aggressive compression while better preserving semantically relevant information than using vanilla reconstruction loss. This compact design allows video generators to efficiently condition on several minutes of visual history. Experiments show that GenMem enables substantially more scalable long-context generation while preserving the consistency of scenes and objects observed from distant video history.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Token Compression, Video Generation, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a3H7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GenMem, a compact video memory representation that compresses every four video frames into 2–32 continuous tokens using a Q-Former-style encoder. The main idea is to train the memory representation through a pretrained video diffusion model rather than relying solely on conventional reconstruction supervision. The method also uses prefix-based training to support flexible token budgets and a zoom-based augmentation to encourage the use of distant history during video continuation. Experiments show improved semantic and motion preservation at low token budgets and stronger subject and background consistency in the evaluated video continuation tasks.

Critical strengths

The paper addresses an important problem in long-context video generation, and learning memory representations according to the needs of a downstream generator is a reasonable and potentially useful direction.

The results at very low token budgets are promising, particularly the improvements in semantic and motion preservation over reconstruction-based memory, and the zoom-augmented continuation experiments provide some evidence that the compressed representation can retain information useful beyond the recent context.

Critical weaknesses

The main concern is that the contribution of generative supervision is not sufficiently isolated. The paper presents flow-matching supervision as the key reason for the improved memory representations, but the actual objective also includes text alignment, global feature alignment, spatial feature alignment, and isotropic regularization. These auxiliary losses provide additional supervision for precisely the semantic and spatial information that the method aims to preserve, so the reported improvements cannot be attributed to flow matching alone. Table 3 compares different training strategies but does not isolate the contributions of these individual losses. At minimum, the paper should compare flow matching alone against the full objective and provide targeted ablations of the alignment terms.

The experimental presentation does not fully reflect the paper’s main contribution. Although GenMem-G is the variant emphasized in the title and abstract, Table 1 and Figure 3 present only GenMem-R in the main tokenizer comparison. GenMem-G is introduced more systematically only in later experiments, despite its low-budget semantic and motion preservation being central to the paper’s claims. Including GenMem-G under matched evaluation conditions would provide a more complete picture of the trade-off between reconstruction fidelity and generation-oriented memory quality.

The evidence for scalability to long video histories is also limited relative to the claims. The continuation experiments evaluate the generation of a subsequent five-second segment from histories drawn from videos lasting approximately 30–60 seconds, while the abstract emphasizes conditioning on several minutes of history. The paper does not provide sufficient measurements of memory encoding cost, generation latency, or peak memory usage as history length increases. The current results therefore support the usefulness of the proposed representation for the evaluated settings, but do not yet establish its practical efficiency or performance at the claimed longer horizons.

Questions for authors

Can the authors provide controlled ablations comparing flow matching alone with the full training objective, as well as the individual contributions of text, global, and spatial alignment? In particular, how much of the low-budget semantic and motion preservation improvement remains when the auxiliary alignment losses are removed?

Can the authors clarify the actual history lengths and memory budgets used in the continuation experiments, and provide measurements of encoding cost, generation latency, and peak memory usage at increasing history lengths? It would also be helpful to report GenMem-G in the main tokenizer comparison under the same evaluation protocol as GenMem-R.

8 comments
AuthorsOpenReview ↗

Thank you for your feedback

We appreciate the reviewer’s assessment that our work addresses an important problem in long-context video generation and shows promising improvements in semantic and motion preservation at very low token budgets. We provide additional experimental results and clarifications in response to the reviewer’s feedback.

AuthorsOpenReview ↗

1. Do the gains come from auxiliary alignment losses rather than generative supervision?

No, auxiliary alignment losses do not by themselves explain the gains from replacing reconstruction supervision with generative supervision. In the training-strategy ablation study in main-paper Table 3, we use the same auxiliary alignment losses across all four training strategies. The reconstruction-loss baseline also uses auxiliary losses. However, the results show that training the memory encoder from scratch with the generation loss preserves semantics and motion better at low budgets than these alternatives. These results show that the gains come from how the memory is trained with generative supervision, not simply from adding auxiliary alignment losses.

AuthorsOpenReview ↗

2. How does flow matching alone compare with the full training objective?

Training with flow matching alone or the full loss yields similar flow-matching loss values and downward trends. Within the rebuttal's limited time and compute budget, we conduct a controlled image-stage ablation by training the memory encoder with either flow matching alone or the full training objective. Both runs use the same images, frozen video backbone, and training settings. For a consistent comparison, we track the flow-matching loss component in both runs over 20k training steps. The loss curves in supplementary Sec. F, Fig. 24 show similar optimization behavior at both 32-token and 2-token budgets.

Reviewer_RqQuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GenMem, a compact video memory representation for long-context video generation. The method uses a Q-Former-based encoder to compress video frames into a small number of memory tokens, which are trained with generative supervision from a pretrained video diffusion model. It also adopts prefix-based training to support flexible token budgets and zoom augmentation to encourage the use of distant video history. Experimental results show that GenMem can compress four frames into 2–32 tokens while maintaining semantic and motion information, and improves subject and background consistency in long-video continuation.

Critical strengths
  • The main contribution of this paper is shifting the training objective of memory compression from reconstruction to generation. By using the flow matching loss of a pretrained Video DiT, the method can leverage the generator's learned visual priors to achieve more effective compression.
  • The proposed tail-token dropout encourages the earlier memory tokens to capture coarse and essential information, while later tokens provide additional details. This is a reasonable design for learning a hierarchical representation that can preserve important information even with a very small number of tokens.
  • Rather than simply providing compressed memory to the generator, the authors design training scenarios where information from distant frames is needed for generation. This helps address the problem of whether the model actually uses its long-term memory.
Critical weaknesses
  • The long-video continuation experiments are conducted on an internal dataset, including 100 held-out evaluation videos. Since the evaluation data is not publicly available, it is difficult for future studies to reproduce the results or compare new methods under the same conditions. Providing a public evaluation set or a reproducible benchmark protocol would make the experimental results more useful for future research.
  • While GenMem demonstrates improved consistency under zoom augmentation, it remains unclear how effectively the model utilizes increasingly distant memories. Evaluating generation consistency across different temporal gaps (e.g., 5s, 10s, 20s, and 30s), while keeping the total history length fixed, would help determine whether long-term memory retrieval remains effective as temporal distance increases.
Questions for authors

Overall, I have a positive impression of this work. However, as noted in the Critical Weaknesses, I encourage the authors to release the training and evaluation datasets to support reproducibility and future research.

Additionally, the evaluation is limited to single-shot videos, leaving subject identity consistency across scene transitions unexplored. If feasible, I encourage the authors to evaluate multi-shot scenarios with changing backgrounds. Otherwise, a discussion of the expected effectiveness and potential limitations of GenMem in such settings would be valuable.

7 comments
AuthorsOpenReview ↗

Thank you for your feedback

We appreciate the reviewer’s positive impression of our work and the recognition of the benefits of generative supervision, hierarchical memory, and our training design for encouraging the use of distant history. Below, we respond to the reviewer’s suggestions with additional experiments and clarifications.

AuthorsOpenReview ↗

1. Will you provide a public evaluation set?

Yes. We will release the 100 single-shot long videos used for both standard and zoom-augmented continuation evaluation, along with the data-processing and evaluation code, to provide a reproducible benchmark protocol. For memory learning, we use public LAION-5B [2] images and Koala-36M [3] videos for training, and we will open-source the code and metadata for reproducing the training dataset. We have included the IDs of 500 OpenVid-1M [4] videos used in our evaluation in supplementary Sec. I.

AuthorsOpenReview ↗

2. Does long-term memory retrieval remain effective as temporal distance increases?

Yes. From a 5 s to a 30 s gap, video quality stays stable, while subject and background consistency decrease by only 1.51 and 0.93 points. We randomly sample 30 videos and evaluate different temporal gaps while keeping the total history length fixed at 35 s. We vary the duration of a 2× zoom-in from 5 to 30 s, then generate a 5 s zoom-out continuation that must recover content from the earlier full view. These results show that GenMem continues to use relevant history effectively even as it becomes more distant.

Temporal gap (s)Quality ↑Subject ↑Background ↑
578.4485.8390.39
1077.9985.7290.36
2078.0785.3790.07
3078.3584.3289.46
Reviewer_GyHgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GenMem, an ultra-compact video memory for long-context video generation. A Q-Former-style encoder compresses four frames into 2–32 memory tokens, which are organized in a hierarchical, coarse-to-fine prefix via token dropout. The encoder is trained with the flow-matching loss of a frozen pre-trained generator rather than a standalone reconstruction loss, and a temporal zoom augmentation encourages the generator to retrieve content from distant history. Experiments on video reconstruction and long-video continuation demonstrate aggressive compression at matched token budgets.

Critical strengths

The compression results are strong and practically relevant: at matched token budgets, GenMem substantially outperforms recent video tokenizers (e.g., Table 3 vs. VideoFlexTok), while retaining semantic and motion information. The comparison between reconstruction and generative supervision across budgets (Table 3, Fig. 4) is informative and supports the claim that the training objective matters at extreme compression. The addressed problem—compact memory for long-horizon video generation—is timely and important, and the paper is clearly written with a coherent system design.

Critical weaknesses
  1. Limited novelty in two of the core designs.

    • Query-based resampling with learnable queries is a standard, widely used mechanism (Perceiver/Perceiver IO; Flamingo's Perceiver Resampler; BLIP-2 Q-Former) and is already adopted in recent tokenizers (FlexTok; VideoFlexTok). Presenting it as a key design overstates its contribution.
    • Training memory/context modules with the generator's loss is also established in long-video generation (e.g., LongLive, MAG, FramePack; TinyHistory's second stage). The incremental difference—frozen generator, prefix dropout, flexible inference budgets—should be stated more precisely rather than contrasting only with reconstruction-trained tokenizers.
  2. Incomplete efficiency evidence. Token compression is motivated by prohibitive memory and computation costs, yet no GPU memory, latency, throughput, or FLOPs comparisons are reported; compression ratio alone is insufficient to assess practical gains.

  3. Missing key baseline. TinyHistory is arguably the most directly related work (lightweight history embeddings with a two-stage scheme including generator-side fine-tuning), and the introduction even claims a 4× token advantage over it, but no empirical comparison is provided.

Questions for authors
  1. Please report end-to-end efficiency (GPU memory, latency, throughput/FLOPs) against the strongest baselines at matched settings.

  2. Please add a comparison with TinyHistory (and ideally StreamingT2V) under the same protocol, including at matched token budgets.

  3. What is the precise novelty over VideoFlexTok and TinyHistory, and can the method be validated on at least one additional backbone?

9 comments
AuthorsOpenReview ↗

Thank you for your feedback

We appreciate the reviewer’s recognition of the strong compression results, the importance of compact video memory, and the value of our comparison between reconstruction and generative supervision. We address the reviewer’s concerns and clarify our contributions below.

AuthorsOpenReview ↗

1. How is GenMem different from VideoFlexTok?

We agree that both our work and VideoFlexTok adopt query-based transformers. However, we target a different task. GenMem is designed to compress long videos into compact memory that conditions subsequent video generation, while VideoFlexTok [1] aims to learn a hierarchical video representation for autoregressive generation. VideoFlexTok does not focus on compressing video history into memory for long-video generation, and its video generation demonstrations are limited to 10 seconds. In comparison, GenMem supports long-video generation through compact history memory. It also achieves higher reconstruction quality while using 8× fewer tokens than VideoFlexTok, as shown in main-paper Table 1.

Our contribution is not to design a new network architecture or structure, but to find an effective way of using query-based transformers to learn compact video memory. We show that this memory can retain the information needed by the video generator even under very small token budgets, supporting high-quality video generation that remains responsive to text prompts.

AuthorsOpenReview ↗

2. What does GenMem add beyond using the generator’s loss?

GenMem uses the generation loss to learn what memory tokens should retain or discard at different compression budgets. LongLive [5], TinyHistory [2], and FramePack [6] use the generation loss to improve generation with memory at fixed compression rates. These methods use the generation loss to improve generation quality, but do not demonstrate how it can be used to learn a more compact and informative memory representation. MAG [3] also uses the generation loss to learn compressed memory, but it learns to compress the generator’s key-value (KV) caches. GenMem instead learns compact memory tokens through a query-based transformer, achieving stronger compression and better quality. Our ablations in main-paper Table 3 further show that using the same generation loss does not guarantee effective memory: how the memory is trained is also crucial.

Comments on the paper

2 comments

Hi Reviewers,

The authors have posted their rebuttal along with new results. Please review their responses, engage with the authors to discuss any remaining concerns.

Best, Area Chair

Hi Reviewers,

The authors have posted their rebuttal along with new results. Please review their responses, engage with the authors to discuss any remaining concerns.

Best, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.