- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes GenMem, a compact video memory representation that compresses every four video frames into 2–32 continuous tokens using a Q-Former-style encoder. The main idea is to train the memory representation through a pretrained video diffusion model rather than relying solely on conventional reconstruction supervision. The method also uses prefix-based training to support flexible token budgets and a zoom-based augmentation to encourage the use of distant history during video continuation. Experiments show improved semantic and motion preservation at low token budgets and stronger subject and background consistency in the evaluated video continuation tasks.
Critical strengths
The paper addresses an important problem in long-context video generation, and learning memory representations according to the needs of a downstream generator is a reasonable and potentially useful direction.
The results at very low token budgets are promising, particularly the improvements in semantic and motion preservation over reconstruction-based memory, and the zoom-augmented continuation experiments provide some evidence that the compressed representation can retain information useful beyond the recent context.
Critical weaknesses
The main concern is that the contribution of generative supervision is not sufficiently isolated. The paper presents flow-matching supervision as the key reason for the improved memory representations, but the actual objective also includes text alignment, global feature alignment, spatial feature alignment, and isotropic regularization. These auxiliary losses provide additional supervision for precisely the semantic and spatial information that the method aims to preserve, so the reported improvements cannot be attributed to flow matching alone. Table 3 compares different training strategies but does not isolate the contributions of these individual losses. At minimum, the paper should compare flow matching alone against the full objective and provide targeted ablations of the alignment terms.
The experimental presentation does not fully reflect the paper’s main contribution. Although GenMem-G is the variant emphasized in the title and abstract, Table 1 and Figure 3 present only GenMem-R in the main tokenizer comparison. GenMem-G is introduced more systematically only in later experiments, despite its low-budget semantic and motion preservation being central to the paper’s claims. Including GenMem-G under matched evaluation conditions would provide a more complete picture of the trade-off between reconstruction fidelity and generation-oriented memory quality.
The evidence for scalability to long video histories is also limited relative to the claims. The continuation experiments evaluate the generation of a subsequent five-second segment from histories drawn from videos lasting approximately 30–60 seconds, while the abstract emphasizes conditioning on several minutes of history. The paper does not provide sufficient measurements of memory encoding cost, generation latency, or peak memory usage as history length increases. The current results therefore support the usefulness of the proposed representation for the evaluated settings, but do not yet establish its practical efficiency or performance at the claimed longer horizons.
Questions for authors
Can the authors provide controlled ablations comparing flow matching alone with the full training objective, as well as the individual contributions of text, global, and spatial alignment? In particular, how much of the low-budget semantic and motion preservation improvement remains when the auxiliary alignment losses are removed?
Can the authors clarify the actual history lengths and memory budgets used in the continuation experiments, and provide measurements of encoding cost, generation latency, and peak memory usage at increasing history lengths? It would also be helpful to report GenMem-G in the main tokenizer comparison under the same evaluation protocol as GenMem-R.
8 comments
1. Do the gains come from auxiliary alignment losses rather than generative supervision?
No, auxiliary alignment losses do not by themselves explain the gains from replacing reconstruction supervision with generative supervision. In the training-strategy ablation study in main-paper Table 3, we use the same auxiliary alignment losses across all four training strategies. The reconstruction-loss baseline also uses auxiliary losses. However, the results show that training the memory encoder from scratch with the generation loss preserves semantics and motion better at low budgets than these alternatives. These results show that the gains come from how the memory is trained with generative supervision, not simply from adding auxiliary alignment losses.
2. How does flow matching alone compare with the full training objective?
Training with flow matching alone or the full loss yields similar flow-matching loss values and downward trends. Within the rebuttal's limited time and compute budget, we conduct a controlled image-stage ablation by training the memory encoder with either flow matching alone or the full training objective. Both runs use the same images, frozen video backbone, and training settings. For a consistent comparison, we track the flow-matching loss component in both runs over 20k training steps. The loss curves in supplementary Sec. F, Fig. 24 show similar optimization behavior at both 32-token and 2-token budgets.
Thank you for your feedback
We appreciate the reviewer’s assessment that our work addresses an important problem in long-context video generation and shows promising improvements in semantic and motion preservation at very low token budgets. We provide additional experimental results and clarifications in response to the reviewer’s feedback.