- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a self-evolving framework to jointly train a memory construction agent and a retrieval agent for long video understanding. This approach aims to address two key limitations in existing memory-based long video understanding methods: the limited adaptability of memory to diverse videos, and the misalignment between memory and retrieval. Specifically, the proposed method first constructs a base memory and subsequently generates diverse memory candidates based on it. The retrieval agent then answers downstream questions using these memory candidates. The accuracy of the answers to these downstream questions is calculated to formulate a reward score for optimizing the memory agent. Subsequently, the memory is reconstructed, and the retrieval agent is further optimized by calculating its reward score based on the updated memory using the same methodology. Furthermore, Bottleneck-Aware Evolution Feedback (BEF) and Capability-Aware Evolution Feedback (CEF) modules are designed to dynamically allocate the optimization budget. Comparative experiments with existing methods demonstrate the superior effectiveness of the proposed approach over previous memory-based methods.
Critical strengths
- This method addresses the isolation between memory optimization and downstream task feedback in memory-based long video understanding approaches, as well as the lack of consideration for the alignment relationship between retrieval and memory. Such a joint optimization strategy is highly beneficial for long video understanding.
- Furthermore, by designing bottleneck feedback and capability feedback mechanisms, this method effectively addresses the issue that the required optimization intensity for different agents varies dynamically across different stages of the alternating optimization process. This design is both reasonable and effective.
Critical weaknesses
-
The trajectory-level reinforcement learning and alternating training procedure adopted by VideoEvolve do not appear to differ fundamentally from those in EvolveR and Evoving-RL. Although both studies are cited in the related work section, the paper does not clearly articulate its core methodological novelty relative to these approaches.
-
Although the authors describe the method as “co-evolution,” the mutual feedback between its two components appears to be confined to training. During inference, the agents seem to operate independently. It would help to clarify how “co-evolution” benefits inference. Moreover, the method relies heavily on cold-start initialization: the supplementary material reports a performance drop of approximately 20 points without it. This finding highlights the substantial contribution of cold-start initialization, but does not, by itself, establish the benefits of co-evolutionary training.
Questions for authors
-
The current experiments do not establish whether the observed gains arise from genuine alignment between the evolved memory and retriever, or simply because both components become independently stronger after optimization. The authors should conduct cross-checkpoint evaluations using matched and mismatched memory–retriever pairs. For example, they could compare , , and the matched pair . In contrast, the current ablations that optimize only the memory or only the retriever merely demonstrate that improving either component is beneficial; they do not substantiate the central claim that co-evolution aligns the two components.
-
Many methods in Table 1 operate under a fixed video frame budget, with some sampling at most 64 or 128 frames. Similarly, among the memory-based methods in Table 2, World-MM[1] samples up to 768 frames, while MemVid[2] samples 128 frames. In contrast, VideoEvolve constructs its base memory by sampling at 0.5 FPS and needs accessing additional video frames. Therefore, the comparisons may not be entirely fair due to differences in frame budgets. Could the authors provide comparisons with representative methods under the same frame budget to ensure a fair evaluation?
-
In Table 2, VideoEvolve under the training-free setting significantly outperforms its trained counterpart. However, the authors do not analyze the reasons behind such a substantial performance gap. Moreover, the supplementary material indicates that the training-free setting relies solely on base memory, without employing memory augmentation or constructing delta memory. This further raises questions about the necessity and effectiveness of training the Memory Controller.
-
Some issues regarding the presentation and writing:
a) The term appears in Section3.4 of the main text without any prior definition or mention; it is only briefly addressed in the supplementary material. Given the importance of the concept, it should be clearly explained and defined within the main text.
b) in Equation(5) lacks a corresponding formula or explicit definition.
c) In Section3.5, a large number of mathematical notations without clearly explaining their meanings. As a result, it is difficult to understand the specific mechanisms of BEF and CEF, as well as how the optimization budget is dynamically allocated among different Evolvers.
[1] Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, and Sung Ju Hwang. Worldmm: Dynamic multi-modal memory agent for long video reasoning. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25599–25609, 2026.
[2] Huaying Yuan, Zheng Liu, Minghao Qin, Hongjin Qian, Yan Shu, Zhicheng Dou, Ji-Rong Wen, and Nicu Sebe. Memory-enhanced retrieval augmentation for long video understanding. arXiv:2503.09149, 2025.
5 comments
Response to Reviewer 1ETB (2/4)
W2: Inference benefits and gains beyond SFT
1. Inference benefits from memory–retriever alignment learned through co-evolution. During training, QA feedback from Retrieval guides the Controller's evidence preservation, while Retrieval adapts to the updated memory. This mutual adaptation is retained in the learned policies: at inference, the Controller builds reusable memory once per video, and Retrieval searches it to answer questions, without online parameter updates. BEF/CEF operate only during training.
2. Co-evolution goes beyond the performance reached by SFT alone. Removing both SFT stages while retaining RL lowers LVBench from 58.9 to 32.8 (Table 11 in the paper), showing the value of initialization. Two comparisons clarify the additional gains:
(2.a) Roughly doubling SFT data yields limited gains. Stage-II SFT increases from 50,871 to 102,732 complete trajectories (+51,861), and from 174,652 to 349,717 decision samples (+175,065). Both runs start from the same Stage-I checkpoint and train for one epoch. Under matched evaluation settings, accuracy increases by only 0.1/1.1 points on Video-MME w/sub/LVBench, indicating diminishing SFT gains in this tested range.
(2.b) Co-evolution further improves the full-data SFT model. Co-evolution starts from the full-data SFT model, with backbones, frozen Writer, and evaluation settings unchanged. The half-data SFT row is the added control; the full-data SFT and co-evolution results are from Table 3 in the paper:
| Training configuration | Video-MME w/sub | LVBench |
|---|---|---|
| After SFT (half Stage-II data) | 66.7 | 49.2 |
| After SFT (full Stage-II data) | 66.8 | 50.3 |
| Full co-evolution | 73.9 | 58.9 |
| Gain: full vs. half SFT | +0.1 | +1.1 |
| Gain: co-evolution vs. full SFT | +7.1 | +8.6 |
All settings use Base-plus-Delta memory and permit revisiting; Video-MME uses the full set with subtitles. Co-evolution adds 7.1/8.6 points even after full-data SFT. We have added this data-scale comparison to the paper.
Q1: Alignment versus independently stronger components
The cross-checkpoint results support memory–retriever alignment: a later memory can reduce accuracy when mismatched, while the evolved pair delivers gains beyond the sum of separate improvements.
1. The cycle-3 matched pair performs best across all nine combinations. M0/R0 is the post-SFT, pre-RL pair; M1/R1 and M3/R3 are from cycles 1 and 3. We cross-pair them without retraining under memory-only inference, with stored keyframes but no new raw-video frames.
Each cell reports Video-MME Long (w/sub) / LVBench accuracy. The LVBench diagonal and the M3/R3 Video-MME Long result also appear in Tables 3–4 in the paper.
| Memory / Retriever | R0 | R1 | R3 |
|---|---|---|---|
| M0 | 52.1 / 45.6 | 56.9 / 46.1 | 58.2 / 50.0 |
| M1 | 54.3 / 46.5 | 60.4 / 48.5 | 62.6 / 52.8 |
| M3 | 56.9 / 47.8 | 59.1 / 47.7 | 65.5 / 57.5 |
2. The same memory update has opposite effects with different Retrievers. With R1 fixed, replacing its matched M1 with the later M3 lowers accuracy by 1.3/0.8 points. With R3 fixed, the same M1-to-M3 replacement instead raises accuracy by 2.9/4.7 points. Updating memory alone does not ensure an overall gain; the Retriever must also be able to use the updated evidence.
3. Replacing both components yields gains beyond the sum of their separate gains. To quantify the additional benefit, we compare gains over M0/R0 using the same cross-pairings (percentage points):
| Change from M0/R0 | Video-MME Long (w/sub) | LVBench |
|---|---|---|
| Replace memory only: M3/R0 | +4.8 | +2.2 |
| Replace Retriever only: M0/R3 | +6.1 | +4.4 |
| Sum of these two gains | +10.9 | +6.6 |
| Replace both: M3/R3 | +13.4 | +11.9 |
| Additional gain beyond the sum | +2.5 | +5.3 |
The 2.5/5.3-point excess supports an additional benefit from memory–retriever alignment. We have added this cross-checkpoint evaluation and analysis to the paper.
Response to Reviewer 1ETB (3/4)
Q2: Matching construction and online frame budgets
We address frame-budget fairness through a controlled comparison with WorldMM-8B [1], matching both memory-construction access and online QA inputs.
1. Online frame limits do not by themselves control memory construction.
(1.a) Direct-video QA and memory-based QA follow different paradigms. The former selects frames to answer each question; VideoEvolve first constructs reusable memory, then retrieves evidence for QA. A 64/128-frame online limit therefore cannot be equated with its construction budget. Memory-based VideoLucy [2] also specifies sampling rates by stage: 1 FPS for coarse memory and 2 FPS for fine and ultra-fine backtracking on Video-MME medium/long videos and LVBench. VideoEvolve constructs Base memory at 0.5 FPS.
(1.b) MemVid also accesses frames before answer generation. MemVid [3] scans the video to form global memory and encodes candidate segments for retrieval; its 128-frame cap applies to the final generator input, not the combined access of all three stages.
2. Compare memory-based methods with 8B retrieval agents under matched frame budgets. We select WorldMM-8B [1] because both methods construct video memory for retrieval and use 8B retrieval agents. With the paradigm and retrieval-model scale matched, we control frame access at both stages:
(2.a) Construction: both methods use the same pool of up to 768 distinct, uniformly sampled frames per video, at matched resolution. All memories, captions, features, and indexes were rebuilt solely from this pool, including VideoEvolve's Base and Delta.
(2.b) Online QA: both methods have at most 64 image inputs per question across all turns, including initial keyframes and visual-tool observations; repeated inputs count again. All images come from the same 768-frame pool, with no access to outside frames.
Both methods are evaluated on the same 900 Video-MME Long questions and 1,549 LVBench questions, without subtitles or audio transcripts:
| Method | Online image inputs / question | Video-MME Long (w/o sub) | LVBench |
|---|---|---|---|
| WorldMM-8B | 64 | 53.7 | 43.5 |
| VideoEvolve-8B | 64 | 60.8 | 57.9 |
With frame budgets matched and both retrievers at 8B, VideoEvolve-8B leads by 7.1 points on Video-MME Long and 14.4 on LVBench. We have added this comparison and its frame-budget definitions to the paper.
References
[1] WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning. CVPR 2026.
[2] VideoLucy: Deep Memory Backtracking for Long Video Understanding. NeurIPS 2025.
[3] Memory-enhanced Retrieval Augmentation for Long Video Understanding. arXiv:2503.09149, 2025.
Response to Reviewer 1ETB (1/4)
Dear Reviewer 1ETB,
Thank you for your careful review and for recognizing the value of connecting memory optimization with downstream feedback and aligning memory with retrieval, as well as the rationale and effectiveness of BEF/CEF in adapting training allocation across stages. We address your comments and questions point by point below.
VideoEvolve co-evolves what evidence to acquire and retain with how to retrieve and use it; EvolveR and Evolving-RL focus on extracting and reusing task experience. Our contribution is this coupled learning framework and its adaptive BEF/CEF guidance, implemented through trajectory-level RL and alternating updates.
1. Different learning targets: experience reuse vs. co-evolution of evidence construction and use.
Following their published mechanisms, a natural adaptation would be:
(1.a) EvolveR: collect video-QA trajectories, distill reusable strategies from successes and failures, retrieve those strategies for subsequent questions, and improve the agent policy through RL.
(1.b) Evolving-RL: extract candidate experience from completed video-QA interactions, evaluate its usefulness on related video questions, and jointly train experience extraction and utilization.
(1.c) Shared focus: experience reuse. Both adaptations improve future QA by reusing problem-solving experience, such as checking nearby segments after locating an event anchor.
(1.d) VideoEvolve: co-evolving evidence construction and use. In contrast to these approaches, we address a fundamental long-video challenge: how can memory built before the questions are known support later reasoning under observation and storage limits? We jointly optimize evidence acquisition, retention, and retrieval, co-evolving what to remember and how to use it.
2. Why this requires co-evolution and adaptive guidance.
(2.a) Missing evidence and ineffective evidence use are different bottlenecks. A reusable memory must serve multiple questions, yet relevant details can be omitted or stored without being found and used. The system must improve both evidence availability and its use.
The qualitative cases in Appendix A.11 illustrate these bottlenecks. In Table 16 (LongVideoBench-long), a fixed Retriever initially abstains when asked where else a lakeside man appears. Adding a lake-scene description and frame lets it link him to an existing scene and answer “in front of a yellow hill.”
Table 19 (Video-MME-v2) shows the answer before and after a co-evolution update. The earlier run retrieves both product prices but answers only 27,900 KRW. After a co-evolution update, it verifies the product–price associations and computes 22,900 + 27,900 = 50,800 KRW. Both memory and Retrieval change here, so this illustrates improved evidence use during co-evolution, rather than isolating one policy's effect.
(2.b) Downstream usefulness connects construction and retrieval. Memory construction receives no downstream question, but its policy is rewarded by how well the current Retriever answers the video's questions. The Retriever then learns over the updated memory. This makes what is retained depend on its downstream usefulness, while retrieval adapts to the evidence actually retained.
(2.c) Diagnosis directs subsequent training. Alongside QA rewards for policy updates, BEF uses paired memory-only/revisit-enabled evaluations to estimate which side needs more training. CEF prioritizes capabilities and samples within each side while preserving coverage. Feedback therefore also determines which side and capabilities to strengthen next.
3. Experimental support for each design choice.
(3.a) Learning which evidence to acquire improves on uniform observation. Table 9(a) uses the same Writer, fixed Retrieval checkpoint, and augmentation allowance:
(3.b) Dual-policy learning and adaptive guidance contribute separately. Table 3 compares one-sided evolution, dual-policy evolution without BEF/CEF, and the full framework:
BEF/CEF are disabled in the first three rows. Training both sides improves LVBench over either side alone, but MMVU shows that updating both policies is insufficient on its own. Adding BEF/CEF yields a further 2.3/6.7 points, respectively, and the best results in this comparison. These controls quantify the benefits of learned evidence selection, dual-policy training, and adaptive evolution guidance.