acceptodds
Under review as a conference paper at ICLR 2027

VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding

Abstract

Long video understanding increasingly relies on external memory to organize massive visual streams into compact representations. However, most memory-based methods dynamically adapt how information is retrieved for different questions, while largely fixing what is remembered. This mismatch makes missing details costly to recover, whereas stored information is valuable only when it can be reliably retrieved. To address this issue, we propose VideoEvolve, a novel self-evolving framework that jointly evolves memory and retrieval for long video understanding. Specifically, starting from a coarse low-frame-rate overview, VideoEvolve couples a Memory Evolver for selective memory augmentation with a Retrieval Evolver for adaptive retrieval over the evolving memory. We then co-evolve the two Evolvers through alternating agentic reinforcement learning (Agentic RL), updating one while freezing the other. To steer this alternating evolution, Bottleneck-Aware Evolution Feedback (BEF) identifies whether the current bottleneck lies in memory or retrieval and directs optimization toward the more limiting side. Furthermore, VideoEvolve introduces Capability-Aware Evolution Feedback (CEF) to alleviate downstream feedback from over-specializing memory to a fixed set of training questions, shifting training toward underdeveloped yet learnable video capabilities. By integrating Agentic RL with BEF and CEF, VideoEvolve transforms downstream reasoning experience into transferable capability updates, providing a concrete path from static long-video systems toward experience-driven, self-improving multimodal intelligence. Extensive experiments on multiple long video understanding benchmarks demonstrate the effectiveness of VideoEvolve.

Primary area:
reinforcement learning
Keywords:
Long Video Understanding, Self-Evolving Agents, Agentic Reinforcement Learning, Multimodal Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1ETBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a self-evolving framework to jointly train a memory construction agent and a retrieval agent for long video understanding. This approach aims to address two key limitations in existing memory-based long video understanding methods: the limited adaptability of memory to diverse videos, and the misalignment between memory and retrieval. Specifically, the proposed method first constructs a base memory and subsequently generates diverse memory candidates based on it. The retrieval agent then answers downstream questions using these memory candidates. The accuracy of the answers to these downstream questions is calculated to formulate a reward score for optimizing the memory agent. Subsequently, the memory is reconstructed, and the retrieval agent is further optimized by calculating its reward score based on the updated memory using the same methodology. Furthermore, Bottleneck-Aware Evolution Feedback (BEF) and Capability-Aware Evolution Feedback (CEF) modules are designed to dynamically allocate the optimization budget. Comparative experiments with existing methods demonstrate the superior effectiveness of the proposed approach over previous memory-based methods.

Critical strengths
  1. This method addresses the isolation between memory optimization and downstream task feedback in memory-based long video understanding approaches, as well as the lack of consideration for the alignment relationship between retrieval and memory. Such a joint optimization strategy is highly beneficial for long video understanding.
  2. Furthermore, by designing bottleneck feedback and capability feedback mechanisms, this method effectively addresses the issue that the required optimization intensity for different agents varies dynamically across different stages of the alternating optimization process. This design is both reasonable and effective.
Critical weaknesses
  1. The trajectory-level reinforcement learning and alternating training procedure adopted by VideoEvolve do not appear to differ fundamentally from those in EvolveR and Evoving-RL. Although both studies are cited in the related work section, the paper does not clearly articulate its core methodological novelty relative to these approaches.

  2. Although the authors describe the method as “co-evolution,” the mutual feedback between its two components appears to be confined to training. During inference, the agents seem to operate independently. It would help to clarify how “co-evolution” benefits inference. Moreover, the method relies heavily on cold-start initialization: the supplementary material reports a performance drop of approximately 20 points without it. This finding highlights the substantial contribution of cold-start initialization, but does not, by itself, establish the benefits of co-evolutionary training.

Questions for authors
  1. The current experiments do not establish whether the observed gains arise from genuine alignment between the evolved memory and retriever, or simply because both components become independently stronger after optimization. The authors should conduct cross-checkpoint evaluations using matched and mismatched memory–retriever pairs. For example, they could compare (MN,RK)(M_N, R_K), (MK,RM)(M_K, R_M), and the matched pair (MK,RK)(M_K, R_K). In contrast, the current ablations that optimize only the memory or only the retriever merely demonstrate that improving either component is beneficial; they do not substantiate the central claim that co-evolution aligns the two components.

  2. Many methods in Table 1 operate under a fixed video frame budget, with some sampling at most 64 or 128 frames. Similarly, among the memory-based methods in Table 2, World-MM[1] samples up to 768 frames, while MemVid[2] samples 128 frames. In contrast, VideoEvolve constructs its base memory by sampling at 0.5 FPS and needs accessing additional video frames. Therefore, the comparisons may not be entirely fair due to differences in frame budgets. Could the authors provide comparisons with representative methods under the same frame budget to ensure a fair evaluation?

  3. In Table 2, VideoEvolve under the training-free setting significantly outperforms its trained counterpart. However, the authors do not analyze the reasons behind such a substantial performance gap. Moreover, the supplementary material indicates that the training-free setting relies solely on base memory, without employing memory augmentation or constructing delta memory. This further raises questions about the necessity and effectiveness of training the Memory Controller.

  4. Some issues regarding the presentation and writing:

    a) The term CkC^k appears in Section3.4 of the main text without any prior definition or mention; it is only briefly addressed in the supplementary material. Given the importance of the CkC^k concept, it should be clearly explained and defined within the main text.

    b) PinvalidP_{\text{invalid}} in Equation(5) lacks a corresponding formula or explicit definition.

    c) In Section3.5, a large number of mathematical notations without clearly explaining their meanings. As a result, it is difficult to understand the specific mechanisms of BEF and CEF, as well as how the optimization budget is dynamically allocated among different Evolvers.

[1] Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, and Sung Ju Hwang. Worldmm: Dynamic multi-modal memory agent for long video reasoning. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25599–25609, 2026.

[2] Huaying Yuan, Zheng Liu, Minghao Qin, Hongjin Qian, Yan Shu, Zhicheng Dou, Ji-Rong Wen, and Nicu Sebe. Memory-enhanced retrieval augmentation for long video understanding. arXiv:2503.09149, 2025.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1ETB (1/4)

Dear Reviewer 1ETB,

Thank you for your careful review and for recognizing the value of connecting memory optimization with downstream feedback and aligning memory with retrieval, as well as the rationale and effectiveness of BEF/CEF in adapting training allocation across stages. We address your comments and questions point by point below.

W1: Methodological distinctions from EvolveR and Evolving-RL

VideoEvolve co-evolves what evidence to acquire and retain with how to retrieve and use it; EvolveR and Evolving-RL focus on extracting and reusing task experience. Our contribution is this coupled learning framework and its adaptive BEF/CEF guidance, implemented through trajectory-level RL and alternating updates.

1. Different learning targets: experience reuse vs. co-evolution of evidence construction and use.

Following their published mechanisms, a natural adaptation would be:

(1.a) EvolveR: collect video-QA trajectories, distill reusable strategies from successes and failures, retrieve those strategies for subsequent questions, and improve the agent policy through RL.

(1.b) Evolving-RL: extract candidate experience from completed video-QA interactions, evaluate its usefulness on related video questions, and jointly train experience extraction and utilization.

(1.c) Shared focus: experience reuse. Both adaptations improve future QA by reusing problem-solving experience, such as checking nearby segments after locating an event anchor.

(1.d) VideoEvolve: co-evolving evidence construction and use. In contrast to these approaches, we address a fundamental long-video challenge: how can memory built before the questions are known support later reasoning under observation and storage limits? We jointly optimize evidence acquisition, retention, and retrieval, co-evolving what to remember and how to use it.

2. Why this requires co-evolution and adaptive guidance.

(2.a) Missing evidence and ineffective evidence use are different bottlenecks. A reusable memory must serve multiple questions, yet relevant details can be omitted or stored without being found and used. The system must improve both evidence availability and its use.

The qualitative cases in Appendix A.11 illustrate these bottlenecks. In Table 16 (LongVideoBench-long), a fixed Retriever initially abstains when asked where else a lakeside man appears. Adding a lake-scene description and frame lets it link him to an existing scene and answer “in front of a yellow hill.”

Table 19 (Video-MME-v2) shows the answer before and after a co-evolution update. The earlier run retrieves both product prices but answers only 27,900 KRW. After a co-evolution update, it verifies the product–price associations and computes 22,900 + 27,900 = 50,800 KRW. Both memory and Retrieval change here, so this illustrates improved evidence use during co-evolution, rather than isolating one policy's effect.

(2.b) Downstream usefulness connects construction and retrieval. Memory construction receives no downstream question, but its policy is rewarded by how well the current Retriever answers the video's questions. The Retriever then learns over the updated memory. This makes what is retained depend on its downstream usefulness, while retrieval adapts to the evidence actually retained.

(2.c) Diagnosis directs subsequent training. Alongside QA rewards for policy updates, BEF uses paired memory-only/revisit-enabled evaluations to estimate which side needs more training. CEF prioritizes capabilities and samples within each side while preserving coverage. Feedback therefore also determines which side and capabilities to strengthen next.

3. Experimental support for each design choice.

(3.a) Learning which evidence to acquire improves on uniform observation. Table 9(a) uses the same Writer, fixed Retrieval checkpoint, and augmentation allowance:

Construction policyVideo-MME Long w/subLongVideoBench Long
Uniform augmentation62.860.1
Learned augmentation67.165.5
Gain+4.3+5.4

(3.b) Dual-policy learning and adaptive guidance contribute separately. Table 3 compares one-sided evolution, dual-policy evolution without BEF/CEF, and the full framework:

Training configurationLVBenchMMVU
Memory evolution only51.169.2
Retrieval evolution only52.770.1
Both, w/o BEF/CEF56.668.4
Full co-evolution58.975.1

BEF/CEF are disabled in the first three rows. Training both sides improves LVBench over either side alone, but MMVU shows that updating both policies is insufficient on its own. Adding BEF/CEF yields a further 2.3/6.7 points, respectively, and the best results in this comparison. These controls quantify the benefits of learned evidence selection, dual-policy training, and adaptive evolution guidance.

AuthorsOpenReview ↗

Response to Reviewer 1ETB (2/4)

W2: Inference benefits and gains beyond SFT

1. Inference benefits from memory–retriever alignment learned through co-evolution. During training, QA feedback from Retrieval guides the Controller's evidence preservation, while Retrieval adapts to the updated memory. This mutual adaptation is retained in the learned policies: at inference, the Controller builds reusable memory once per video, and Retrieval searches it to answer questions, without online parameter updates. BEF/CEF operate only during training.

2. Co-evolution goes beyond the performance reached by SFT alone. Removing both SFT stages while retaining RL lowers LVBench from 58.9 to 32.8 (Table 11 in the paper), showing the value of initialization. Two comparisons clarify the additional gains:

(2.a) Roughly doubling SFT data yields limited gains. Stage-II SFT increases from 50,871 to 102,732 complete trajectories (+51,861), and from 174,652 to 349,717 decision samples (+175,065). Both runs start from the same Stage-I checkpoint and train for one epoch. Under matched evaluation settings, accuracy increases by only 0.1/1.1 points on Video-MME w/sub/LVBench, indicating diminishing SFT gains in this tested range.

(2.b) Co-evolution further improves the full-data SFT model. Co-evolution starts from the full-data SFT model, with backbones, frozen Writer, and evaluation settings unchanged. The half-data SFT row is the added control; the full-data SFT and co-evolution results are from Table 3 in the paper:

Training configurationVideo-MME w/subLVBench
After SFT (half Stage-II data)66.749.2
After SFT (full Stage-II data)66.850.3
Full co-evolution73.958.9
Gain: full vs. half SFT+0.1+1.1
Gain: co-evolution vs. full SFT+7.1+8.6

All settings use Base-plus-Delta memory and permit revisiting; Video-MME uses the full set with subtitles. Co-evolution adds 7.1/8.6 points even after full-data SFT. We have added this data-scale comparison to the paper.

Q1: Alignment versus independently stronger components

The cross-checkpoint results support memory–retriever alignment: a later memory can reduce accuracy when mismatched, while the evolved pair delivers gains beyond the sum of separate improvements.

1. The cycle-3 matched pair performs best across all nine combinations. M0/R0 is the post-SFT, pre-RL pair; M1/R1 and M3/R3 are from cycles 1 and 3. We cross-pair them without retraining under memory-only inference, with stored keyframes but no new raw-video frames.

Each cell reports Video-MME Long (w/sub) / LVBench accuracy. The LVBench diagonal and the M3/R3 Video-MME Long result also appear in Tables 3–4 in the paper.

Memory / Retriever   R0      R1      R3   
M0  52.1 / 45.6    56.9 / 46.1    58.2 / 50.0  
M1  54.3 / 46.5    60.4 / 48.5    62.6 / 52.8  
M3  56.9 / 47.8    59.1 / 47.7    65.5 / 57.5  

2. The same memory update has opposite effects with different Retrievers. With R1 fixed, replacing its matched M1 with the later M3 lowers accuracy by 1.3/0.8 points. With R3 fixed, the same M1-to-M3 replacement instead raises accuracy by 2.9/4.7 points. Updating memory alone does not ensure an overall gain; the Retriever must also be able to use the updated evidence.

3. Replacing both components yields gains beyond the sum of their separate gains. To quantify the additional benefit, we compare gains over M0/R0 using the same cross-pairings (percentage points):

Change from M0/R0Video-MME Long (w/sub)LVBench
Replace memory only: M3/R0+4.8+2.2
Replace Retriever only: M0/R3+6.1+4.4
Sum of these two gains+10.9+6.6
Replace both: M3/R3+13.4+11.9
Additional gain beyond the sum+2.5+5.3

The 2.5/5.3-point excess supports an additional benefit from memory–retriever alignment. We have added this cross-checkpoint evaluation and analysis to the paper.

AuthorsOpenReview ↗

Response to Reviewer 1ETB (3/4)

Q2: Matching construction and online frame budgets

We address frame-budget fairness through a controlled comparison with WorldMM-8B [1], matching both memory-construction access and online QA inputs.

1. Online frame limits do not by themselves control memory construction.

(1.a) Direct-video QA and memory-based QA follow different paradigms. The former selects frames to answer each question; VideoEvolve first constructs reusable memory, then retrieves evidence for QA. A 64/128-frame online limit therefore cannot be equated with its construction budget. Memory-based VideoLucy [2] also specifies sampling rates by stage: 1 FPS for coarse memory and 2 FPS for fine and ultra-fine backtracking on Video-MME medium/long videos and LVBench. VideoEvolve constructs Base memory at 0.5 FPS.

(1.b) MemVid also accesses frames before answer generation. MemVid [3] scans the video to form global memory and encodes candidate segments for retrieval; its 128-frame cap applies to the final generator input, not the combined access of all three stages.

2. Compare memory-based methods with 8B retrieval agents under matched frame budgets. We select WorldMM-8B [1] because both methods construct video memory for retrieval and use 8B retrieval agents. With the paradigm and retrieval-model scale matched, we control frame access at both stages:

(2.a) Construction: both methods use the same pool of up to 768 distinct, uniformly sampled frames per video, at matched resolution. All memories, captions, features, and indexes were rebuilt solely from this pool, including VideoEvolve's Base and Delta.

(2.b) Online QA: both methods have at most 64 image inputs per question across all turns, including initial keyframes and visual-tool observations; repeated inputs count again. All images come from the same 768-frame pool, with no access to outside frames.

Both methods are evaluated on the same 900 Video-MME Long questions and 1,549 LVBench questions, without subtitles or audio transcripts:

MethodOnline image inputs / questionVideo-MME Long (w/o sub)LVBench
WorldMM-8B6453.743.5
VideoEvolve-8B6460.857.9

With frame budgets matched and both retrievers at 8B, VideoEvolve-8B leads by 7.1 points on Video-MME Long and 14.4 on LVBench. We have added this comparison and its frame-budget definitions to the paper.

References

[1] WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning. CVPR 2026.

[2] VideoLucy: Deep Memory Backtracking for Long Video Understanding. NeurIPS 2025.

[3] Memory-enhanced Retrieval Augmentation for Long Video Understanding. arXiv:2503.09149, 2025.

Reviewer_RyVzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores co-evolving memory and retrieval in the task of long video understanding. To address the issue of limited memory adaptability and memory-retrieval misalignment in existing methods, this paper proposes a self-evolving framework named VideoEvolve that jointly evolves memory and retrieval. Specifically, the authors design Memory Evolver and Retrieval Evolver to realize selective memory augmentation and adaptive retrieval over evolving memory, respectively. These two evolvers co-evolve through alternating agentic RL. Besides, the authors also introduce bottleneck-aware evolution feedback to identify optimization bottlenecks and capability-aware evolution feedback to alleviate overspecialized memory caused by downstream feedback. The evaluation results of multiple long video understanding benchmarks show that the proposed VideoEvolve achieves state-of-the-art performance.

Critical strengths
  1. The exploitation of the co-evolution of memory and retrieval in long video understanding is technically sound and interesting.

  2. The proposed VideoEvolve achieves state-of-the-art performance on multiple long video understanding benchmarks.

  3. The authors provide the core source code of the proposed VideoEvolve.

Critical weaknesses
  1. The authors did not report the results on the Charades-STA dataset, which is also used as a long-video understanding evaluation in previous work, such as ParaVT.

  2. In Table 2, the authors use Qwen3.8-27B for memory construction, which is larger than previous work that uses a 7B model. Does the large model lead to performance improvement?

  3. The design of capability-aware ability is mainly from the Video-MME-v2 benchmark. Such a design is somewhat incremental.

  4. The training infrastructure details are not provided.

  5. There is no qualitative analysis of training dynamics during the co-evolution of the memory evolver and retrieval evolver.

Questions for authors
  1. The authors did not report the results on the Charades-STA dataset, which is also used as a long-video understanding evaluation in previous work, such as ParaVT. Could the authors provide an explanation for this?

  2. In Table 2, the authors use Qwen3.8-27B for memory construction, which is larger than previous work that uses a 7B model. Does the large model lead to performance improvement?

  3. Could the authors provide the training infrastructure details and training cost?

  4. Could the authors provide a qualitative analysis of training dynamics during the co-evolution? This would be better for understanding the co-evolution.

4 comments
AuthorsOpenReview ↗

Response to Reviewer RyVz (1/2)

Dear Reviewer RyVz,

Thank you for your careful review and for recognizing the technical value of memory–retrieval co-evolution, our strong performance across long-video benchmarks, and the release of our core source code. We address your comments and questions point by point below.

W1 & Q1: Evaluation on Charades-STA

1. Evaluation protocol. Thank you for suggesting this complementary temporal-grounding evaluation. We evaluate VideoEvolve on the official Charades-STA test set and report mIoU. The SFT and Agentic RL checkpoints use the same evidence-access settings and tool budgets, without additional Charades-STA-specific fine-tuning. The ParaVT result is quoted from Table 1 of its paper.

2. Results.

Method / checkpointmIoU
ParaVT-8B50.1
VideoEvolve, after SFT52.3
VideoEvolve, after Agentic RL52.6

VideoEvolve achieves 52.6 mIoU, 2.5 points above ParaVT's paper-reported result.

3. Interpretation. (a) Training exposure. ParaVT includes Charades-STA data in both SFT and RL, while we perform no additional fine-tuning on this dataset's training split. (b) Transferability. The result shows that VideoEvolve's memory and retrieval capabilities transfer from our primary long-video setting to temporal grounding. (c) Agentic RL gain. The additional gain is modest (0.3 points), possibly because our RL objective targets video question answering without dedicated temporal-boundary supervision. We have included the protocol and results in the appendix.

W2 & Q2: Writer size and co-evolution gains

1. Controlled setting. Table 2 uses the 27B Writer. To measure this dependence, we replace the frozen Qwen3.8-27B Writer with Qwen2.5-VL-7B-Instruct, while keeping the Controller/Retrieval backbones, training budget, and evaluation settings unchanged. Both Writers remain frozen; Video-MME uses subtitles.

2. Results.

CheckpointVideo-MME, 27B WriterVideo-MME, 7B WriterLVBench, 27B WriterLVBench, 7B Writer
After SFT66.863.350.347.6
After Agentic RL73.971.458.956.4
Gain (points)+7.1+8.1+8.6+8.8

3. Interpretation. (a) Absolute performance. The 27B Writer is 2.5 points better after RL on both benchmarks, showing that Writer strength affects final accuracy. (b) Co-evolution gains. The 7B Writer still gains 8.1/8.8 points on Video-MME/LVBench, versus 7.1/8.6 with the 27B Writer. Thus, the larger Writer improves absolute performance, but the co-evolution gains persist with the smaller Writer. We have added this control to Appendix A.9.

W3: Capability labels and the contribution of CEF

1. CEF contribution and evidence. Video-MME-v2 provides the capability labels; we do not claim the taxonomy as our contribution. Our contribution is using diagnostic feedback to organize the next training cycle. BEF first determines whether Memory or Retrieval needs more training. CEF then (a) increases sampling of the weaker capabilities separately for each policy and (b) selects informative samples whose rollouts show different outcomes, while retaining exploration and previously learned cases. Full CEF improves over uniform, class-balanced, and hard-example sampling by 1.0/1.9, 0.7/1.7, and 0.5/1.1 points on LVBench/MMVU. Removing the second mechanism reduces performance by 1.6/2.2 points.

2. Ablation results. All sampling strategies use the same initialization, model configuration, training budget, and BEF settings.

Sampling strategyLVBenchMMVU
Uniform instances (w/o CEF)57.973.2
Class-balanced58.273.4
Hard-example58.474.0
Capability weights, no frontier diagnosis57.372.9
Full CEF58.975.1

Full CEF performs best on both LVBench and MMVU, which are outside the Video-MME-v2 training/diagnostic pool. This shows that combining capability adaptation with frontier-aware selection is more effective than uniform, class-balanced, or generic hard-example sampling.

AuthorsOpenReview ↗

Response to Reviewer RyVz (2/2)

W4 & Q3: Training infrastructure and cost

1. SFT infrastructure. We use 8 NVIDIA H20 GPUs with LLaMA-Factory and DeepSpeed ZeRO-3. Both stages train for one epoch in BF16, with effective batch sizes of 24 and 32 for Stages I and II. The visual encoder is frozen; the language model and multimodal mergers are updated.

2. RL infrastructure. We use 8 NVIDIA H800 GPUs with verl, FSDP, and vLLM. Memory Evolver training uses 6 GPUs for the Controller and 2 for the frozen Retrieval scorer; Retrieval Evolver training uses all 8 GPUs. The Writer and embedding model remain frozen and run on remote services. Full settings are in Appendix A.3–A.6 and Tables 7–8.

3. Measured training cost. SFT costs cover each complete stage; RL costs are per-round averages.

Training stageGPU configurationTime (h)GPU-hours
SFT Stage I8×H2010.2782.18
SFT Stage II, full-data8×H2054.94439.52
Memory Evolver RL (per round)8×H80019.75158.00
Retrieval Evolver RL (per round)8×H80013.41107.28

W5 & Q4: Qualitative dynamics of co-evolution

We have added qualitative examples of co-evolution to Appendix A.11 of the revised paper. Key trajectory excerpts (Tables 12–19 in the revised paper) illustrate how the Memory Evolver learns what to remember and the Retrieval Evolver learns to find and use that evidence.

1. Retrieval Evolver: improve answers with unchanged memory.

(1.a) Correctly interpret stored events (Table 12).

The question asks which paper animal is made second. Both runs receive the same first tool response: cat → pig → dog. Before the update, the retriever overlooks the pig and answers “dog”; after the update, it correctly interprets the sequence and answers “pig.”

(1.b) Find relevant frames in memory (Table 13).

Asked about a formula’s highlight and annotation colors, the retriever initially finds an unrelated formula at 984 s and answers “red/red.” Adding the page title to the subtitle query locates the target page at 968 s, yielding the correct “yellow/red.” The memory is unchanged; the search finds the relevant image.

2. Retrieval Evolver: use memory with fewer tool calls.

(2.a) Stop when memory evidence suffices (Table 14).

Asked how the vlogger reaches Shibuya Crossing, both versions answer “train.” Before the update, the retriever repeatedly reads a summary. Afterward, it finds gate, platform and boarding records and answers immediately: tool calls fall 12 → 2, repeats 10 → 0.

(2.b) Query memory directly (Table 15).

The question asks the date of a solar eclipse. Repeated summary reads are replaced by a direct search of OCR text stating “April 8, 2024.” The answer stays correct while tool calls fall 10 → 1: the retriever chooses the tool suited to a text-based fact.

3. Memory Evolver: add records useful for QA.

(3.a) Add clues linking people and scenes (Table 16).

The question asks where else a man standing beside a lake appears. The same frozen 27B retriever initially abstains. A new lake-scene description and frame let it match the man to an existing scene and correctly answer “in front of a yellow hill.” Added memory supplies the missing connection.

(3.b) Record intermediate steps for reasoning (Table 17).

A geometry question asks for the difference between a derived value and a given value. New records cover auxiliary-line construction at 112.5–125 s. The retriever reads these alongside existing values, identifies AD = EF = 1, and changes “cannot calculate” to “0.” New observations and existing facts are used together.

4. Memory–retrieval cooperation: use facts to answer.

(4.a) Memory records attributes; retrieval selects what is needed (Table 18).

The question asks what word is printed on a cup. Its Entity (object) record already contains both appearance attributes and the text “FROZEN,” but the retriever initially answers “Blue.” After the update, one targeted search extracts the correct word. Memory provides searchable attributes; retrieval selects the one required by the question.

(4.b) Memory stores facts; retrieval combines them (Table 19).

The question asks for two products’ total price. Memory records the products and their prices in separate segments. The retriever initially returns only 27,900 won, despite seeing both prices. After the update, it checks the product–price associations and computes 22,900 + 27,900 = 50,800 won. Memory preserves the individual facts; retrieval connects them to perform the required calculation.

Thank the authors for their responses, which have addressed my major concerns. Therefore, I keep my rating.

Dear Reviewer RyVz,

Thank you for your follow-up and for confirming that our response has addressed your major concerns. We appreciate your careful evaluation and constructive feedback. We have incorporated the clarified experimental settings and additional results into the revised manuscript to further improve its clarity and completeness.

Reviewer_KK8cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VideoEvolve for long-video question answering. It jointly improves what a system stores in video memory and how it retrieves evidence from that memory. A Memory Evolver selectively augments an initial video summary, while a Retrieval Evolver searches the resulting memory and can revisit the source video. The two are trained through alternating reinforcement-learning updates. Bottleneck-Aware Evolution Feedback (BEF) allocates training between them, and Capability-Aware Evolution Feedback (CEF) prioritizes training examples by capability. Results on five long-video benchmarks and component ablations support the approach.

Critical strengths
  1. Paper is well written and easy to follow.
  2. The method is well motivated. Retrieval cannot recover information omitted from memory without revisiting the video, while the value of stored information depends on whether it can be found. Alternately training memory construction and retrieval directly addresses this dependency.
  3. The ablations make a reasonable case that the gain is not solely from training one stronger component.
Critical weaknesses
  1. Table 1 raises concerns about the fairness of the comparison. In Table 1, baselines are single-model methods (such as VideoR1 and VideoRFT) and single-agent methods (such as VideoZoomer and ParaVT) whereas VideoEvolve is a multi-agent system that includes a 27B-parameter agent component. This gives VideoEvolve substantially greater model capacity than the baselines.
  2. Table 2 omits relevant, strong memory-based baselines, such as VideoARM [1] and VideoLucy [2].
  3. As a multi-agent system, VideoEvolve is likely to incur higher inference latency and token consumption than single-model methods. The paper lacks an analysis of these inference costs.
  4. The training pipeline involves multiple stages and appears rather complex.

[1] Videoarm: Agentic reasoning over hierarchical memory for long-form video understanding. CVPR 2026.

[2] VideoLucy: Deep Memory Backtracking for Long Video Understanding. NeurIPS 2025.

Questions for authors
  1. Please see weakness.
  2. The proposed BEF and CEF components appear to be ineffective in the training-free variant.
  3. Can authors release more training details and training cost?
  4. Have the authors tried jointly optimizing the Memory Evolver and Retrieval Evolver, rather than optimizing them alternately?
5 comments
AuthorsOpenReview ↗

Response to Reviewer KK8c (1/3)

Dear Reviewer KK8c,

Thank you for your careful review and for recognizing our clear presentation, the motivation for co-evolving memory and retrieval, and the supporting ablation studies. We address your comments and questions point by point below.

W1: Model capacity and comparison fairness

The co-evolution gains persist with a shared 8B backbone and with a smaller 7B Writer. Below, we report these two additional controls, together with the fixed-component ablations in Table 3 in the paper.

1. Shared 8B backbone with role-specific LoRA adapters. The original system uses separate 4B Memory Controller, 27B Memory Writer, and 8B Retrieval Agent backbones. In this control, all three roles share one Qwen3-VL-8B-Instruct backbone, while each role uses a rank-64 LoRA adapter. The three adapters contain 566.1M parameters in total.

ComparisonVideo-MME w/subLVBench
Reported 7B/8B SOTA (Table 1 in the paper)69.4 (ParaVT-8B)41.5 (VideoZoomer-7B)
Shared 8B + LoRA, after SFT60.638.7
Shared 8B + LoRA, after Agentic RL70.154.3
Agentic RL (Co-evolution) Gain+9.5+15.6
Margin over the SOTA+0.7+12.8

With a shared 8B backbone, co-evolution still gains 9.5/15.6 points on Video-MME/LVBench and outperforms the listed baselines, without requiring the original three-backbone configuration.

2. Co-evolution gains with a 7B Writer. To examine reliance on the original Writer, we replace the frozen 27B Writer with Qwen2.5-VL-7B-Instruct. The Controller and Retrieval Agent backbones, training budget, and evaluation settings remain the same.

ComparisonVideo-MME w/subVideo-MME w/subLVBenchLVBench
Writer27B Writer7B Writer27B Writer7B Writer
After SFT66.863.350.347.6
After Agentic RL73.971.458.956.4
Agentic RL (Co-evolution) Gain+7.1+8.1+8.6+8.8

The 7B Writer retains 8.1/8.8-point co-evolution gains despite a 2.5-point drop in final accuracy on both benchmarks: the 27B Writer improves absolute performance but is not a prerequisite for these gains.

3. Co-evolution ablations at fixed model capacity. With identical backbones, full co-evolution outperforms the stronger one-sided variant by 6.2 points on LVBench; the gain remains 3.9 points without BEF/CEF (Table 3 in the paper). This supports evolving memory and retrieval together rather than adding model capacity.

4. Summary and manuscript revisions. Taken together, these controls show that the observed co-evolution gains are not solely attributable to the original system's larger model capacity. Table 1 in the paper is a system-level comparison rather than a parameter-matched ablation. We have added the new controls to Appendix A.9 and clarified each role's backbone configuration and parameter sharing in Section 4.1 and Table 1 in the paper.

W2: Comparisons with VideoARM and VideoLucy

1. Additional baselines and evaluation setting.

For a matched comparison, all Video-MME Long results below use the w/o-subtitles setting reported by VideoARM and VideoLucy.

MethodConfigurationLVBenchVideo-MME Long (w/o sub)
VideoLucy (training-free)Qwen2.5-VL-7B + DeepSeek-R1 (671B total/37B active)58.866.8
VideoARM (training-free)OpenAI o3 + GPT-4.179.775.3
VideoEvolve (training-free)Qwen3.8-27B + Qwen3.8-27B76.776.9
VideoEvolve (training-based)4B Controller + 8B Retrieval; 27B Writer frozen58.966.5

2. Training-free results. Without updating any parameters, VideoEvolve with Qwen3.8-27B ranks first on Video-MME Long (w/o sub) and second on LVBench. We report this compatibility test to show that frontier models can be directly integrated; our focus remains the training-based co-evolution of what to remember and how to retrieve.

3. Training-based results. With a 4B Controller, an 8B Retrieval Evolver, and a frozen 27B Writer, VideoEvolve scores 58.9/66.5 on LVBench/Video-MME Long (w/o sub), versus VideoLucy's 58.8/66.8 with DeepSeek-R1 (671B total/37B active). VideoEvolve thus achieves near-matching accuracy with far fewer total backbone parameters: learned co-evolution delivers strong results without invoking a massive reasoning model.

AuthorsOpenReview ↗

Response to Reviewer KK8c (2/3)

W3: Inference latency and token consumption

1. Faster online QA with higher accuracy. VideoEvolve answers in 5.21 s/question, reducing online latency by 56.7%/66.2% versus ParaVT-8B/LongVT-RFT-7B.

We evaluate Video-MME Long (300 videos, 900 questions) on 2 NVIDIA H20 GPUs, TP=2, with one concurrent question. VideoEvolve uses the full trained model; both baselines start with 64 frames.

MethodMean online QA (s/question)Accuracy (%)Mean cumulative LLM tokens/question
ParaVT-8B12.0462.9102.44K
LongVT-RFT-7B15.4048.915.61K
VideoEvolve (full)5.2167.145.67K

Accuracy is also higher by 4.2/18.2 points versus ParaVT/LongVT. VideoEvolve uses 55.4% fewer tokens than ParaVT, but 2.9× LongVT's tokens. Thus, its advantage is the lowest online latency among these methods, not the lowest token count.

2. Question-independent memory enables efficient reuse. VideoEvolve constructs question-independent memory once per video, allowing different questions to reuse the same stored evidence without rebuilding memory. This amortizes the construction cost: more questions about the same video mean a lower memory-construction cost per question. In contrast, the evaluated ParaVT and LongVT baselines perform video-level processing for each new question. By preparing reusable evidence in advance, our memory-based design supports fast online QA.

3. Optional revisiting trades resources for accuracy. Disabling raw-video revisiting retains 57.5% on LVBench versus 58.9%, a 1.4-point difference (Table 3 in the paper). Text memory and saved keyframes remain accessible. Memory-based QA remains strong without new source frames; optional revisiting adds accuracy.

W4: Training pipeline complexity

Training has two parts: one-time SFT initialization of the Retrieval Evolver, followed by Agentic RL that alternately optimizes the Memory and Retrieval Evolvers. BEF and CEF only adjust the training focus; they introduce neither additional models nor training stages. No parameter update occurs at inference.

1. One-time SFT provides necessary initialization. Stage I teaches tool use, while Stage II broadens visual evidence seeking. Under the same subsequent co-evolution, LVBench accuracy is 58.9 with both stages, 54.2 with Stage II only, 40.6 with Stage I only, and 32.8 without SFT (Table 11 in the paper). Even after subsequent RL, removing SFT leaves a 26.1-point deficit. Neither single-stage variant recovers the full result, supporting the complementary roles of the two stages.

2. Agentic RL benefits from co-evolution and feedback. With feedback disabled, evolving both policies reaches 56.6 on LVBench, compared with 52.7 for the stronger one-sided variant (+3.9). A sequential alternative with matched cumulative per-policy training budgets reaches 55.4 versus 58.9 for the full scheme. It retains CEF but disables BEF and is evaluated at the sequential-stage endpoints rather than the full scheme's round-3 checkpoint. This comparison supports the full training scheme, not update order in isolation. BEF and CEF add no model parameters, yet removing both lowers all seven metrics in Table 3 in the paper by 2.2–6.7 points.

We have revised the presentation around these two parts and explicitly distinguish one-time initialization from training-time co-evolution. With the same model configuration, these ablations show that co-optimizing the two policies brings additional gains without adding trainable agents.

AuthorsOpenReview ↗

Response to Reviewer KK8c (3/3)

Q2: Training-time scope and effectiveness of BEF/CEF

1. BEF/CEF are inactive without training. Without policy updates, there is no update budget or capability-based sample selection for them to schedule.

2. BEF/CEF improve training-based evolution. As shown in Table 3 in the paper, removing both BEF and CEF reduces MMVU accuracy from 75.1% to 68.4% and MLVU accuracy from 72.4% to 69.4%. BEF allocates update budgets, while CEF selects capability-focused samples.

Q3: Training details and cost

1. Training setup.

(1.a) SFT. SFT uses LLaMA-Factory with DeepSpeed for distributed training on 8 NVIDIA H20 GPUs. Both stages train for one epoch: Stage I uses an effective batch size of 24 for 789 optimization steps, and Stage II uses 32 for 10,929 steps.

(1.b) RL. RL uses the verl framework with FSDP to distribute policy training across 8 NVIDIA H800 GPUs. During Memory Evolver RL, 6 GPUs train the Memory Controller, while 2 host the frozen Retrieval policy that scores candidate memories. During Retrieval Evolver RL, all 8 GPUs train the Retrieval policy. Each RL phase runs for one epoch over its allocated training groups; step counts vary by round with group budgets and batch sizes.

Full data-processing and training protocols are provided in Appendix A.3–A.6, with SFT and RL settings in Tables 7 and 8 in the paper, respectively. We have also submitted the core source code of VideoEvolve.

2. Training cost. SFT entries report the cost of each complete stage; RL entries report the average cost per round. GPU-hours cover the GPUs listed below.

Training stageGPU configurationTime (h)GPU-hours
SFT Stage I8×H2010.2782.18
SFT Stage II8×H2054.94439.52
Memory Evolver RL (per round)8×H80019.75158.00
Retrieval Evolver RL (per round)8×H80013.41107.28

Q4: Joint optimization versus alternating optimization

We jointly optimize the Memory and Retrieval Evolvers and compare this setting with our alternating optimization. Under joint optimization, both Evolvers are updated from the state available at the start of each round. Alternating optimization performs better on all five metrics by 2.0–3.0 points.

The alternating row reuses the matched “w/o BEF and CEF” setting in Table 3 in the paper; the joint-optimization row reports the new control.

Update scheduleVideo-MME Overall w/subVideo-MME Long w/subLongVideoBench OverallLongVideoBench LongLVBench
Alternating71.764.367.363.156.6
Joint Optimization69.762.164.360.553.9
Alternating Gain+2.0+2.2+3.0+2.6+2.7

These gains indicate that allowing Retrieval to adapt immediately to the updated memory is more effective than delaying this adaptation by one round.

1. The difference is whether Retrieval trains on old or updated memory. Alternating optimization first updates the Memory Evolver, rebuilds memory, and then trains the Retrieval Evolver on the updated memory. In the joint-optimization control, both Evolvers are updated from the round-start state, so Retrieval trains on the old memory and adapts to the new memory one round later. Because memory rebuilding is discrete, we implement joint optimization by updating both policies from the same round-start state.

2. The comparison isolates the update order. Both variants disable BEF and CEF and use the same initialization, per-policy update and rollout budgets, uniform sampling, and evaluation protocol.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Review Feedback and Rebuttal Discussion

Dear PC, SAC, AC, and Reviewers,

We sincerely thank the PC, SAC, AC, and reviewers for their careful evaluation and constructive discussion. Reviewer RyVz confirmed that the main concerns were resolved and maintained a rating of 3 (Weak acceptance), while Reviewer KK8c also confirmed that the main concerns were addressed and raised the rating from 2 to 3 (Weak acceptance).

The reviewers recognized these strengths:

  • Writing, motivation, and ablations (Reviewer KK8c): clear presentation, well-motivated memory–retrieval co-evolution, and ablations supporting gains beyond strengthening one component.
  • Technical value and results (Reviewer RyVz): technically sound and interesting co-evolution, state-of-the-art results across multiple benchmarks, and provision of core source code.
  • Joint optimization and feedback (Reviewer 1ETB): the value of connecting memory optimization to downstream feedback, with reasonable and effective bottleneck/capability feedback mechanisms.

We have incorporated the clarifications and additional experiments into the revised main text and appendix, including the requested fair comparisons, inference costs, and training details. Key revisions are marked in blue for comparison with the original submission. Key additions:

(1) Fair comparisons and model-capacity controls. Appendix A.9, Tables 17–18, adds shared-8B-backbone and 7B Writer controls. With role-specific LoRA adapters, the shared backbone retains 9.5/15.6-point co-evolution gains; the 7B Writer retains 8.1/8.8 points on Video-MME (w/sub)/LVBench. These gains are not solely attributable to larger model capacity. Table 2 adds VideoARM/VideoLucy comparisons. Appendix A.8, Table 12, matches WorldMM-8B's frame access: the same 768-frame construction pool, at most 64 online image inputs per question, and no subtitles or ASR. VideoEvolve leads by 7.1/14.4 points on Video-MME Long/LVBench.

(2) Co-evolution beyond SFT and independent improvements. Appendix A.9, Table 21, shows that roughly doubling Stage-II SFT data yields 0.1/1.1 points on Video-MME (w/sub)/LVBench, whereas co-evolution adds 7.1/8.6 points beyond full-data SFT. Section 4.4, Table 5, presents nine cross-checkpoint memory–retriever pairings. Under memory-only inference, the same later memory can hurt an earlier retriever but benefit the evolved retriever. The final pair's gain exceeds summed separate component-replacement gains by 2.5/5.3 points on Video-MME Long (w/sub)/LVBench. These results support mutual adaptation beyond independently stronger components; Appendix A.9 details the analysis.

(3) Stronger design ablations and broader evaluation. Appendix A.9 adds alternating-versus-joint update comparisons, CEF comparisons against uniform, class-balanced, and hard-example sampling, and fixed-memory retriever substitution (Tables 19–20, 22). These complement the component and memory-augmentation ablations. Appendix A.8, Table 13, adds Charades-STA temporal grounding: 52.6 mIoU without additional dataset-specific fine-tuning, versus ParaVT's paper-reported 50.1, extending evaluation beyond video QA.

(4) Qualitative evidence of what changes. Appendix A.11, Tables 23–30, presents eight paired cases with key tool calls, evidence, answers, and selected source frames. They show more accurate memory use, fewer redundant searches, useful memory augmentation, and better combination of stored facts. A fixed retriever answers a previously unresolved person–scene question after relevant memory is added; another case preserves the correct answer while reducing tool calls from 12 to 2. These examples distinguish missing evidence from ineffective evidence use.

(5) Clearer methodology and reproducibility. Sections 2–3 clarify evidence construction/use versus experience reuse, define C^k and the invalid-trajectory penalty, and distinguish SFT, co-evolution, and inference without parameter updates. Appendix A.5 illustrates BEF/CEF diagnosis, budget allocation, and sampling. Hardware, software, hyperparameters, and evaluation protocols are detailed in Appendices A.6 and A.8; Appendix A.6, Table 11, reports stage-wise training time and GPU-hours. Section 4.5, Table 6, reports latency and tokens: online QA takes 5.21 s/question, versus 12.04/15.40 for ParaVT/LongVT under the reported protocol, excluding offline memory construction.

Together, these revisions strengthen the evidence for learning what to remember and how to retrieve, while making the comparisons, mechanisms, and computational trade-offs transparent. Core source code was already provided at submission, as Reviewer RyVz noted. We have expanded the experimental details in response to the reviews and will release model weights upon acceptance to further facilitate reproduction.

Thank you again for your time and consideration.

Sincerely,

The Authors of Submission 547

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.