acceptodds
Under review as a conference paper at ICLR 2027

MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms

Abstract

Query-agnostic streaming video understanding requires vision-language models to continuously compress an indefinitely growing visual stream into a bounded memory before future queries are known. The performance depends critically on the memory mechanism—what observations to preserve, how to represent and consolidate them, and what information to retrieve when a query eventually arrives. Rather than designing a single memory architecture by hand, we formulate memory design as a search problem over executable memory programs. We introduce a lightweight domain-specific language that expresses memory mechanisms through structured primitives for representation, admission, retention, consolidation, budgeting, and retrieval, while enforcing causal and bounded-memory constraints. Although structured, the derived program space remains large and contains heterogeneous, conditionally dependent design choices whose effects can only be assessed via downstream execution. We therefore propose MemEvo, an LLM-driven auto-research framework that uses pretrained LLM as a semantics-aware proposal model to iteratively generate and refine candidate memory programs based on accumulated experimental feedback. At runtime, a deterministic evaluation pipeline validates and evaluates each candidate, while the underlying vision-language model remains frozen throughout discovery. We finally produce a training-free, bounded-memory mechanism. Extensive experiments on StreamingBench and OVO-Bench demonstrate strong streaming video understanding performance together with substantial context and inference efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Multimodal Large Language Models, Automated Memory Design, Query-Agnostic Video Memory, Auto Research

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_L9dTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies query-agnostic memory construction for streaming video understanding with frozen vision-language models. It represents a memory mechanism as a program with recent, episodic, and gist memory, and uses an LLM to propose and implement changes within a constrained DSL. Candidates are evaluated on NExT-GQA, and the selected program is subsequently tested on StreamingBench and OVO-Bench. The resulting system improves accuracy over the reported frozen-backbone baselines while substantially reducing reader context. The main question for this review is whether the evidence establishes the value of the proposed discovery procedure, beyond the effectiveness of the resulting memory configuration.

Critical strengths

The formulation makes a useful distinction between searching for a reusable memory-update program and optimizing memory contents for individual videos. The separation between query-independent updates and query-conditioned retrieval is appropriate for the task. Explicitly including bounded working state in Section 3.1 is also a sound design choice: bounding only the reader-facing memory would not suffice for a streaming system.

Searching on NExT-GQA and freezing the program before external benchmark evaluation is a meaningful strength. The improvements across two benchmarks and two Qwen backbones suggest that the selected configuration has practical value beyond its discovery dataset. Appendix A provides a helpful account of unsuccessful alternatives as well as successful changes, and Appendix D candidly illustrates how historical evidence can mislead current-scene understanding. These analyses are more informative than reporting only the final aggregate scores.

Critical weaknesses

The central contribution of LLM-guided discovery is not isolated experimentally. The three memory roles, packing order, and staged search schedule are prescribed, while many reported changes concern capacities, retrieval depths, thresholds, and segment lengths. The extensible operator interface could make this more than configuration tuning, but the paper does not establish how much that capability contributes. An improving trajectory over 89 candidates shows that search found a better configuration; it does not establish an advantage from semantic proposals, feedback-conditioned refinement, or operator invention. There is no budget-matched random or conventional configuration-search baseline, and no comparison with LLM proposals that do not receive experimental history. One informative control under the same evaluation budget, together with a clear account of which useful operators were newly implemented, would substantially strengthen the main claim.

The external comparisons do not sufficiently distinguish the benefit of the discovered mechanism from simpler context selection. In Figure 4a, recent memory alone scores 79.80, versus 81.40 for the complete system; this is a much smaller increment than the improvement from the 75.7 backbone result in Table 1. Moreover, R+G reaches 80.88, while R+E slightly underperforms R alone. These results support some value from historical memory, but provide limited aggregate support for the necessity of the selected episodic branch. The initial reference program is not evaluated on the external benchmarks, so the downstream benefit of discovery itself remains unclear. The same controls would be particularly useful on OVO-Bench BT, where historical retention is directly tested. Appendix B also states that baseline results are collected from prior papers, with different frame rates and incompletely aligned inference settings. Such comparisons provide context, but the small margins over the closest methods, such as 81.4 versus 81.3 on RTVU, should not be interpreted as established superiority without a matched protocol and an uncertainty estimate.

The efficiency evidence supports lower query-time reader cost more clearly than lower overall streaming cost. Section 4.3 explicitly profiles the reader, yet describes the latency reduction as end-to-end inference efficiency. Appendix C explains that the writer uses one foreground GPU for visual extraction and a second background GPU for summarization and consolidation. Its measured average processing time is already close to the one-second sampling interval. This is useful profiling, but it does not establish a resource-matched system-level advantage over the uniform-history baseline. In particular, Equation 2 permits flushing pending state at query arrival, and it is unclear whether the reported query latency includes this work and any unfinished writer jobs. A precise accounting of GPU allocation, writer compute, pending-state finalization, and query response time under an arriving stream would resolve the scope of the efficiency claim; faster reader inference alone remains a valid result.

Important details needed to reproduce the executable mechanism and its selection remain unspecified. For example, the paper names the selected coverage-change retention operator but does not define its scoring rule, and it calls the admission threshold a median without specifying how that median is estimated causally with bounded state. The numerical reader-context budget and the procedures for pooling and merging visual gist payloads are also unclear. These details determine the actual program behavior rather than merely implementation convenience. There are related ambiguities in the search record: Figure 2 and Appendix A.2 select episodic/gist retrieval depths of (8, 4), whereas Figure 3 displays (4, 8). Appendix A.1 reports identical scores for all nine coverage-retention variants but still chooses one specific variant, without an explicit tie-breaking rule or measurements explaining that choice. The paper should provide a consistent final configuration and enough operator and selection details to make the claimed discovery auditable.

Questions for authors
  1. Which operators were available in the initial library, and which were newly implemented by the agent? How much of the final improvement depends on the latter, compared with searching compositions and parameters of the initial library under the same budget?
  2. What sampling, frame cap, image resolution, prompting, and answer-scoring protocol produced the bare-backbone rows in Table 1? Under that same protocol, how do recent-only memory and the initial reference program perform on OVO-Bench BT?
  3. When a query arrives during an unfinished gist segment or a background consolidation job, what memory is made available to the reader, and is the time to finalize that snapshot included in the reported latency? How are reader and writer GPUs allocated in this measurement?
  4. Which retrieval depths were actually used for the external results? Why was the optical-flow retention variant selected among nine equal-scoring candidates, and does it change the stored or retrieved evidence on the discovery videos?
Reviewer_kxebOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Streaming video understanding tasks require the system to memorize important information from the video within a storage budget, even when future queries are not known in advance. Design choices of this system are heterogeneous, strongly coupled, and discrete. Instead of hand-crafting memory mechanisms, the authors propose a domain-specific language (DSL) for the design choices. The memory is organized into three complementary roles: recent memory, episodic evidence, and gist memory. This defines a practical search space for memory programs and allows automatic research with LLM agents to develop a better video understanding system.

Experiments show that MemEvo allows agents to explore alternative strategies and investigate potentials of cost-efficient settings. The discovered memory program improves the system for both immediate perception and backward memory lookup. Furthermore, different memory roles contribute to better performance on different types of tasks. Therefore MemEvo shows that automatic discovery of memory programs can be an alternative to individually designed memory mechanisms.

Critical strengths
  • S1: The approach of using a domain-specific language is compelling.

Identifying the space of possible memory mechanisms as the space of executable programs looks like a novel idea in the video understanding literature. Furthermore, the suggested three complementary functional roles of memory resembles the structure of a human brain: the prefrontal cortex processes recent memory whereas the medial temporal lobe indexes episodic evidence and stores gist memory. This makes the proposed approach compelling.

  • S2: Gradually searching the program space allows the research agent to start from stable baselines.

The research agent initially starts with a deliberately small library O0\mathcal{O}_0 of operators, which is extended with new strategies in further rounds. This approach allows the research agent start from a small search space and gradually improve the program with the introduced stategies. This helps avoid searching from a large space of programs from the beginning and being lost.

Critical weaknesses
  • W1: The proposed DSL remains a Cartesian product of individual strategies.

According to the proposed DSL (equation 5), a memory program pp is equivalent to a tuple (H,a,r,BE,s,z,c,BG,kE,δE,kG,δG)(H, a, r, B_E, s, z, c, \mathbf{B}_G, k_E, \delta_E, k_G, \delta_G), so that the set of possible programs is the Cartesian product of sets of possible strategies and hyperparameters. While the research LLM agent can discover a complicated strategy for each element, the overall logic of the memory program is fixed. I recommend the authors to test more expressive DSL syntaxes in future work. For example, what happen if the syntax allows R, E, G to have a different sequential composition of basic operators such as Recent or Admit? Could the language include control flow statements?

Questions for authors
  • Q1: More evaluation on StreamingBench to support the roles of different memories.

The real-time visual understanding (RTVU) subset of StreamingBench focuses on understanding real time changes of the video. Therefore, only the recent memory RR already allows MemEvo to reach the validation score of 79.80, compared to 75.7 of the Qwen3-VL baseline. What is the effect of adding episodic evidence or gist memory for the Contextual Understanding subset of StreamingBench, or the Backward Tracing subset of OVO-Bench?

  • Q2: Effect of the fixed stage search schedule.

The authors use a fixed schedule for searching strategies, where each stage searches for E, Retrieval, and G in that order. Why is this order choosed? What happen if the order is changed or there are no stages at all? Answering this question would be helpful to understand the nonlinear interaction of different memory roles and their strategies.

Reviewer_LSLxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines a query-agnostic streaming video QA problem using a frozen MLLM: some limited memory has to first be written causally prior to knowing the query and then consumed when the query arrives. Rather than manually creating the memory, the authors formulate a little DSL with an unchanging structure—a recent window, an episodic memory built from admission and retention operators, a gist memory built from segmentation, encoding and consolidation operators, and a query-time read that retrieves from the latter two under a context budget—and view the selection of operators and parameters as a program-search task. MemEvo takes GPT-5.6 as a proposal model which, given the DSL, experiment history and a Pareto archive, designs candidate edits (implementing new operators when necessary); each candidate is pre-flighted and tested with a frozen Qwen3-VL backbone and a Qwen3-VL embedding model for retrieval. The search proceeds in stages (episodic, retrieval, gist) on the NExT-GQA validation split, optimising the mean probability of the gold option. The resulting program is frozen and tested on StreamingBench RTVU and OVO-Bench RT/BT with Qwen2.5-VL and Qwen3-VL backbones, where it reports the best overall scores among the compared methods with a much shorter reader context. A role ablation on StreamingBench, reader/writer profiling, and two case studies are also provided.

Critical strengths

1 A pure, recyclable formulation. The correct abstraction of the problem is casting the memory mechanism (as opposed to memory contents) as an executable program in a DSL which imposes causality and limited capacity (Sec. 3.1–3.2). The DSL and runtime could still be beneficial to the community regardless of whether the LLM-induced search proves necessary. 2 Very strong and extensive findings with an unusually detailed account of the search process. The program achieves the best overall scores in Table 1, with the largest increase on Backward Tracing, and transfers to Qwen2.5-VL while the reader context stays bounded (Fig. 4b). All rounds are reported in Appendix A including those that were unsuccessful or plateaued, which allows a reader to audit the search.

Critical weaknesses

1 The main statement that LLM-driven program search can be an effective substitute to hand design is not analyzed in comparison with the plain options. There is no equal-budget baseline (random, evolutionary or Bayesian search over the same DSL) and no expert-written program in the DSL, so the paper does not demonstrate that the LLM proposal model adds any value. Table 3 and Appendix A show that almost all candidates are reparameterisations of the fixed skeleton and the structurally new operators were not adopted, so in practice the search resembled a staged, LLM-picked grid sweep, contradicting Sec. 1 and 3.3. The decision signal is also weak: many rounds finish in exact ties, promotions rest on very small dev-objective differences without any noise estimate, and the search was run only once. The tie-breaking “designated stage selection procedure” is never discussed. 2 The experiments cannot separate what the found memory contributes, and the development set does not seem capable of demonstrating the bounded-memory behavior the paper is about. Fig. 4a shows that recent memory alone already scores well above the backbone row in Table 1, so most of the headline improvement is attributable to the recent window rather than to the discovered E/G memory; the ablation is also shown on RTVU only, not on OVO-Bench BT, the only subset that actually tests long-range memory. NExT-GQA videos are short enough that Retain cannot realistically ever fire under the chosen admission rule and capacity (consistent with the identical scores in App. A.1 E4), so the retention and consolidation operators were chosen without empirical signal and then applied to much longer videos. The efficiency and SOTA assertions rest on uncontrolled comparisons: reductions are measured against uniformly sampled history rather than HERMES/FluxMem/OASIS, writer-side cost is not compared, Table 1 rows use different sampling rates, and the margins above the strongest competitors are within noise.

Questions for authors

1 Please report the reference program’s results on RTVU and OVO RT/BT, and the role ablation of Fig. 4a repeated on OVO-Bench BT, so that the contribution of the discovered E/G memory can be separated from that of the recent window. 2 Can you provide an equal-budget non-LLM search over the same DSL and/or an expert-written program, together with the variation of the final program across repeated search runs? 3 What proportion of NExT-GQA dev videos actually triggered Retain and Merge during search, and what precisely is the “designated stage selection procedure” used to break ties?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.