acceptodds
Withdrawn from ICLR 2027

Write Once, Query Many: Evolving Compact Reusable Memory for Long Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long videos often exceed the context capacity of multimodal large language models (MLLMs), while agentic search repeats costly video processing and grounding for each query. Reusable memory can amortize this cost, but must preserve information for questions unknown during construction. We propose Video-EMem, an evolvable video memory harness built on time-anchored perceptual observations. This representation combines compact free-form text with sparse images to preserve semantic content and complementary visual details across temporal scales, supporting temporal navigation without a fixed semantic schema. A Writer encodes each video once by consolidating overlapping multiscale observations, and a Reader retrieves and organizes evidence solely from the resulting memory. With the underlying models frozen, we evolve their workflows and operators using distinct feedback: query-independent source fidelity and coverage for the Writer, and QA failures in evidence localization and utilization for the Reader. Experiments support this separation for generalization beyond development queries, with our evolution strategy improving average accuracy by **7.2%** over the unevolved harness. While reducing storage by about **99.86%** on average relative to raw videos, Video-EMem improves Qwen3-VL-30B by an average of 2.6% across three long-video QA benchmarks over direct input of 2,048 frames. Using the same o3 answer model as the state-of-the-art agentic method, it improves average accuracy by 2.5% with lower per-query latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video, video understanding, MLLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zy3EOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Video-EMem for “write-once, query-many” long video understanding: it first encodes a video into a reusable memory under a fixed storage budget, then answers subsequent questions solely from that memory. The memory consists of text records anchored to temporal intervals and sparse images. The Writer merges multi-scale captions, allocates token budgets, compresses the text, and repairs information lost during compression, before selecting images that complement details. The Reader retrieves text relevant to the question, retrieves images as needed, and organizes the evidence into context for answer generation. The paper achieves SOTA performance on various long video benchmarks.

Critical strengths
  • The paper focuses on an important problem: constructing reusable memory for long video understanding.
  • The proposed method achieves strong overall performance on the long video benchmarks.
  • The ablation studies are reasonably comprehensive.
Critical weaknesses
  • The presentation introduces too many meaningless symbols that add little explanatory value.
  • Essential implementation details are missing, including the actual prompts, the implementation of ρMLLM\rho_{\mathrm{MLLM}}, and the concrete procedure used to evolve the operators.
  • The paper lacks a complete, end-to-end example.
  • Excessive defensive writing reduces readability.
  • The efficiency discussion emphasizes low per-query cost without adequately accounting for the potentially substantial cost of the Writer. Memory construction costs and per-question answering costs should be reported separately.
Questions for authors
  • Please simplify the presentation by removing unnecessary formalism and reducing defensive language.
  • Please provide sufficient implementation details for each step of the Writer, Reader, and evolution procedure. Each component should have an informative operational description, for example, the LLM prompt, rather than being represented only by a symbol with specified inputs and outputs.
  • Please include an end-to-end example of memory construction and retrieval, with the complete intermediate outputs and execution traces of the Writer and Reader, as well as the changes of Writer and Reader along the evolving progress.
  • Please report model-call costs more comprehensively, separating the one-time cost of constructing memory from the cost of answering each subsequent question.
Reviewer_iGkFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Video-EMem, a compact reusable memory system for long-video QA. It first encodes each video into time-anchored textual observations and sparse complementary images, then answers future questions by retrieving and organizing evidence only from this frozen memory. The authors evolve the Writer using source-preservation feedback over generated source captions, and evolve the Reader using attributed QA failures, while keeping underlying models frozen. Experiments on LVBench, Video-MME-L, and LongVideoBench-L report accuracy improvements over direct VLM and agentic baselines, as well as substantially lower per-query latency.

Critical strengths

Fast query-time access

The main practical benefit is clear. Once the memory is written, the Reader retrieves from a small stored representation and makes one answer-model call. This gives a large query-time advantage over multi-round video agents while keeping strong accuracy.

Strong experiments

The paper compares against direct VLMs and agentic methods on three long-video benchmarks. The Writer/Reader ablations, memory-budget study, visual-memory study, and transfer from LVBench to the other benchmarks make the empirical story convincing.

Critical weaknesses

The reusable-memory framing is incremental

Query-independent memory construction followed by query-specific retrieval is not new. LangRepo already has explicit write and read operations over a multi-scale video repository. WorldMM constructs multi-scale episodic memory, semantic memory, and visual memory before query-time retrieval. DVD also builds a video database before retrieval. The contribution is therefore better viewed as a compact bounded memory design plus workflow evolution, rather than a new write-once/query-many paradigm.

The effect of evolution is not explained clearly

The main new component is the evolution of the Writer and Reader, but the paper mostly reports final scores. It does not show concrete before/after examples of operators, representative accepted revisions, or analysis of why particular revisions improve performance. This makes it difficult to understand what the evolution procedure actually learns and to reproduce the claimed gains. In addition, Writer evolution measures preservation of generated source captions; the paper does not evaluate whether those captions are faithful to the original video.

Questions for authors

(1) Could the authors clarify the “Joint QA Evolution” ablation? In particular, what modules are optimized, what feedback is used, and how does it differ from the proposed separate Writer/Reader evolution?

(2) Could the authors provide a few representative examples of the evolution process, showing the initial operator behavior, the revised behavior, and why the revision improves the result? It would also be helpful to discuss how source-caption preservation relates to factual preservation of the original video.

Reviewer_GtiaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Video-EMem introduces a reusable memory framework for long-video understanding. Videos are compressed into compact representations consisting of time-anchored textual observations and sparse complementary images, allowing multiple questions to be answered without repeatedly processing the original video. Memory construction and retrieval are handled by separate Writer and Reader workflows, evolved independently using source-preservation and question-answering feedback. Underlying models remain frozen throughout evolution. Experiments on three long-video QA benchmarks demonstrate improved accuracy, substantial storage reduction relative to raw videos, and lower per-query latency than selected direct-input and agentic baselines.

Critical strengths
  1. Processing long videos once and reusing compact memory across multiple queries offers a practical alternative to repeated video analysis. Using the same o3 answer model, Video-EMem achieves 72.3% average accuracy compared with 69.8% for DVD, while reducing per-query latency.

  2. Separately evolving the Writer and Reader with different feedback objectives is a meaningful design choice. The improvements also transfer to Video-MME-L and LongVideoBench-L without further adaptation, suggesting that the learned workflows generalize beyond the development benchmark.

Critical weaknesses
  1. The separate evolution of the Writer and Reader builds on existing LLM-driven workflow optimization techniques. Table 2 demonstrates that the proposed procedure is effective, but an additional ablation could help clarify its contribution. How large would the performance gap be compared with a fixed but carefully designed workflow?

  2. The criterion used to assess information preservation raises concerns. The text-Writer's coverage diagnostic compares memory against model-generated captions rather than the original video. Details omitted by the captioner may therefore remain undetected during subsequent consolidation. Figure 5 illustrates that retained images can recover some missing details, but the evaluation does not quantify how reliably this supplementation preserves information absent from the initial descriptions.

Questions for authors
  1. Appendix B illustrates several operator changes discovered during evolution. Which changes contribute most to the reported improvements? Would similar strategies emerge if evolution were performed on a different set of development videos? This would help distinguish reusable workflow improvements from refinements specific to the current setting.

  2. How does the framework handle cases where a question cannot be answered because relevant evidence was omitted during memory construction? Section 3.4.2 excludes information absent from memory when diagnosing Reader failures. Can such cases be distinguished from retrieval failures or failures to utilize available evidence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.